discourse/config
Martin Brennan e4350bb966
FEATURE: Direct S3 multipart uploads for backups (#14736)
This PR introduces a new `enable_experimental_backup_uploads` site setting (default false and hidden), which when enabled alongside `enable_direct_s3_uploads` will allow for direct S3 multipart uploads of backup .tar.gz files.

To make multipart external uploads work with both the S3BackupStore and the S3Store, I've had to move several methods out of S3Store and into S3Helper, including:

* presigned_url
* create_multipart
* abort_multipart
* complete_multipart
* presign_multipart_part
* list_multipart_parts

Then, S3Store and S3BackupStore either delegate directly to S3Helper or have their own special methods to call S3Helper for these methods. FileStore.temporary_upload_path has also removed its dependence on upload_path, and can now be used interchangeably between the stores. A similar change was made in the frontend as well, moving the multipart related JS code out of ComposerUppyUpload and into a mixin of its own, so it can also be used by UppyUploadMixin.

Some changes to ExternalUploadManager had to be made here as well. The backup direct uploads do not need an Upload record made for them in the database, so they can be moved to their final S3 resting place when completing the multipart upload.

This changeset is not perfect; it introduces some special cases in UploadController to handle backups that was previously in BackupController, because UploadController is where the multipart routes are located. A subsequent pull request will pull these routes into a module or some other sharing pattern, along with hooks, so the backup controller and the upload controller (and any future controllers that may need them) can include these routes in a nicer way.
2021-11-11 08:25:31 +10:00
..
cloud/cloud66 DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
environments FIX: remove 'crawl_images' site setting (#14646) 2021-10-19 17:12:29 +05:30
initializers FEATURE: Cache CORS preflight for MessageBus (#14616) 2021-10-15 00:23:53 -03:00
locales FEATURE: Add read-only scope to API keys (#14856) 2021-11-10 17:48:00 +02:00
application.rb FIX: Check env for multisite config path even if config file exists (#14536) 2021-10-06 13:24:50 -05:00
boot.rb DEV: Remove deprecated bootsnap options (#11929) 2021-02-02 14:39:51 +01:00
cdn.yml.sample Initial release of Discourse 2013-02-05 14:16:51 -05:00
database.yml remove some hardcoded 'localhost's from dev environment (#14801) 2021-11-03 11:26:44 +08:00
deploy.rb.sample enough with the malloc limit, not needed 2016-05-25 21:09:07 +10:00
dev_defaults.yml FEATURE: Add post edits count to user activity (#13495) 2021-08-02 10:15:53 -04:00
discourse_defaults.conf FEATURE: Make the multisite config path configurable (#14308) 2021-09-10 14:19:52 -05:00
discourse.config.sample enough with the malloc limit, not needed 2016-05-25 21:09:07 +10:00
discourse.pill.sample Improve bluepill sample config. 2014-01-31 16:09:35 -05:00
environment.rb DEV: replace mailcatcher references with mailhog (#14500) 2021-10-05 15:48:06 +05:30
logrotate.conf Replace Clockwork with Sidetiq 2013-08-14 21:39:40 +02:00
multisite.yml.production-sample DEV: Remove db_id from sample multisite config. 2020-05-29 10:48:29 +08:00
nginx.global.conf Address @Supermathie's concerns in PR1430 2013-09-30 16:28:22 -04:00
nginx.sample.conf FEATURE: Optimize images before upload (#13432) 2021-06-23 12:31:12 -03:00
projections.json DEV: Use .hbr for raw template file extension (#8883) 2020-02-11 13:38:12 -06:00
puma.rb remove daemonize setting (#12232) 2021-03-01 16:42:50 +11:00
routes.rb FIX: improvements for download local dates (#14588) 2021-10-14 09:22:44 +11:00
sidekiq.yml FEATURE: introduce ultra_low priority queue 2019-01-17 14:53:19 +11:00
site_settings.yml FEATURE: Direct S3 multipart uploads for backups (#14736) 2021-11-11 08:25:31 +10:00
spring.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
thin.yml.sample Add sample Capistrano deployment files 2013-05-02 19:53:37 -07:00
unicorn_launcher FIX: Increase timeout when trying to reload unicorn. 2018-12-04 13:43:14 +08:00
unicorn_upstart.conf enough with the malloc limit, not needed 2016-05-25 21:09:07 +10:00
unicorn.conf.rb Revert "DEV: suppress assets logs from qunit tests (#13871)" 2021-07-29 13:28:24 +08:00