DEV: Introduce hidden s3_inventory_bucket site setting (#27304)

This commit introduces a hidden `s3_inventory_bucket` site setting which
replaces the `enable_s3_inventory` and `s3_configure_inventory_policy`
site setting.

The reason `enable_s3_inventory` and `s3_configure_inventory_policy`
site settings are removed is because this feature has technically been
broken since it was introduced. When the `enable_s3_inventory` feature
is turned on, the app will because configure a daily inventory policy for the
`s3_upload_bucket` bucket and store the inventories under a prefix in
the bucket. The problem here is that once the inventories are created,
there is nothing cleaning up all these inventories so whoever that has
enabled this feature would have been paying the cost of storing a whole
bunch of inventory files which are never used. Given that we have not
received any complains about inventory files inflating S3 storage costs,
we think that it is very likely that this feature is no longer being
used and we are looking to drop support for this feature in the not too
distance future.

For now, we will still support a hidden `s3_inventory_bucket` site
setting which site administrators can configure via the
`DISCOURSE_S3_INVENTORY_BUCKET` env.
This commit is contained in:
Alan Guo Xiang Tan
2024-06-10 13:16:00 +08:00
committed by GitHub
parent 3bff633ce0
commit 8cf4ed5f88
13 changed files with 62 additions and 260 deletions

View File

@@ -1,21 +0,0 @@
# frozen_string_literal: true
require "s3_inventory"
module Jobs
# if upload bucket changes or inventory bucket changes we want to update s3 bucket policy and inventory configuration
class UpdateS3Inventory < ::Jobs::Base
def execute(args)
unless SiteSetting.enable_s3_inventory? && SiteSetting.Upload.enable_s3_uploads &&
SiteSetting.s3_configure_inventory_policy
return
end
%i[upload optimized].each do |type|
s3_inventory = S3Inventory.new(Discourse.store.s3_helper, type)
s3_inventory.update_bucket_policy if type == :upload
s3_inventory.update_bucket_inventory_configuration
end
end
end
end

View File

@@ -15,10 +15,6 @@ module Jobs
end
end
def s3_helper
Discourse.store.s3_helper
end
def prepare_for_all_sites
inventory = S3Inventory.new(s3_helper, :upload)
@db_inventories = inventory.prepare_for_all_sites
@@ -26,8 +22,7 @@ module Jobs
end
def execute(args)
return if !SiteSetting.enable_s3_inventory
require "s3_inventory"
return if (s3_inventory_bucket = SiteSetting.s3_inventory_bucket).blank?
if !@db_inventories && Rails.configuration.multisite && GlobalSetting.use_s3?
prepare_for_all_sites
@@ -37,13 +32,13 @@ module Jobs
preloaded_inventory_file =
@db_inventories[RailsMultisite::ConnectionManagement.current_db]
S3Inventory.new(
s3_helper,
:upload,
s3_inventory_bucket:,
preloaded_inventory_file: preloaded_inventory_file,
preloaded_inventory_date: @inventory_date,
).backfill_etags_and_list_missing
else
S3Inventory.new(s3_helper, :upload).backfill_etags_and_list_missing
S3Inventory.new(:upload, s3_inventory_bucket:).backfill_etags_and_list_missing
end
end
end