mirror of
https://github.com/discourse/discourse.git
synced 2024-11-30 20:54:00 -06:00
faf5b4d3e9
Incorporates learnings from /t/64227: * Changes the code to set access control posts in the rake task to be an efficient UPDATE SQL query. The original version was timing out with 312017 post uploads, the new query took ~3s to run. * Changes the code to mark uploads as secure/not secure in the rake task to be an efficient UPDATE SQL query rather than using UploadSecurity. This took a very long time previously, and now takes only a few seconds. * Spread out ACL syncing for uploads into jobs with batches of 100 uploads at a time, so they can be parallelized instead of having to wait ~1.25 seconds for each ACL to be changed in S3 serially. One issue that still remains is post rebaking. Doing this serially is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates but this is limited by max_old_rebakes_per_15_minutes. It would be better to fan this rebaking out into jobs like we did for the ACL sync, but that should be done in another PR.
40 lines
1.5 KiB
Ruby
40 lines
1.5 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
module Jobs
|
|
# Sometimes we need to update a _lot_ of ACLs on S3 (such as when secure media
|
|
# is enabled), and since it takes ~1s per upload to update the ACL, this is
|
|
# best spread out over many jobs instead of having to do the whole thing serially.
|
|
class SyncAclsForUploads < ::Jobs::Base
|
|
def execute(args)
|
|
return if !Discourse.store.external?
|
|
return if !args.key?(:upload_ids)
|
|
|
|
# TODO (martin) Change the logging here to debug after acl_stale implemented.
|
|
#
|
|
# Note...these log messages are set to warn to ensure this is working
|
|
# as intended in initial production trials, this will be set to debug
|
|
# after an acl_stale column is added to uploads.
|
|
time = Benchmark.measure do
|
|
Rails.logger.warn("Syncing ACL for upload ids: #{args[:upload_ids].join(", ")}")
|
|
Upload.includes(:optimized_images).where(id: args[:upload_ids]).find_in_batches do |uploads|
|
|
uploads.each do |upload|
|
|
begin
|
|
Discourse.store.update_upload_ACL(upload, optimized_images_preloaded: true)
|
|
rescue => err
|
|
Discourse.warn_exception(
|
|
err,
|
|
message: "Failed to update upload ACL",
|
|
env: {
|
|
upload_id: upload.id,
|
|
filename: upload.original_filename
|
|
}
|
|
)
|
|
end
|
|
end
|
|
end
|
|
Rails.logger.warn("Completed syncing ACL for upload ids in #{time.to_s}. IDs: #{args[:upload_ids].join(", ")}")
|
|
end
|
|
end
|
|
end
|
|
end
|