mirror of
https://github.com/discourse/discourse.git
synced 2025-05-29 01:31:35 +08:00
PERF: Speed up secure media and ACL sync rake tasks (#16849)
Incorporates learnings from /t/64227: * Changes the code to set access control posts in the rake task to be an efficient UPDATE SQL query. The original version was timing out with 312017 post uploads, the new query took ~3s to run. * Changes the code to mark uploads as secure/not secure in the rake task to be an efficient UPDATE SQL query rather than using UploadSecurity. This took a very long time previously, and now takes only a few seconds. * Spread out ACL syncing for uploads into jobs with batches of 100 uploads at a time, so they can be parallelized instead of having to wait ~1.25 seconds for each ACL to be changed in S3 serially. One issue that still remains is post rebaking. Doing this serially is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates but this is limited by max_old_rebakes_per_15_minutes. It would be better to fan this rebaking out into jobs like we did for the ACL sync, but that should be done in another PR.
This commit is contained in:
@ -15,6 +15,11 @@
|
||||
# on the current secure? status, otherwise there would be a lot of additional
|
||||
# complex queries and joins to perform. Over time more of these specific
|
||||
# queries will be implemented.
|
||||
#
|
||||
# NOTE: When updating this to add more cases where uploads will be marked
|
||||
# secure, consider uploads:secure_upload_analyse_and_update as well, which
|
||||
# does not use this class directly but uses an SQL version of its rules for
|
||||
# efficient updating of many uploads in bulk.
|
||||
class UploadSecurity
|
||||
@@custom_public_types = []
|
||||
|
||||
|
Reference in New Issue
Block a user