Commit Graph

24 Commits

Author SHA1 Message Date
bd0a7553c4 DEV: Detect when s3 inventory failure is caused by etag difference (#10427) 2020-08-13 09:30:28 +10:00
b950b3fb3f DEV: Add verified to uploads and fill in S3 inventory (#10406)
When we run the S3 inventory, mark uploads that exist as verified true, those that don't as verified false, and uploads not included in the check / not yet checked as verified nil.
2020-08-11 14:43:51 +10:00
16c65a94f7 PERF: Preload S3 inventory data for multisite clusters 2020-07-29 10:31:55 +01:00
ec4024fe6d FIX: Keep by_users check in S3 inventory
Partial revert of 8515d8fa - the by_users check is ensuring we don't raise errors for fixtures
2020-07-21 17:19:56 +01:00
7f2b5a446a PERF: Remove post_upload recovery in daily EnsureS3UploadsExistence job (#10173)
This is a very expensive process, and it should only be required in exceptional circumstances. It is possible to run a similar recovery using `rake uploads:recover` (5284d41a8e/lib/upload_recovery.rb (L135-L184))
2020-07-06 16:26:40 +01:00
73b04976e5 FIX: Use updated_at in the S3 inventory job (#8823)
When we change upload's sha1 (e.g. when resizing images) it won't match the data in the most recent S3 inventory index. With this change the uploads that have been updated since the inventory has been generated are ignored.
2020-01-31 11:02:44 +01:00
9919ee1900 FIX: remove the tmp inventory files after the s3 uploads check. 2019-08-13 11:52:57 +05:30
ad04ce9f43 FIX: remove post upload record creation inside 'find_missing_uploads' method. 2019-07-19 01:44:08 +05:30
35d6fff69e PERF: use url instead of file key in temporary inventory table. 2019-06-13 22:03:58 +05:30
2a7065c505 FIX: skip uploads without etag in s3 inventory check. 2019-05-20 00:09:52 +05:30
e8fafbc123 List and restore missing post uploads from S3 inventory. 2019-05-04 01:16:20 +05:30
4ea21fa2d0 DEV: use #frozen_string_literal: true on all spec
This change both speeds up specs (less strings to allocate) and helps catch
cases where methods in Discourse are mutating inputs.

Overall we will be migrating everything to use #frozen_string_literal: true
it will take a while, but this is the first and safest move in this direction
2019-04-30 10:27:42 +10:00
ad44243a57 Removed unused let blocks (#7446)
The bodies of these blocks were never evaluated.
2019-04-29 15:08:56 +08:00
a8f410a9c5 FEATURE: Create new helper method 'Discourse.stats' (#7388) 2019-04-17 12:45:04 +05:30
df6ef856e6 DEV: save missing s3 uploads count in redis 2019-04-04 19:05:57 +05:30
243fb8d9ad Fix the build. 2019-03-13 17:39:07 +08:00
563b953224 DEV: Add 'backfill_etags_' to the method name since it also backfilling the etags 2019-02-19 21:54:35 +05:30
0472bd4adc FIX: Remove 'backfill_etags' keyword argument from 'uploads:missing' rake task
And etags backfilling code is optimized
2019-02-15 00:34:35 +05:30
b5fbd7385f FIX: run the rake task only for uploads created before a day from inventory date 2019-02-14 17:53:08 +05:30
f7d03b2406 Stop the automatic creation of inventory 'gz' file in fixtures 2019-02-14 14:25:45 +05:30
7b5931013a Update rake task to backfill etags from s3 inventory 2019-02-14 05:18:06 +05:30
1045bbc35b FIX: S3 inventory data can be splitted into multiple csv files 2019-02-14 03:41:52 +05:30
ff12c4b2d4 FIX: Bucket name is missing in S3 inventory data path 2019-02-06 19:16:08 +05:30
b4f713ca52 FEATURE: Use amazon s3 inventory to manage upload stats (#6867) 2019-02-01 10:10:48 +05:30