Commit Graph

183 Commits

Author SHA1 Message Date
efc481d9c0 DEV: Use puts instead of printing newline (follow up on c5b7bda1) 2019-01-05 01:20:00 +05:30
c5b7bda198 DEV: Show migrate_to_s3 output on a new line 2019-01-04 18:09:54 +05:30
9e50813252 FIX: Pass all necessary options in migrate_to_s3 rake task. 2019-01-02 09:11:23 +08:00
5381096bfd PERF: new 'migrate_to_s3' rake task 2018-12-26 17:34:49 +01:00
fd272eee44 FEATURE: Make uploads:missing task compatible with s3 uploads 2018-11-27 00:54:51 +05:30
bc41057949 minor copy edit 2018-11-20 12:07:56 +05:30
1a9a2bd5c1 DEV: Report the missing uploads count 2018-11-19 12:06:46 +05:30
4a12cfaecb Remove trailing whitespace for Rubocop 2018-11-13 17:19:26 +05:30
ea94323766 FIX: 'migrate_from_s3' rake task should respect max sizes (#6598)
Rather than hardcode a maximum size of 20MB for uploads migrated from S3, the task should use site settings for this value.
2018-11-13 12:27:38 +01:00
14ff47f6f1 Fix typo. 2018-11-08 16:42:12 +08:00
7290145641 PERF: Speed up migrate_to_s3 rake task.
* Prioritizes non-image uploads
* Does one remap per upload instead of 3 remaps previously
* Every 100 uploads migrated, do 2 remaps which fixes broken
  URLs
* Exclude email_logs table from remap
2018-11-08 16:39:56 +08:00
0232a3b5e5 PERF: Exclude tables when remapping in migrate_to_s3 rake task. 2018-11-08 12:37:36 +08:00
918633aa12 FIX: upload:migrate_to_s3 rake task not remapping properly. 2018-10-10 15:09:21 +08:00
24c55bd613 Add dry run option to UploadRecovery. 2018-09-12 21:53:01 +08:00
c053f8ccf6 New rake task uploads:recover. 2018-09-12 01:52:30 -07:00
6d01e0aa04 DEV: Print the error class in uploads:list_posts_with_broken_images. 2018-09-12 01:06:51 -07:00
94ff428571 Pass the right value to rake task. 2018-09-10 20:07:28 +08:00
4a966c639d DEV: Update uploads:list_posts_with_broken_images to recover from tombstone. 2018-09-10 17:01:41 +08:00
68572b8afc Print error messages on why upload fails to save. 2018-09-10 16:02:13 +08:00
0aca80e92a Fixes to uploads:list_posts_with_broken_images. 2018-09-10 15:16:29 +08:00
8496537590 Add RECOVER_FROM_S3 to uploads:list_posts_with_broken_images rake task. 2018-09-10 15:14:30 +08:00
72834f19ff DEV: Add rake tasks to list posts with broken images. 2018-09-05 16:54:15 +08:00
33f4aa2835 Fix migrate_to_s3 rake task, File#close! is not a valid method. 2018-08-16 13:26:58 +08:00
Sam
1fc2597626 better error handling for upload extension fixer 2018-08-10 11:28:22 +10:00
0879610ffd Add missing require in uploads:fix_incorrect_extensions. 2018-08-08 15:39:37 +08:00
17047806b9 Add a rake task to fix uploads with wrong extension. 2018-08-08 13:15:17 +08:00
14bbd5d167 FIX: use the right URL when downloading the file from S3 2018-08-02 10:04:41 +02:00
0f1137dafa FIX: 'migrate_from_s3' rake task wasn't handling short urls 2018-08-01 22:58:46 +02:00
Sam
5f64fd0a21 DEV: remove exec_sql and replace with mini_sql
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
9e7d5a3cdf FIX: 'uploads:recover_from_tombstone' rake task wasn't restoring attachments 2018-03-28 15:51:47 +02:00
5782a23a69 Allow uploads:missing to skip optimized image check. 2018-03-12 20:08:53 +08:00
cbf4204ecc FIX: uploads:migrate_from_s3 wasn't destroying records before creating new ones 2018-01-24 13:22:07 +01:00
28d432263e rework the uploads:migrate_from_s3 rake task 2018-01-23 18:48:06 +01:00
Sam
83a346cd68 FIX: migrate_from_s3 rake task not working 2018-01-22 11:42:09 +11:00
b6c9d7a9d2 move requires to top of file 2018-01-19 17:51:42 +01:00
2d49250107 FIX: support recovering images using 'upload://' scheme 2018-01-19 15:40:17 +01:00
Sam
70bb2aa426 FEATURE: allow specifying s3 config via globals
This refactors handling of s3 so it can be specified via GlobalSetting

This means that in a multisite environment you can configure s3 uploads
without actual sites knowing credentials in s3

It is a critical setting for situations where assets are mirrored to s3.
2017-10-06 16:20:01 +11:00
5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
b7e6620109 FIX: 'uploads:migrate_from_s3' rake task 2017-06-23 10:15:11 +02:00
94cee37edd FIX: uploads:recover_from_tombstone rake task 2017-06-22 17:00:05 +02:00
cdbe027c1c Refactor FileHelper to use keyword arguments. 2017-05-24 13:54:26 -04:00
9641d2413d REFACTOR: upload workflow creation into UploadCreator
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
b5d6d3c326 FIX: Make recover from tombstone script work for multisite. 2016-09-22 12:28:36 +08:00
333d6f9f10 More improvements to restore from tombstone script. 2016-09-22 10:39:39 +08:00
1d4451db5d Improve restore from tombstone script by not having to loop through all posts. 2016-09-21 18:01:59 +08:00
b62b6ba12c Improve restore from tombstone rake task. 2016-09-21 17:47:27 +08:00
3af5b19002 Add rake task to restore missing uploads from tombstone. 2016-09-21 16:56:32 +08:00
e4b75f604c FIX: Make clean up upload script a safer task to run. 2016-09-05 10:06:02 +08:00
df8d24734a Regenerate missing optimized images for all dbs by default. 2016-09-02 13:06:31 +08:00
e4883725e6 Script to analyze uploads on a site. 2016-09-02 12:07:06 +08:00