Commit Graph

7534 Commits

Author SHA1 Message Date
3d55f2e3b7 FIX: Improvements and fixes to the image downsizing script (#9950)
Fixed bugs, added specs, extracted the upload downsizing code to a class, added support for non-S3 setups, changed it so that images aren't downloaded twice.

This code has been tested on production and successfully resized ~180k uploads.

Includes:

* DEV: Extract upload downsizing logic
* DEV: Add support for non-S3 uploads
* DEV: Process only images uploaded by users
* FIX: Incorrect usage of `count` and `exist?` typo
* DEV: Spec S3 image downsizing
* DEV: Avoid downloading images twice
* DEV: Update filesizes earlier in the process
* DEV: Return false on invalid upload
* FIX: Download images that currently above the limit (If the image size limit is decreased, then there was no way to resize those images that now fall outside the allowed size range)
* Update script/downsize_uploads.rb (Co-authored-by: Régis Hanol <regis@hanol.fr>)
2020-06-11 14:47:59 +02:00
b7e70850e4 FIX: Allow users to add emails which were deleted before 2020-06-11 14:54:11 +03:00
2e8075bac3 FIX: Ignore Redis readonly errors in RateLimiter#rollback!.
This is similar to what we're doing in `RateLimiter#performed!`.
2020-06-11 15:13:11 +08:00
90dc6c8ce5 DEV: Fix deprecation warning when using Redis#close with redis namespace. 2020-06-11 14:09:19 +08:00
531eca985e DEV: Log error backtrace in Discourse.warn_exception without logster. 2020-06-11 10:49:46 +08:00
5f3fd23fa8 DEV: Add stub flush_sw task to prevent rebuild errors (#10021) 2020-06-10 13:07:37 -07:00
ac555610d9 DEV: Trigger an event before creating an upload 2020-06-10 16:30:53 -03:00
81a265cecd Version bump to v2.5.0.beta7 2020-06-10 13:38:18 -04:00
5bfe1ee4f1 FEATURE: Improve UX support for multiple email addresses (#9691) 2020-06-10 19:11:49 +03:00
a3dfd553a1 Revert "Bump redis to 4.2.0."
This reverts commit 98bc28cea259b823552e1b953506bcf2f76347a9.
2020-06-10 14:52:05 +08:00
98bc28cea2 Bump redis to 4.2.0. 2020-06-10 14:28:56 +08:00
70a88111dd FIX: prevent re-flagging when we have reviewed flags before (#10010)
FIX: prevent re-flagging when we have reviewed flags before

Fixes an edge case where a review can be reflagged when:
User flags as inappropriate.
Moderator rejects the flag.
Another user re-flags the post as spam.

Before, anyone was able to re-flag as inappropriate despite it being flagged
previously. With this, users are unable to re-flag for the same reason
regardless of reviewable status.
2020-06-09 15:26:10 -07:00
3094459cd9 FEATURE: multiple use invite links (#9813) 2020-06-09 20:49:32 +05:30
c35649ccf1 DEV: Remove Redis noop logging.
When Redis drops into readonly mode, it floods the log. The log isn't
actionable and we have other means of knowing that an app is in readonly
mode.
2020-06-09 16:45:34 +08:00
a1c13eb3c6 DEV: Redis failover should only clear redis recently readonly. 2020-06-09 16:36:31 +08:00
4065cd198a DEV: Print backtrace of error when plugin fails to initialize. 2020-06-09 10:25:43 +08:00
d4caf69ed7 DEV: makes SKIP_INSTALL_PLUGINS called last and global (#9990) 2020-06-05 17:59:23 +02:00
1c48853ede DEV: allow skipping of a single plugin when installing
SKIP_INSTALL_PLUGINS can be used to tell plugin:install_all_official
to skip a plugin.

Comma seperated list
2020-06-05 18:26:25 +10:00
e82d4d8a75 DEV: Update rails_failover to avoid monkey patching Rails config. 2020-06-05 09:05:19 +08:00
36841210fb Move UploadSecurity public types into PUBLIC_TYPES constant
* this is so plugins can extend the public types if required
2020-06-05 10:56:57 +10:00
2d880b42a3 UX: Add simple-list setting type (#9970) 2020-06-04 10:44:54 -04:00
76af25f753 FEATURE: Support append when bulk tagging via rake (#9978)
New argument option for the `tags:bulk_tag_category` rake task.
Backwards compatible - defaults to false (no append) if no argument given.
2020-06-04 09:33:48 -04:00
248db310d3 DEV: Fix lint. 2020-06-04 17:25:00 +08:00
aaece34e8b DEV: Update rails_failover so that we can move middleware up the stack. 2020-06-04 17:14:13 +08:00
8e1681d356 Bump rails_failover. 2020-06-04 15:22:35 +08:00
07774894a6 DEV: notify when the rake ask is completed 2020-06-04 12:00:06 +05:30
a93d24501c FIX: base import script was not updating first_post_created_at column
FEATURE: new rake task to update first_post_created_at column

The not-equal operator (`<>`) in PostgreSQL does not compare values
with NULL. We should instead use `IS DISTINCT FROM` when comparing
values with NULL.
2020-06-04 11:26:40 +05:30
2188ccccd5 DEV: Remove initiating_user keyword arg from EmailUpdater.
The guardian contains the acting user.
2020-06-04 13:21:56 +08:00
d3c972c30c FIX: Rescue from readonly errors when looking up auth tokens.
Since this is rare, we don't want to check for
`Discourse.pg_readonly_mode?` on every request since we have to reach
for Redis. Instead, just rescue the error here.
2020-06-04 09:12:49 +08:00
81e6bc7a0f FEATURE: Add uploads:batch_migrate_from_s3 task to limit total posts migrated at once (#9933)
Allow limiting the number of migrations to do at once, both to do migrations that
have impact limited to multiple off-peak usage hours to reduce user impact from
a migration, and to allow tests that do only a very small number for test
purposes. ("Give me a ping, Vasili. One ping only, please.")
2020-06-04 09:48:11 +10:00
a89574ccb9 FIX: Inline error when converting html to markdown
Looks like some html elements like `aside` and `section` will throw an error
when checking if they are inline or not. The commit simply handles

```
Job exception: undefined method `inline?' for nil:NilClass
```

and adds a test for it.
2020-06-03 15:59:19 -06:00
f683c5d0e0 DEV: Check English locale for errors in CI
Moves the most important checks into a linter. It gets executed by Lefthook as well as the docker rake task and Github actions. Doing those checks in rspec takes too long and it produces errors when the discourse:test Docker image contains old, invalid locale files.
2020-06-03 21:54:58 +02:00
ba00cc8ec4 DEV: Move Discourse.getURL and related functions to a module (#9966)
* DEV: Move `Discourse.getURL` and related functions to a module

* DEV: Remove `Discourse.getURL` and `Discourse.getURLWithCDN`

* FIX: `get-url` is required for server side code

* DEV: Deprecate `BaseUri` too.
2020-06-03 12:45:26 -04:00
76ce814a62 DEV: Use safe_each_connection patch when preloading Rails.
This is to ensure we still boot even if there is a problem with any of
the databases.
2020-06-03 15:36:50 +08:00
57a3d4e0d2 FEATURE: whitelist theme repo mode (experimental)
In some restricted setups all JS payloads need tight control.

This setting bans admins from making changes to JS on the site and
requires all themes be whitelisted to be used.

There are edge cases we still need to work through in this mode
hence this is still not supported in production and experimental.

Use an example like this to enable:

`DISCOURSE_WHITELISTED_THEME_REPOS="https://repo.com/repo.git,https://repo.com/repo2.git"`

By default this feature is not enabled and no changes are made.

One exception is that default theme id was missing a security check
this was added for correctness.
2020-06-03 13:19:57 +10:00
062db10c52 FIX: EmailValidator needs to validate format of email. 2020-06-03 10:34:37 +08:00
4e3a84c687 FIX: If creating a flag for a watched word, include the reason 2020-06-02 11:49:02 -04:00
439db7ca1e DEV: Add REDIS_RAILS_FAILOVER env to test our new redis failover. 2020-06-02 17:24:14 +08:00
6358e79f0f FIX: catch NoMethodError during precompile_css too (#9940)
Ran into this on heroku:

    NoMethodError: undefined method `enabled?' for #<Theme:0x0000558f069e7718>
    vendor/bundle/ruby/2.6.0/gems/activemodel-6.0.1/lib/active_model/attribute_methods.rb:431:in `method_missing'
    app/models/theme.rb:155:in `block (2 levels) in transform_ids'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:85:in `each'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:85:in `each'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/query_methods.rb:260:in `select'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/query_methods.rb:260:in `select'
    app/models/theme.rb:155:in `block in transform_ids'
    app/models/theme.rb:105:in `get_set_cache'
    app/models/theme.rb:140:in `transform_ids'
    app/models/theme.rb:321:in `list_baked_fields'
    app/models/theme.rb:317:in `resolve_baked_field'
    lib/stylesheet/manager.rb:289:in `theme_digest'
2020-06-02 15:18:03 +10:00
fe9c82994d DEV: Actually disconnect from Redis connections after fork. 2020-06-02 11:40:16 +08:00
09dc5eb5ea Version bump to v2.5.0.beta6 2020-06-01 14:13:48 -04:00
389bdcf5ab DEV: Fix implementation for DiscourseRedis#exists. 2020-06-01 13:11:32 +08:00
27ad562ff5 DEV: Rubocop fix 2020-06-01 06:07:07 +02:00
7df688d108 FIX: Handle files removed between glob and mtime 2020-06-01 05:50:50 +02:00
df62407f35 DEV: Implement multiple keys support for DiscourseRedis#exists. 2020-06-01 11:20:26 +08:00
3894555b2e DEV: Prefer Discourse.redis.reconnct. 2020-06-01 10:55:53 +08:00
2b2434b82d Start Discourse in an initializer (#9930)
* DEV: To be pedantic, there is more than EMBER in there now

* DEV: Use less globals. Have `Discourse` start in an initializer

* DEV: Remove another global
2020-05-29 14:37:02 -04:00
e159fb06df FEATURE: Download remote images even for old posts (#9925)
When a post is rebaked, the admins expect it to work the same regardless of how old the post is.
2020-05-29 17:13:55 +01:00
28f46c171c FIX: Pull hotlinked images even when edited by system users (#9890)
Previously the pull hotlinked images job was skipped after system edits. This ensured that we never had an infinite loop of system-edit/pull-hotlinked/system-edit/pull-hotlinked etc.

A side effect was that edits made by system for any other reason (e.g. API, removing full quotes) would prevent pulling hotlinked images. This commit removes the system edit check, and replaces it with another method to avoid an infinite job scheduling loop.
2020-05-29 13:07:47 +01:00
7635c18a14 DEV: ensures highlightjs is correctly tested (#9923) 2020-05-29 13:05:44 +02:00