Commit Graph

989 Commits

Author SHA1 Message Date
5011435ec7 DEV: do not correct sha when correctly uploads 2020-08-13 11:52:57 +10:00
f48fa30ecd DEV: fix_missing_s3 attempts to re-download if missing
unverified uploads get re-downloaded and corrected if they exist in the task
2020-08-13 11:22:14 +10:00
451b9b245f DEV: Rename new upload rake tasks
These tasks are s3-specific, so update the names to make that very clear
2020-08-12 23:26:13 +01:00
ec173a72d9 DEV: add more counts to analyze missing 2020-08-12 17:28:58 +10:00
97fe68e980 DEV: look for avatars when analyzing missing uploads 2020-08-12 14:04:21 +10:00
990cd8c2e4 FEATURE: introduce tasks for dealing with legacy broken uploads
`rake uploads:analyze_missing` can be used get rich information regarding
uploads missing from s3 (where verified is false)

`rake uploads:fix_missing` is a work in progress task for automatically
correcting certain historic issues. At the moment it simply rebakes all
posts with missing uploads, but it will improve over time
2020-08-12 13:33:04 +10:00
abebc4e05e DEV: Use verbose mode when running tests in CI 2020-08-07 17:04:55 +01:00
cd4f251891 FEATURE: Poll breakdown 2.0 (#10345)
The poll breakdown modal replaces the grouped pie charts feature.

Includes:

* MODAL: Untangle `onSelectPanel`
Previously modal-tab component would call on click the onSelectPanel callback with itself (modal-tab) as `this` which severely limited its usefulness. Now showModal binds the callback to its controller.

"The PR includes a fix/change to d-modal (b7f6ec6) that hasn't been extracted to a separate PR because it's not currently possible to test a change like this in abstract, i.e. with dynamically created controllers/components in tests. The percentage/count toggle test for the poll breakdown feature is essentially a test for that d-modal modification."
2020-08-06 17:57:06 +02:00
850d5696f2 FIX: Disable concurrent migration for multisite:migrate.
Rails migration is not thread safe.
2020-08-05 15:12:56 +08:00
1d281353b6 FIX: Don't run seeds if multisite migration fails. 2020-08-05 15:12:56 +08:00
e8328968bb DEV: allow qunit:test to run concurrently
Previously we were fighting on pid file which makes it hard to run multiple
2020-08-05 16:57:12 +10:00
ef92b72ff6 DEV: ensure 'posts:reorder_posts' rake task work on single topic
when the topic_id argument is provided.
2020-08-04 14:57:49 +02:00
712ab33ff8 DEV: fix 'posts:reorder' rake task
Running the reorder rake task was triggering the following error:

PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "post_timings_unique"

I re-worked the queries and refactored to use the same couple of queries for all similar tables/columns.
2020-08-04 11:07:07 +02:00
e0d9232259 FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
5e5973cabe FIX: Our test build of highlight.js was broken
Some definitions rely on others, in particular the c/cpp/c-like ones,
and we were appending the bundle of all files in the folder.

Instead for testing I've limited us to just three definitions. This has
the benefit of being a lot smaller to download/parse in test mode too.
2020-07-20 15:34:24 -04:00
9889b7277f FIX: Silence route-recognizer source map errors in development mode 2020-07-15 15:42:04 -04:00
e8bf304f05 FIX: Add popper sourcemap 2020-07-15 15:29:12 -04:00
339549d14a Support plugin and Theme compatibility version manifests (#9995)
Adds a new rake task `plugin:checkout_compatible_all` and
`plugin:checkout_compatible[plugin-name]` that check out compatible plugin
versions.

Supports a .discourse-compatibility file in the root of plugins and themes that
list out a plugin's compatibility with certain discourse versions:

eg: .discourse-compatibility
```
2.5.0.beta6: some-git-hash
2.4.4.beta4: some-git-tag
2.2.0: git-reference
```

This ensures older Discourse installs are able to find and install older
versions of plugins without intervention, through the manifest only.

It iterates through the versions in descending order. If the current Discourse
version matches an item in the manifest, it checks out the listed plugin target.
If the Discourse version is greater than an item in the manifest, it checks out
the next highest version listed in the manifest.

If no versions match, it makes no change.
2020-07-06 14:48:00 -07:00
69803599a9 DEV: Refactor seed data filter
Added a small helper class to for seed data because we need to add the
same filter to multisite:migrate as we have in db:migrate. Having this
filter in both places means we can get rid of the SKIP_SEED flag.
2020-06-26 14:36:50 -06:00
c16ad39f8e DEV: Run seeds irregardless of post deploy migration flag.
Follow up to 01937b2d
2020-06-26 11:04:34 +08:00
01937b2de2 Revert "FIX: Seed needs to run before optimizing site icons."
This reverts commit 715ddf38618555c7ba798f8526f85f79a7a5d365.
2020-06-26 11:03:47 +08:00
715ddf3861 FIX: Seed needs to run before optimizing site icons. 2020-06-26 08:58:53 +08:00
01b6349a67 DEV: Add skip seed flag (#10116)
* add a flag to skip seed

* only seed when running post deployment migrations
2020-06-25 10:14:58 -06:00
0384b6d910 FIX: multisite:migrate failing to properly seed data. 2020-06-23 09:10:02 +08:00
2e1efbde52 FIX: Pass local scope variable to inner function 2020-06-22 19:23:59 +03:00
e29afa200a FIX: Cleanup migrations with timestamps in the future
A future-dated migration was accidently introduced by me in 45c399f0. This was removed in b9762afc, but other migrations had already been generated based on its incorrect date. This commit removes the offending data in the schema_migrations table, and corrects the version in the published_pages migration.

This commit also adds a check to db:migrate which raises an error when invalid migration timestamps are used.
2020-06-17 15:58:22 +01:00
45eb97c202 FIX: Thread safety issues with multisite:migrate and SeedFu. 2020-06-17 16:15:43 +08:00
669c940ec3 Revert "DEV: Remove the remaining ENV["TRAVIS"] usage (#10041)"
This reverts commit 78aff841e3215c875e79692fbeaf8d18c4514006.

See https://review.discourse.org/t/dev-remove-the-remaining-env-travis-usage-10041/12737/4?u=cvx
2020-06-16 19:42:00 +02:00
78aff841e3 DEV: Remove the remaining ENV["TRAVIS"] usage (#10041) 2020-06-16 17:41:15 +02:00
a2713578dd DEV: Allow plugins to exclude seed data
This allows plugins to specify if they would like to filter out any seed
data files from running during migrations.
2020-06-15 15:30:25 -06:00
5f3fd23fa8 DEV: Add stub flush_sw task to prevent rebuild errors (#10021) 2020-06-10 13:07:37 -07:00
d4caf69ed7 DEV: makes SKIP_INSTALL_PLUGINS called last and global (#9990) 2020-06-05 17:59:23 +02:00
1c48853ede DEV: allow skipping of a single plugin when installing
SKIP_INSTALL_PLUGINS can be used to tell plugin:install_all_official
to skip a plugin.

Comma seperated list
2020-06-05 18:26:25 +10:00
76af25f753 FEATURE: Support append when bulk tagging via rake (#9978)
New argument option for the `tags:bulk_tag_category` rake task.
Backwards compatible - defaults to false (no append) if no argument given.
2020-06-04 09:33:48 -04:00
07774894a6 DEV: notify when the rake ask is completed 2020-06-04 12:00:06 +05:30
a93d24501c FIX: base import script was not updating first_post_created_at column
FEATURE: new rake task to update first_post_created_at column

The not-equal operator (`<>`) in PostgreSQL does not compare values
with NULL. We should instead use `IS DISTINCT FROM` when comparing
values with NULL.
2020-06-04 11:26:40 +05:30
81e6bc7a0f FEATURE: Add uploads:batch_migrate_from_s3 task to limit total posts migrated at once (#9933)
Allow limiting the number of migrations to do at once, both to do migrations that
have impact limited to multiple off-peak usage hours to reduce user impact from
a migration, and to allow tests that do only a very small number for test
purposes. ("Give me a ping, Vasili. One ping only, please.")
2020-06-04 09:48:11 +10:00
f683c5d0e0 DEV: Check English locale for errors in CI
Moves the most important checks into a linter. It gets executed by Lefthook as well as the docker rake task and Github actions. Doing those checks in rspec takes too long and it produces errors when the discourse:test Docker image contains old, invalid locale files.
2020-06-03 21:54:58 +02:00
57a3d4e0d2 FEATURE: whitelist theme repo mode (experimental)
In some restricted setups all JS payloads need tight control.

This setting bans admins from making changes to JS on the site and
requires all themes be whitelisted to be used.

There are edge cases we still need to work through in this mode
hence this is still not supported in production and experimental.

Use an example like this to enable:

`DISCOURSE_WHITELISTED_THEME_REPOS="https://repo.com/repo.git,https://repo.com/repo2.git"`

By default this feature is not enabled and no changes are made.

One exception is that default theme id was missing a security check
this was added for correctness.
2020-06-03 13:19:57 +10:00
6358e79f0f FIX: catch NoMethodError during precompile_css too (#9940)
Ran into this on heroku:

    NoMethodError: undefined method `enabled?' for #<Theme:0x0000558f069e7718>
    vendor/bundle/ruby/2.6.0/gems/activemodel-6.0.1/lib/active_model/attribute_methods.rb:431:in `method_missing'
    app/models/theme.rb:155:in `block (2 levels) in transform_ids'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:85:in `each'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:85:in `each'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/query_methods.rb:260:in `select'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/query_methods.rb:260:in `select'
    app/models/theme.rb:155:in `block in transform_ids'
    app/models/theme.rb:105:in `get_set_cache'
    app/models/theme.rb:140:in `transform_ids'
    app/models/theme.rb:321:in `list_baked_fields'
    app/models/theme.rb:317:in `resolve_baked_field'
    lib/stylesheet/manager.rb:289:in `theme_digest'
2020-06-02 15:18:03 +10:00
7635c18a14 DEV: ensures highlightjs is correctly tested (#9923) 2020-05-29 13:05:44 +02:00
d9a02d1336 Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eeed56b321daf18ee6bbfe681a51d1bf4.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6fd81a2a34aff6144bd36b9ac459964a, reversing
changes made to 2660c2e21d84bea667e1ea339f91cda352328062.
2020-05-22 20:25:56 -07:00
779dc30d2e FIX: Join bookmarks migration on users to avoid missing user records
* the post_actions table has no FK to users, so if a user has been
  deleted we may end up with dangling post_action records, which then
  interferes with the bookmarks migration because bookmarks DO have
  an FK to users
2020-05-19 16:31:05 +10:00
869f9b20a2 PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
42b300b9be DEV: Skip I18n in plugins until they can be updated 2020-05-14 10:40:53 -04:00
1c0c425f38 DEV: Detect correct data dir using ENV 2020-05-12 13:10:57 -03:00
8311374a5a DEV: Replace version.js.erb with pre generated file 2020-05-11 15:43:09 -04:00
d2b16a7618 DEV: Generate emoji/data on demand (#9744)
These emoji rarely change and can be re-generated when we make changes
to them.
2020-05-11 15:29:46 -04:00
34e5f807ec DEV: User UPPER_CASE for constants 2020-05-08 15:50:55 -04:00