Commit Graph

1021 Commits

Author SHA1 Message Date
d81359246a DEV: Be more lenient in CLI confirmation (#16290)
If someone types `yes` rather than `YES`, continue anyway.

The chance of typing `yes`, when you actually want to stop, is non-existent. The chance of typing `yes` when you meant `YES` is  high, and it's very frustrating when the script quite because you got the case wrong!
2022-03-25 20:14:41 +00:00
f3aab19829 DEV: Promote historic post_deploy migrations (#16288)
This commit promotes all post_deploy migrations which existed in Discourse v2.7.13 (timestamp <= 20210328233843)

This reduces the likelihood of issues relating to migration run order

Also fixes a couple of typos in `script/promote_migrations`
2022-03-25 15:48:20 +00:00
2fc70c5572 DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
6f6406ea03 DEV: Fix random typos (#16066) 2022-02-28 10:20:58 +08:00
5374e587a3 DEV: Add message-bus analysis script (#15979)
This will count how many messages are published per-channel and produce a table of channels ordered by 'most messages'
2022-02-18 20:21:17 +00:00
3bf3b9a4a5 DEV: pull email address validation out to a new EmailAddressValidator
We validate the *format* of email addresses in many places with a match against
a regex, often with very slightly different syntax.

Adding a separate EmailAddressValidator simplifies the code in a few spots and
feels cleaner.

Deprecated the old location in case someone is using it in a plugin.

No functionality change is in this commit.

Note: the regex used at the moment does not support using address literals, e.g.:
* localpart@[192.168.0.1]
* localpart@[2001:db8::1]
2022-02-17 21:49:22 -05:00
6394d7cddf DEV: Improve phpBB3 import script (#15956)
* Optional import of custom user fields from phpBB 3.1+
* Optional import of likes from phpBB3
  Requires the phpBB "Thanks for posts" extension
* Fix import of bookmarks from phpBB3
* Update `created_at` of existing user
* Support mapping of phpBB forums to existing Discourse categories
  This is in addition to the ability of merging phpBB forums and importing into newly created Discourse categories.
2022-02-16 13:04:31 +01:00
33d6ed60a4 DEV: Don't import year of birth (#15937)
The cakeday plugin doesn't use the year.
2022-02-14 18:10:35 +01:00
6a41ec179c FIX: Default settings for phpBB3 import were broken (#15913) 2022-02-11 18:18:54 +01:00
9e43f0303d DEV: Include DISCOURSE_REDIS_REPLICA_HOST in cache_critical_dns (#15877)
This is the replacement for DISCOURSE_REDIS_SLAVE_HOST
2022-02-09 14:41:26 +00:00
ea2fd75d10 DEV: Fix some regexes in phpBB3 import script (#15829)
1. bbcode hashes don't always have exactly 8 characters.

2. colors aren't always hex values, it can be a color string ("red", "blue", etc).

3. The closing tag of smileys doesn't always include a `:` character (the start of the regex was already right for this particular issue)
2022-02-07 16:16:46 +01:00
ed2f700440 DEV: Wait for initdb to complete in docker.rake (#15614)
On slower hardware it can take a while to init the database. If we don't wait, the `rake db:create` step will fail.
2022-01-17 17:45:39 +00:00
c5fd8c42db DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
0e87f882a7 DEV: Use discourse image for postgres in GitHub Actions (#15291)
The discourse base image already contains a postgres installation, so pulling a separate postgres image is a little wasteful. Using the copy of Postgres in the discourse image saves about 20 seconds on every GitHub actions run.

This commit sets up Postgres with a few performance-improving flags, which we were already using for the `rake docker:test` task (used on our internal CI system).
2021-12-14 17:20:06 +00:00
cfabdb72bc FIX: Ambiguous column in downsize_uploads (#14972) 2021-11-16 16:23:32 +01:00
48a08cc397 FIX: Vanilla importer fixes (#14699)
Import script was out of date
2021-10-27 14:22:37 +02:00
69f0f48dc0 DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
46d96c9feb DEV: Apply rubocop to script/import_scripts/phorum.rb (#14727)
Followup to b24002018a10e12b7bdee3a8f5218345716af6ff
2021-10-26 19:16:52 +01:00
b24002018a Update phorum.rb
Add attachment/file/upload handling to bring them in from phorum to discourse
2021-10-26 12:41:50 -04:00
451cd4ec3f DEV: Fix thor deprecation warning (#14680)
```
Deprecation warning: Thor exit with status 0 on errors. To keep this behavior, you must define `exit_on_failure?` in `DiscourseCLI`
```
2021-10-21 21:01:05 +02:00
97178cd777 FIX: phpbb import - attachments not embedded in posts (#14570) 2021-10-11 14:27:54 +02:00
a413a1e015 DEV: process image uploads in the Zendesk API import script (#14524) 2021-10-06 12:24:12 -04:00
a4d0d866aa DEV: Bulk imports should find existing users by email (#14468)
Without this change, bulk imports unconditionally create new user records even when a user with the same email address exists.
2021-09-29 00:20:06 +02:00
1656b7ed01 DEV: Make db_timestamp_mover work with tables with unique constraints (#14027)
Some tables in the database have constraints on columns with dates. Because of them, the script for moving timestamps can fail from time to time. This PR makes the script work with such tables.

In general, in PostgreSQL it is not always possible to defer constraint checks to the transaction commit (Primary Keys and Unique Constraints can be deferred, but them should be declared as DEFERRABLE to make it possible. Indices created with CREATE UNIQUE INDEX can't be deferred at all).

Since we can't defer constraint checks, I've made it work using a little hack. For example, if we need to move all timestamps by one day, the script will move timestamps by 1000 years and one day, and then return timestamps back by 1000 years. The script use this hack only for columns that have unique constraints.
2021-08-12 19:24:21 +04:00
cd9262b7d3 DEV: minor improvements in the vanilla import script. (#14026)
We're parsing the post raw based on the record format now.
2021-08-12 15:07:44 +05:30
b66674fec2 DEV: ignore the given_daily_likes table when moving timestamps on Try (#13971)
This will fix the try-reset build that failed today. Probably this going to happen again with other tables that have constraints on date columns. I'm going to modify the script to make it work without ignoring such tables. After that, the only table we're going to need to ignore will be the 2FA table.

Before I fixed that, don't hesitate to tag me if the try-reset build fail again.
2021-08-06 18:27:23 +04:00
f9aaed7020 FIX: MyBB importer exposes deleted posts (#13700)
The MyBB importer exposes posts marked as deleted in MyBB. This may lead to a privacy issue.
2021-07-22 09:55:02 +02:00
a10fd95a7e DEV: removes unused version_bump script (#13811) 2021-07-21 10:35:24 -04:00
265e32e3e2 DEV: uses main instead of master in version_bump script (#13805) 2021-07-21 10:40:35 +02:00
efc8d5f134 DEV: ignore the 2FA table when moving timestamps (#13793)
or 2FA will be broken after moving. We use this script for moving timestamps when restoring Try.
2021-07-20 15:49:20 +04:00
7fe3e4bab8 DEV: fix joining in the script for moving timestamps (#13721)
Without checking if t.table_schema = '#{@schema}' the SELECT with JOIN in the script were returning every column twice in case there is a 'backup' scheme with exactly the same tables as in the 'public scheme'
2021-07-13 17:21:15 +04:00
8a0349a01f DEV: ignore some tables when updating timestamps using db_timestamps_mover.rb (#13714)
Some tables have constraints on columns with a date which can cause problems when moving timestamps. By now I think it's enough to just ignore them.
2021-07-13 12:36:41 +04:00
696bd0bf05 DEV: add a script for moving timestamps in database (#13682)
We're going to use this script for updating timestamps on Try, but it can be used with a local database during development as well.

Usage:

Commands:
  ruby db_timestamp_updater.rb yesterday <date> move all timestamps by x days so that <date> will be moved to yesterday
  ruby db_timestamp_updater.rb 100              move all timestamps forward by 100 days
  ruby db_timestamp_updater.rb -100             move all timestamps backward by 100 days
The script moves all timestamps in the database by the same amount of days forward or backward. No need to change the script if we add a new column in the future.

The more simple solution would be just to move timestamps in several tables (topics, posts, and so on). I didn't want to go that way because it could generate additional work in the future. For example, if we add a new column with a timestamp and users can see that timestamp we'd need to add that column to the script. Or, for example, if we move a post's timestamp to the future but forget to move a timestamp of topic timer or user action it can cause weird bugs.
2021-07-09 20:04:28 +04:00
37b8ce79c9 FEATURE: Add last visit indication to topic view page. (#13471)
This PR also removes grey old unread bubble from the topic badges by
dropping `TopicUser#highest_seen_post_number`.
2021-07-05 14:17:31 +08:00
6ea4bbd2ec DEV: Prefer .pluck_first over .pluck.first (#13607) 2021-07-02 10:03:54 +08:00
88da06cba0 FIX: typo in discourse
occurences -> occurrences
2021-06-30 11:08:29 -04:00
20070f5089 DEV: Update script/promote_migrations (#13513)
Introduces the --plugins-base flag for updating plugins in a different directory. Followup to 49f39434c48c6f05ab2037f0f31c9da521401cfb
2021-06-24 13:57:23 +01:00
046a875222 DEV: Improve script/downsize_uploads.rb (#13508)
* Only shrink images that are used in Posts and no other models
* Don't save the upload if the size is the same
2021-06-24 00:09:40 +02:00
49f39434c4 DEV: Introduce script/promote_migrations tool
Post-deploy migrations exist to allow for seamless Discourse upgrades. By design, they cause migrations to run out of numerical order. This has the potential to cause some unexpected edge cases. To reduce the likelihood of these edge cases, we will promote historical post_deploy migrations to regular migrations after a full Discourse stable release cycle.

This script is intended to be run at least during every Discourse release cycle.

This means that truly seamless upgrades will not be possible between non-consecutive Discourse versions. (Upgrades will still work, but may cause some server errors for users during the upgrade)
2021-06-23 17:43:38 +01:00
5efed91128 FIX: Set random values for digest_attempted_at
Setting a random value in the interval 1 week ago ... now works better
because this spreads digest scheduling over a week because digests are
sent one week from the date of the last digest.
2021-06-22 12:05:15 +08:00
365d339985 DEV: fix Flarum import script (#13385) 2021-06-15 19:08:55 +05:30
3bb765ac92 DEV: Remove the remaining Travis code (#13255)
The second attempt at #10041 now that all our plugins use GitHub Actions CI instead.
2021-06-02 20:29:47 +02:00
4bf195d502 FEATURE: Flarum import script (#13139) 2021-05-27 02:30:50 +02:00
d0779a87bb FIX: Use max_category_nesting when importing categories (#13105)
It allowed for a parent category and a sub-category.
2021-05-26 12:40:26 +03:00
59097b207f DEV: Correct typos and spelling mistakes (#12812)
Over the years we accrued many spelling mistakes in the code base. 

This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change 

- comments
- test descriptions
- other low risk areas
2021-05-21 11:43:47 +10:00
4b7b586861 DEV: Remove unused rswag script
We no longer use this script for helping generate rswag formatted docs because we can use json schema now.
2021-05-07 09:36:55 +08:00
c1517e428e DEV: Add vBulletin5 bulk importer (#12904)
This is a pretty straightforward bulk importer, just tailored to the vBulletin 5 database structure.

Also made a few minor improvements to the base importer -- should be self explanatory in the code.
2021-04-30 11:03:33 -05:00
8b87cb07c3 DEV: Return a non-zero exit code when discourse remap is aborted (#12873)
This makes it much easier to integrate the script with external tools
2021-04-28 14:10:35 +01:00
e7892df10d FIX: handle charset=windows-1252 in mbox import script (#12832)
Co-authored-by: Loïc Dachary <loic@dachary.org>
2021-04-27 15:43:31 +02:00
5bec0e5763 fix vbulletin importer to import unreferenced attachments (#12187) 2021-04-19 21:05:16 +02:00