Commit Graph

129 Commits

Author SHA1 Message Date
869f9b20a2 PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
867bc3b48e FIX: Change base importer to create new Bookmark records (#9603)
Also add a spec and fixture with a mock importer that we can use to test the create_X methods of the base importer
2020-05-01 11:34:55 +10:00
d216483c53 FIX: Importing with pgbouncer failed
Checking if all records have been imported uses a temp table in PostgreSQL. This fails when pgbouncer is used unless the temp table is created inside a transaction.
2020-03-26 16:41:09 +01:00
5b2b769eb7 DEV: Ensure uploads aren't deleted during imports
Sidekiq might delete uploads if you, for some reason, create upload records before using them in posts.
2020-03-24 17:14:16 +01:00
ba1b840816 DEV: Don't deactivate suspended users during import
Otherwise a cleanup job might delete those deactivated users.
2020-03-14 00:00:36 +01:00
edc8d58ac3 FEATURE: Add site setting to disable staged user cleanup
... and disabled the cleanup during imports, otherwise a running Sidekiq might delete users before posts are created
2020-03-09 10:26:41 +01:00
0843e3e6ce FIX: add support for sub-sub-categories in base_importer
Also delegates 'post_already_imported?' and 'user_already_imported?' to the base importer.
2020-02-05 10:40:28 +01:00
0c52537f10 DEV: update rubocop to version 0.77
We like to stay as close as possible to latest with rubocop cause the cops
get better.

This update required some code changes, specifically the default is to avoid
explicit returns where implicit is done

Also this renames a few rules
2019-12-10 11:48:39 +11:00
b48ca9dee9 DEV: Simplify username validation in base importer
The `UsernameValidator` does already all the hard work. No need to do any additional checks in the import script. The checks were out-of-date anyway.
2019-10-01 20:33:09 +02:00
a61ff16740 DEV: Make attachment markdown reusable 2019-07-25 14:04:18 +02:00
c70d0c6659 Use an invalid domain for fake email addresses in importers 2019-05-30 22:22:24 +02:00
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
9be70a22cd DEV: introduce new API to look up dynamic site setting
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup

This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class

It also removes a couple of risky stubs from the auth provider test
2019-05-07 11:00:30 +10:00
b58867b6e9 FEATURE: New 'Reviewable' model to make reviewable items generic
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.

Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
2019-03-28 12:45:10 -04:00
9334d2f4f7 FEATURE: add more granular user option levels for email notifications (#7143)
Migrates email user options to a new data structure, where `email_always`, `email_direct` and `email_private_messages` are replace by

* `email_messages_level`, with options: `always`, `only_when_away` and `never` (defaults to `always`)
* `email_level`, with options: `always`, `only_when_away` and `never` (defaults to `only_when_away`)
2019-03-15 10:55:11 -04:00
941e096df4 Fix error in base import script
Follow-up to 655a08dbbda1636409b15cc5611fcd3afc63263d
2019-03-06 21:58:25 +01:00
655a08dbbd FIX: Return actual errors if PostCreator fails (#7096) 2019-03-06 21:29:37 +01:00
8d5dfe1e01 FIX: Don't import parts of the email address as name 2019-02-17 22:59:18 +01:00
1e67bcb456 PERF: bulk feature topic users & reset topic counters after an import 2019-01-17 21:48:23 +01:00
788719d271 DEV: speed up posts base imports 2019-01-04 15:30:17 +01:00
71a5369fef FIX: do not convert quote tags to markdown 2018-12-11 20:09:46 +05:30
c39a1022cc PERF: user imports would slow down the more users were imported 2018-10-22 11:14:13 +02:00
afa22a0c6f REFACTOR: more 'fake_email' to base importer 2018-10-22 11:12:40 +02:00
5f2fb0fe33 Show original options when an error happens while importing an user 2018-10-18 10:21:12 +02:00
15f657309a FEATURE: Zendesk importer that uses its API to get data 2018-08-28 10:21:39 -04:00
1390eb2957 Disable bootstrap mode before start importing 2018-07-25 12:12:26 +05:30
fb022098f6 Base importer: Calculate category colors depending on parent category 2018-06-27 20:27:11 +02:00
Sam
5f64fd0a21 DEV: remove exec_sql and replace with mini_sql
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
b4e0cddcc9 disable all outgoing emails in base importer 2018-06-07 22:49:38 +05:30
f9ab3848ed FEATURE: support disabling emails for non-staff users 2018-06-07 18:31:08 +05:30
aeb511e8ff FEATURE: SMF1 importer 2018-05-28 11:02:19 +02:00
eceeef8413 Imported categories use colors from settings instead of brown 2018-05-22 21:55:54 +02:00
6e8bf1271f increase purge site settings for imports in case jobs like clean_up_uploads cause a mess 2018-04-09 12:44:59 -04:00
2834705bd1 Improve base import script
* Make sure the category description is imported correctly
  (the about topic usually had the wrong excerpt).
* Allow import scripts to mark topics as closed or archived.
* Allow import scripts to store the topic's original id.
  It will be stored in topic_custom_fields as import_topic_id.
2018-03-16 22:39:06 +01:00
25ec077eca rename 'min_private_message_{post/title}_length' to 'min_personal_message_{post/title}_length' 2018-02-01 13:25:29 +05:30
bb54eb1192 Improvements to mbox importer
* store time it took to index message in DB (to find performance issues)
* ignore listserv specific files
* better examples for split_regex
* first email in mbox shouldn't contain the split string
* always lock the DB in exclusive mode
* save email within transaction
* messages can be grouped by subject and use original order (for Listserv)
* adds option to index emails without running the import
2018-01-17 12:04:57 +01:00
a93bf186c5 FIX: validation of website failed in base importer 2017-12-28 14:51:43 +01:00
94ceb06b02 fix user lookup by email in disqus import script 2017-11-16 16:26:18 +01:00
aec5de451b FIX: compare imported ids as string instead of converting to integer, same as LookupContainer 2017-10-23 14:01:57 -04:00
0d257058c9 make rubocop 🤖 happy 2017-10-23 10:21:50 +02:00
20e7e285d6 add users with invalid email addresses (#5224)
* add users with invalid email addresses

* start to add suspend user stff

* don't create a suspend_user function

* include reason for suspension
2017-10-23 10:18:44 +02:00
5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
ed5d322196 FIX: handle existing users in imports 2017-07-24 16:22:56 +05:30
d38b55e0c8 Merge pull request #4959 from farmdawgnation/minor-mbox-import-fix
Add a nil check on the connection before attempting to exec it
2017-07-21 09:45:15 +09:00
d0b027d88d FEATURE: phase 1 of supporting multiple email addresses 2017-07-20 11:22:27 +09:00
5b11391588 Add a nil check on the connection before attempting to exec it 2017-07-07 11:11:43 -04:00
e155cb6db1 FIX: JIVE API Importer
- Added the 'hierarchical=false' query parameter to retrieve comments (replies) chronologically
- Calling 'create_post' (singular) wasn't updating the LookupContainer and thus we couldn't populate the 'reply_to_post_number' column
2017-04-26 15:36:35 +02:00
0d250c3935 50% faster vBulletin 4 importer 2017-02-01 14:33:09 +01:00
f687bfb7e2 Allow to process imported content. (add additional callbacks) 2016-10-26 16:51:34 +01:00
30f2081c1b importer/base.rb: improve progress reporting 2016-09-20 11:27:45 -07:00