Commit Graph

146 Commits

Author SHA1 Message Date
2834705bd1 Improve base import script
* Make sure the category description is imported correctly
  (the about topic usually had the wrong excerpt).
* Allow import scripts to mark topics as closed or archived.
* Allow import scripts to store the topic's original id.
  It will be stored in topic_custom_fields as import_topic_id.
2018-03-16 22:39:06 +01:00
25ec077eca rename 'min_private_message_{post/title}_length' to 'min_personal_message_{post/title}_length' 2018-02-01 13:25:29 +05:30
bb54eb1192 Improvements to mbox importer
* store time it took to index message in DB (to find performance issues)
* ignore listserv specific files
* better examples for split_regex
* first email in mbox shouldn't contain the split string
* always lock the DB in exclusive mode
* save email within transaction
* messages can be grouped by subject and use original order (for Listserv)
* adds option to index emails without running the import
2018-01-17 12:04:57 +01:00
a93bf186c5 FIX: validation of website failed in base importer 2017-12-28 14:51:43 +01:00
94ceb06b02 fix user lookup by email in disqus import script 2017-11-16 16:26:18 +01:00
aec5de451b FIX: compare imported ids as string instead of converting to integer, same as LookupContainer 2017-10-23 14:01:57 -04:00
0d257058c9 make rubocop 🤖 happy 2017-10-23 10:21:50 +02:00
20e7e285d6 add users with invalid email addresses (#5224)
* add users with invalid email addresses

* start to add suspend user stff

* don't create a suspend_user function

* include reason for suspension
2017-10-23 10:18:44 +02:00
5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
ed5d322196 FIX: handle existing users in imports 2017-07-24 16:22:56 +05:30
d38b55e0c8 Merge pull request #4959 from farmdawgnation/minor-mbox-import-fix
Add a nil check on the connection before attempting to exec it
2017-07-21 09:45:15 +09:00
d0b027d88d FEATURE: phase 1 of supporting multiple email addresses 2017-07-20 11:22:27 +09:00
5b11391588 Add a nil check on the connection before attempting to exec it 2017-07-07 11:11:43 -04:00
e155cb6db1 FIX: JIVE API Importer
- Added the 'hierarchical=false' query parameter to retrieve comments (replies) chronologically
- Calling 'create_post' (singular) wasn't updating the LookupContainer and thus we couldn't populate the 'reply_to_post_number' column
2017-04-26 15:36:35 +02:00
0d250c3935 50% faster vBulletin 4 importer 2017-02-01 14:33:09 +01:00
f687bfb7e2 Allow to process imported content. (add additional callbacks) 2016-10-26 16:51:34 +01:00
30f2081c1b importer/base.rb: improve progress reporting 2016-09-20 11:27:45 -07:00
72d7c055f4 Lots of ugprades to the Disqus importer script 2016-08-17 15:10:46 -04:00
fccc0c93c5 Store the original username during import (#4349) 2016-07-27 22:30:15 +02:00
56d2626187 support 'read_restricted' categories in importers 2016-07-27 18:38:23 +02:00
3b119dac6d FIX: importer performance of update_tl0, and give feedback during vanilla_mysql's create_permalinks 2016-07-06 10:58:43 -04:00
b42f28d4c3 Improved mailing list import.
Now uses a SQLite database to store messages rather than JSON files
for performance and memory considerations.
2016-06-14 11:45:17 -04:00
f07b3e7df9 PERF: base importer updates user_stats with sql (mostly) 2016-05-17 18:39:04 -04:00
b716886240 update bbPress importer 2016-03-07 18:22:15 +01:00
8a486d8cea Allow importers to set empty names 2016-02-21 23:35:47 +01:00
eb166e78b6 Don't try to import invalid websites 2016-02-21 17:38:04 +01:00
08e10c2a9d FEATURE: Zoho importer 2015-12-04 15:06:46 -05:00
26006f12a2 Increased 'error resistance' in base importer class 2015-12-01 15:38:21 +01:00
6c6d406e63 Small fixes to import base. 2015-10-29 16:47:27 -04:00
6238a43f93 Fix base and vBulletin importer 2015-10-21 19:07:31 +02:00
Sam
606aeb9d55 improvements to importer
- improve perf of test for existing posts
- always use a system guardian when importing posts
- for lithuim importer requery raw (transform is not repeatable)
2015-10-16 10:41:21 +11:00
29d27ec2ef FIX: Prevent cast error when import_id contains strings and numbers
Some importers insert numbers and strings as import_id into the same
custom_fields table. This change prevents cast errors during imports.
2015-09-30 16:38:06 +02:00
821124f3fd FEATURE: Skip batches if all records exist
Update all import scripts to take advantage of all_records_exist?
2015-09-21 17:13:02 -07:00
200ee15991 FIX: Importer not handling usernames correctly 2015-09-21 16:27:47 -07:00
8c03dd16af Adds support for importing password hashes used by "migratepassword" plugin
Adds setting to phpBB3 importer for importing passwords (default: off)
Plugin: https://github.com/discoursehosting/discourse-migratepassword
2015-08-20 22:22:47 +02:00
bb93a345eb UX: Use smaller messages for moderator actions. 2015-07-27 10:09:21 -04:00
2dd01c61b0 Improves the base importer
- Move some methods into their own classes in order to make it easier
  to reuse them outside of classes extending the base importer. For
  compatibility reasons the old methods are still in the base importer
  and delegate to the new objects. The following methods and hashes were
  extracted:
    - all the lookup maps for existing and imported data
    - all the methods used for uploads and attachments
- No need to store failed users and groups. This information wasn't
  used anyway.
- Print progress instead of category names when importing categories.
- Allow importers to override if bbcode_to_md should be used (until now
  it always used ARGV)
- Allow importers to add additional site settings that automatically get
  restored after the importer finishes.
- Show how many posts and messages are imported per minute. This should
  help detecting when the import is slowing down and needs to be
  restarted.
- Use max_image_width and max_image_height from settings instead of
  hard-coded values for uploaded images.
2015-07-16 15:28:42 +02:00
Sam
bfd28ca9a9 ignore deleted ... too much trouble 2015-07-16 15:11:28 +10:00
dcc4267490 FIX: base script import to handle category color/text color 2015-05-19 12:40:35 +02:00
e8527c5e3e it's ok if import script can't lower trust level of an admin user 2015-04-17 11:34:21 -04:00
78f5fe190a FIX: base importer should warn when there's an error in the user_profile 2015-04-17 16:31:46 +02:00
f8080c0138 add a spinner for importers 2015-03-30 12:29:48 -04:00
b1d78a4fd5 FIX: use batches for speed, and show progress from importers' update_tl0 2015-03-20 17:05:14 -04:00
5084e2bdf1 Merge pull request #3284 from techAPJ/patch-1
FEATURE: new site setting min_first_post_length
2015-03-19 19:55:27 +01:00
b706307ac7 FEATURE: new site setting min_first_post_length 2015-03-20 00:20:38 +05:30
9913dfa6c6 Imported users shouldn't get an email immediately after the import
This sets the last_emailed_at for imported users to now which postpones the sending of digest emails for the configured amount of time. Therefore, the first digest email gets sent after 1 week (the default value of default_digest_email_frequency).
As suggested in https://meta.discourse.org/t/dont-trigger-digests-on-migration/26345/7
2015-03-18 21:02:40 +01:00
53591ade00 Adds a few enhancements to the base importer
- Allows importing globally pinned topics
- Doesn't restore the original value of a SiteSetting anymore if an importer changed its value
- Allows all file extensions during the import
- Adds the ability to import bookmarks
2015-03-13 21:24:11 +01:00
b5426763e4 FIX: Importers should allow categories with existing name if the parent category is different
This changes the content of `@categories_lookup` from `Category` objects
to IDs since the category names aren't needed anymore. The lookup
method has been renamed too.
2015-03-12 21:15:02 +01:00
6030fe1f9d Added location.present? test to user profile updating. 2015-03-07 08:42:42 -06:00
de167165e3 FIX: don't change default_digest_email_frequency during import 2015-02-20 10:30:30 -05:00