Commit Graph

467 Commits

Author SHA1 Message Date
6e8bf1271f increase purge site settings for imports in case jobs like clean_up_uploads cause a mess 2018-04-09 12:44:59 -04:00
f2d00e5eff FEATURE: Use Message-ID for detecting email replies to group
Ignores the site setting "find_related_post_with_key" and always tries to honor the `In-Reply-To` and `References` header for emails sent to a group.

The senders email address must be included in the `To` or `CC` header of a previous email sent to the group and the `Message-ID` of that email must be included in the current email's `In-Reply-To` or `References` header.
2018-04-05 11:00:38 +02:00
8d283e78e7 Fix broken queries in Drupal 6 migration. 2018-03-31 17:53:40 +02:00
cf9965fd54 Add rake task for reordering posts by creation_date 2018-03-29 14:42:41 +02:00
a0d5e243fd Improve the import script for "Get Satisfaction"
* Works with the current column layout exported as Excel file
* Tries to fix invalid CSV when it wasn't exported from Excel
* Imports categories
* Imports topics into the correct category
* Allows skipping archived topics
* Allows skipping private topics
* Makes use of the latest features from the base importer
* Some minor fixes and documentation updates
2018-03-19 14:58:17 +01:00
dad1e4b17d don't import data only available in a plugin in phpBB 3 importer 2018-03-17 00:36:12 +01:00
2834705bd1 Improve base import script
* Make sure the category description is imported correctly
  (the about topic usually had the wrong excerpt).
* Allow import scripts to mark topics as closed or archived.
* Allow import scripts to store the topic's original id.
  It will be stored in topic_custom_fields as import_topic_id.
2018-03-16 22:39:06 +01:00
bb6ac51607 Update PHPBB3 importer
- Extract name from profile_fields_data table
- Use 'posts.poster_ip' to track topic views
2018-03-16 18:32:27 +01:00
9b651adadb FIX: mbox importer should ignore emails without date 2018-03-13 13:42:57 +01:00
Sam
ddcca1d5cc lint 2018-03-08 15:16:12 +11:00
4b8ff7a057 Update xenforo.rb (#5622)
#Fix for the error: xenforo.rb: 160: in `gsub!': invalid byte sequence in UTF-8 (ArgumentError)
2018-03-07 22:57:38 -05:00
07bc52dc69 import topic views in phpbb3 2018-03-07 19:27:03 +01:00
dc32ee5cbf Improvements to mbox import script
* Ignore errors during indexing and show information about the message causing the problem
* Always activate imported users if they aren't staged
2018-03-06 11:32:12 +01:00
49400337b6 Improve IPBoard3 importer
- fix encoding issues
- close topics
- don't import queued & not approved topics
- don't import deleted posts
2018-03-05 10:24:42 +01:00
ae9b1e8554 FEATURE: IP.Board 3 importer 2018-03-01 09:43:32 +01:00
82e68670bf FIX: SimplePress importer wasn't handling increment imports properly 2018-02-24 00:42:17 +01:00
Sam
720e1965e3 FEATURE: add category suppress from latest
In the past we used suppress_from_homepage, it had mixed semantics
it would remove from category list if category list was on home and
unconditionally remove from latest.

New setting explicitly only removes from latest list but leaves the
category list alond
2018-02-22 09:56:35 +11:00
479f7ed18f Ignore case when removing mailing list name from subject 2018-02-12 21:41:58 +01:00
31f5ca6180 improve simple press importer 2018-02-07 17:37:51 +01:00
25ec077eca rename 'min_private_message_{post/title}_length' to 'min_personal_message_{post/title}_length' 2018-02-01 13:25:29 +05:30
cc830f7119 sorry rubocop 2018-01-30 14:37:15 -05:00
db17c531d6 Feature: Invision Power Board Importer 2018-01-30 08:44:02 -08:00
192a0886e2 FIX: BBCode to Markdown conversion in phpBB3 importer was broken
This fixes the conversion for quotes, code blocks and lists (except for nested lists). It also discourages the usage of the ruby-bbcode-to-md gem.
2018-01-30 12:50:42 +01:00
c26db2116c FIX: phpBB3 importer imported some users as anonymous users 2018-01-30 12:50:42 +01:00
db287ad58e FIX: invalid SQL in vanilla_mysql import script 2018-01-19 11:53:07 -05:00
6500343431 FIX: mbox importer didn't detected already indexed files 2018-01-17 17:03:53 +01:00
bb54eb1192 Improvements to mbox importer
* store time it took to index message in DB (to find performance issues)
* ignore listserv specific files
* better examples for split_regex
* first email in mbox shouldn't contain the split string
* always lock the DB in exclusive mode
* save email within transaction
* messages can be grouped by subject and use original order (for Listserv)
* adds option to index emails without running the import
2018-01-17 12:04:57 +01:00
77a92e8878 Allow user staging via setting (#5468) 2018-01-04 09:17:35 +01:00
a93bf186c5 FIX: validation of website failed in base importer 2017-12-28 14:51:43 +01:00
cc46ab5bb1 smf2: deal with duplicate category names (#5435) 2017-12-27 16:56:36 +11:00
cafe69caac Refactor mbox import script 2017-12-13 22:03:31 +01:00
3190c13c22 import staged users as inactive in mbox import 2017-12-13 08:45:43 +05:30
16738cfb1b FEATURE: convert plain text emails to markdown 2017-12-06 01:47:51 +01:00
32dd1e66be improvements to the mbox import script
* ignores dot-files and empty emails
* new setting to prefer HTML over plaintext emails during import
* restore original site settings at the end of import
* elided content of HTML mails was not put inside details block
2017-11-18 17:16:44 +01:00
94ceb06b02 fix user lookup by email in disqus import script 2017-11-16 16:26:18 +01:00
06a6ddc3ba handle plaintext and HTML emails in mbox importer 2017-11-15 20:22:11 +01:00
5d8508c523 New yahoo groups importer 2017-11-09 11:07:57 +01:00
667b025d12 make rubocop happy 2017-11-07 14:20:46 -05:00
4f0bdec370 some improvements for importers (#5295)
* decode html entities within code blocks

* Only import users that actually participated in the bbpress part of Wordpress; import password hashes

* create permalinks for topics

* Better handling of [code] blocks
2017-11-07 17:50:43 +01:00
9cbb90c5ed add importer for modx forum (#5239) 2017-10-31 10:50:34 +11:00
aec5de451b FIX: compare imported ids as string instead of converting to integer, same as LookupContainer 2017-10-23 14:01:57 -04:00
0d257058c9 make rubocop 🤖 happy 2017-10-23 10:21:50 +02:00
20e7e285d6 add users with invalid email addresses (#5224)
* add users with invalid email addresses

* start to add suspend user stff

* don't create a suspend_user function

* include reason for suspension
2017-10-23 10:18:44 +02:00
6c829c24d7 escaping the subject isn't needed in the mbox importer 2017-10-19 15:25:20 +02:00
c41880ab19 Improvements to the experimental mbox importer
* Disable journaling to improve performance in Docker
* Use the email cooking method
* Store IncomingEmail in order find related posts by Message-ID
* Escape HTML in imported messages
2017-10-19 14:27:40 +02:00
e9fb2c5756 phpBB3 importer: Fix validation of site settings 2017-08-29 01:48:30 +02:00
3f8e535692 FEATURE: bbpress importer can import private messages 2017-08-21 16:39:57 -04:00
5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
32ef30cff9 phpBB3 importer: Permalink for first post of each topic was missing 2017-07-25 23:29:56 +02:00
5be52fa821 phpBB3 importer: Use permalink normalization for category links 2017-07-25 21:14:46 +02:00