Commit Graph

38 Commits

Author SHA1 Message Date
c1517e428e DEV: Add vBulletin5 bulk importer (#12904)
This is a pretty straightforward bulk importer, just tailored to the vBulletin 5 database structure.

Also made a few minor improvements to the base importer -- should be self explanatory in the code.
2021-04-30 11:03:33 -05:00
a85d5edbf1 DEV: set digest_attempted_at during migrations (#11369) 2020-12-14 10:58:14 +11:00
93ff54e184 FIX: improvements for vanilla bulk import (#10212)
Adjustments to the base:
1. PG connection doesn't require host - it was broken on import droplet
2. Drop `topic_reply_count` - it was removed here - https://github.com/discourse/discourse/blob/master/db/post_migrate/20200513185052_drop_topic_reply_count.rb
3. Error with `backtrace.join("\n")` -> `e.backtrace.join("\n")`
4. Correctly link the user and avatar to quote block

Adjustments to vanilla:
1. Top-level Vanilla categories are valid categories
2. Posts have `format` column which should be used to decide if the format is HTML or Markdown
3. Remove no UTF8 characters
4. Remove not supported HTML elements like `font` `span` `sub` `u`
2020-07-14 15:58:27 +10:00
47a1157458 DEV: various bugfixes in bulk importer 2020-06-19 17:53:06 +02:00
5143309014 DEV: ensure values are converted to integers in bulk importer 2020-06-18 17:42:14 +02:00
823b940b9d PERF: improve loading of indexes in bulk import
Similar strategy as for c52191d in which we stream the results from the database into
an automatically growing array instead of using a hash.
2020-06-18 16:32:27 +02:00
c52191d49e PERF: improve loading a imported_ids in bulk imports
- Stream the queries that load the imported_ids
- Use an array instead of a hash for keeping the mapping between imported_ids and new ids
- Ensure we always treat the imported_ids as integers instead of strings
2020-06-16 19:55:08 +02:00
d0d5a138c3 DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
0c52537f10 DEV: update rubocop to version 0.77
We like to stay as close as possible to latest with rubocop cause the cops
get better.

This update required some code changes, specifically the default is to avoid
explicit returns where implicit is done

Also this renames a few rules
2019-12-10 11:48:39 +11:00
067696df8f DEV: Apply Rubocop redundant return style 2019-11-14 15:10:51 -05:00
b788948985 FEATURE: English locale with international date formats
Makes en_US the new default locale
2019-05-20 13:47:20 +02:00
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
a20f58554b IMPORT: create category definitions in import:ensure_consistency task 2019-04-11 12:06:37 +05:30
714f6cde79 FIX: Remove duplicate definition of create_categories. 2019-03-04 10:32:09 +02:00
71a5369fef FIX: do not convert quote tags to markdown 2018-12-11 20:09:46 +05:30
735a48415d FEATURE: option to use ruby-bbcode-to-md in bulk import script
ruby-bbcode-to-md provides better bbcode to markdown conversion
2018-12-10 10:28:07 +05:30
0365d50797 Improve vBulletin bulk import script to support table prefix.
Improve base bulk import script to convert list tags to ul/li.
2018-12-10 10:10:44 +05:30
3c9c95ac83 Update Rubocop to 0.60 2018-12-04 10:48:16 +01:00
9248ad1905 DEV: Enable Style/SingleLineMethods and Style/Semicolon in Rubocop (#6717) 2018-12-04 11:48:13 +08:00
a0f0bac752 Add a comment to run the 'import:ensure_consistency' rake task after a bulk import 2018-11-21 16:28:35 +01:00
0e04e3990e Improve Vanilla bulk import script 2018-08-16 22:00:26 +05:30
7aa93b84c1 FIX: bulk importers shouldn't insert rows with id less than 1 2018-03-09 14:26:18 -05:00
200c6673f1 FIX: bulk importers wiping all email addresses without warning or errors 2018-03-08 23:36:39 -05:00
1093dacc03 FIX: bulk importers need to create category description topics 2018-03-07 12:10:22 -05:00
0edd386b48 FEATURE: Vanilla bulk importer 2018-02-02 16:28:51 -05:00
a224459960 bulk importer shouldn't try to update primary key sequences to -1 2018-01-19 15:01:00 -05:00
90c14106fa Enhance BulkImport pre_cook (#5015)
* Enhance BulkImport pre_cook

* BulkImport: Trim <br> at begining and ending [quote][quote/]
2017-09-04 11:04:54 +02:00
2d909f7894 new phpBB PostgreSQL bulk import script 2017-08-03 21:21:58 +05:30
bac21d317b Bulk import likes from vBulletin thanks (#5014) 2017-08-01 10:01:45 +02:00
7836b064f4 [FIX] invalid byte sequence in UTF-8 (#5003)
Invalid encoding fixed prior to empty check
2017-07-31 15:34:11 -04:00
0daa177805 Enhance bulk import scripts (#5010)
* Enhance bulk import scripts

* Fix: restore running statement of BulkImport::VBulletin
2017-07-31 10:56:57 +02:00
5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
d89d279416 Update UserEmail primary key sequence when performing bulk import. 2017-07-25 19:15:22 +05:30
5bba959cd5 FIX: vBulletin bulk importer: emails and stats 2017-07-24 19:49:22 +07:00
57d6a5dc9c FIX: vBulletin bulk importer 2017-07-24 14:22:00 +02:00
d0b027d88d FEATURE: phase 1 of supporting multiple email addresses 2017-07-20 11:22:27 +09:00
8048aeac22 FIX: BBCode converter in bulk importers 2017-05-03 16:56:52 +02:00
58951e232f base bulk importer 2017-04-24 23:00:41 +02:00