Currently, the first two rows returned by `DiscourseDB#query_array` are
silently dropped during the column size check in
`DiscourseDB#load_mapping`. This happens because the rows object, while
an enumerator, isn't fully compliant, it doesn't rewind during
introspection. As a result, calls like `#first`, `#peek`, or `#any?`
advance the iterator.
Ideally, we’d fix this by updating the `query_array` enumeration
implementation. However, customizing the enumerator to be fully
compliant would likely introduce unnecessary perf overhead for all use
cases. So, this fix works around that limitation by building the map a
little differently.
Currently, if a batch "copy" of an import step results in all rows being
skipped, the `after_commit_of_skipped_rows` callback is never triggered.
This happens because the callback is nested inside a block that only
runs when at least one row is inserted.
This change ensures the DB copy operation returns both inserted and
skipped rows, allowing the caller to respond appropriately in either
case.
---------
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
* It only imports users and emails so far
* It stores mapped IDs and usernames in a SQLite DB. In the future, we might want to copy those into the Discourse DB at the end of a migration.
* The importer is split into steps which can mostly be configured with a simple DSL
* Data that needs to be shared between steps can be stored in an instance of the `SharedData` class
* Steps are automatically sorted via their defined dependencies before they are executed
* Common logic for finding unique names (username, group name) is extracted into a helper class
* If possible, steps try to avoid loading already imported data (via `mapping.ids` table)
* And steps should select the `discourse_id` instead of the `original_id` of mapped IDs via SQL
* Splits the existing script into multiple classes
* Adds command for generating IntermediateDB schema (`migrations/bin/cli schema generate`)
* Changes the syntax of the IntermediateDB schema config
* Adds validation for the schema config
* It uses YAML schema aka JSON schema to validate the config file
* It generates the SQL schema file and Ruby classes for storing data in the IntermediateDB
* Moves existing files around. All essential scripts are in `migrations/bin`, and non-essential scripts like benchmarks are in `migrations/scripts`
* Dependabot configuration for migrations-tooling (disabled for now)
* Updates test configuration for migrations-tooling
* Shorter configuration for intermediate DB for now. We will add the rest table by table.
* Adds a couple of benchmark scripts
* RSpec setup especially for migrations-tooling and the first tests
* Adds sorting/formatting to the `generate_schema` script
