Currently, the first two rows returned by `DiscourseDB#query_array` are
silently dropped during the column size check in
`DiscourseDB#load_mapping`. This happens because the rows object, while
an enumerator, isn't fully compliant, it doesn't rewind during
introspection. As a result, calls like `#first`, `#peek`, or `#any?`
advance the iterator.
Ideally, we’d fix this by updating the `query_array` enumeration
implementation. However, customizing the enumerator to be fully
compliant would likely introduce unnecessary perf overhead for all use
cases. So, this fix works around that limitation by building the map a
little differently.
Currently, if a batch "copy" of an import step results in all rows being
skipped, the `after_commit_of_skipped_rows` callback is never triggered.
This happens because the callback is nested inside a block that only
runs when at least one row is inserted.
This change ensures the DB copy operation returns both inserted and
skipped rows, allowing the caller to respond appropriately in either
case.
---------
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
By default, rails makes timestamp columns (`created_at` and
`updated_at`) non-nullable, we also have some required core and plugins
columns we wouldn't necessarily want to enforce in the intermediate DB
schema. It'll be better to set the default values for these during
import instead of enforcing these at the converter level.
This change adds support for globally modifying a column’s `nullable`
state, defaulting all `created_at` columns to be `nullable` while
allowing for table level overrides.
---------
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
* It only imports users and emails so far
* It stores mapped IDs and usernames in a SQLite DB. In the future, we might want to copy those into the Discourse DB at the end of a migration.
* The importer is split into steps which can mostly be configured with a simple DSL
* Data that needs to be shared between steps can be stored in an instance of the `SharedData` class
* Steps are automatically sorted via their defined dependencies before they are executed
* Common logic for finding unique names (username, group name) is extracted into a helper class
* If possible, steps try to avoid loading already imported data (via `mapping.ids` table)
* And steps should select the `discourse_id` instead of the `original_id` of mapped IDs via SQL
* Splits the existing script into multiple classes
* Adds command for generating IntermediateDB schema (`migrations/bin/cli schema generate`)
* Changes the syntax of the IntermediateDB schema config
* Adds validation for the schema config
* It uses YAML schema aka JSON schema to validate the config file
* It generates the SQL schema file and Ruby classes for storing data in the IntermediateDB
* Updates GitHub Action for migrations
* Rubocop: Always `EnforcedShorthandSyntax` for hashes in the `migrations` directory
* Automatically load all available converter steps
* Enable YJIT at runtime, if available
* Progressbar shows skipped records and other small improvements
* Remove unused `report_progress_in_percent` option from step
* Remove `use_custom_progress_increment` option from the step because we can figure it out by looking at the progress
* Introduce `StepTracker` to for logging warnings and errors and tracking step progress
* Make it easier to log warnings and errors in all methods of `Step` without the need to pass around a `stats` object
The previous approach of splitting Thor commands into multiple files caused problems when the same method name was used in multiple commands.
This also loads the Rails environment only for commands that need it. That makes the CLI boot faster for most commands or when the help should be shown. That's also why we can't use `Rails.root` in the CLI.
* DEV: Implement uploads command entrypoint
- Setup Thor UploadsCommand for CLI
- First pass at modularizing various parts of the exising `uploads_import` script
* DEV: First attempt at modularizing missing uploads fixer task
Move missing upload fix to a dedicated uploads task implementation unit
* DEV: First attempt at modularizing missing uploads uploader task
Move uploader to a dedicated uploads task implementation unit
* DEV: First attempt at modularizing missing uploads optimizer task
Move optimizer to a dedicated uploads task implementation unit
* DEV: Various follow up fixes to get optimization working
- Start threads early
- Improve "log" message formatting
- Add missing `copy_to_tempfile` method on "uploader" task
* DEV: Refactor a bit more
Deduplicate and move most of threading premitives to base task as-is
* DEV: Remove redundant condition in uploads db migration
* DEV: More deduplication
Move task retry logic to base class and tidy up other implementation
details carried over from the existing script
* Updates GitHub Actions
* Switches from `bundler/inline` to an optional group in the `Gemfile` because the previous solution didn't work well with rspec
* Adds the converter framework and tests
* Allows loading private converters (see README)
* Switches from multiple CLI tools to a single CLI
* Makes DB connections reusable and adds a new abstraction for the `IntermediateDB`
* `IntermediateDB` acts as an interface for IPC calls when a converter steps runs in parallel (forks). Only the main process writes to the DB.
* Includes a simple example implementation of a converter for now.
* Moves existing files around. All essential scripts are in `migrations/bin`, and non-essential scripts like benchmarks are in `migrations/scripts`
* Dependabot configuration for migrations-tooling (disabled for now)
* Updates test configuration for migrations-tooling
* Shorter configuration for intermediate DB for now. We will add the rest table by table.
* Adds a couple of benchmark scripts
* RSpec setup especially for migrations-tooling and the first tests
* Adds sorting/formatting to the `generate_schema` script
