* It only imports users and emails so far
* It stores mapped IDs and usernames in a SQLite DB. In the future, we might want to copy those into the Discourse DB at the end of a migration.
* The importer is split into steps which can mostly be configured with a simple DSL
* Data that needs to be shared between steps can be stored in an instance of the `SharedData` class
* Steps are automatically sorted via their defined dependencies before they are executed
* Common logic for finding unique names (username, group name) is extracted into a helper class
* If possible, steps try to avoid loading already imported data (via `mapping.ids` table)
* And steps should select the `discourse_id` instead of the `original_id` of mapped IDs via SQL
* Splits the existing script into multiple classes
* Adds command for generating IntermediateDB schema (`migrations/bin/cli schema generate`)
* Changes the syntax of the IntermediateDB schema config
* Adds validation for the schema config
* It uses YAML schema aka JSON schema to validate the config file
* It generates the SQL schema file and Ruby classes for storing data in the IntermediateDB
The previous approach of splitting Thor commands into multiple files caused problems when the same method name was used in multiple commands.
This also loads the Rails environment only for commands that need it. That makes the CLI boot faster for most commands or when the help should be shown. That's also why we can't use `Rails.root` in the CLI.
* DEV: Implement uploads command entrypoint
- Setup Thor UploadsCommand for CLI
- First pass at modularizing various parts of the exising `uploads_import` script
* DEV: First attempt at modularizing missing uploads fixer task
Move missing upload fix to a dedicated uploads task implementation unit
* DEV: First attempt at modularizing missing uploads uploader task
Move uploader to a dedicated uploads task implementation unit
* DEV: First attempt at modularizing missing uploads optimizer task
Move optimizer to a dedicated uploads task implementation unit
* DEV: Various follow up fixes to get optimization working
- Start threads early
- Improve "log" message formatting
- Add missing `copy_to_tempfile` method on "uploader" task
* DEV: Refactor a bit more
Deduplicate and move most of threading premitives to base task as-is
* DEV: Remove redundant condition in uploads db migration
* DEV: More deduplication
Move task retry logic to base class and tidy up other implementation
details carried over from the existing script
* Updates GitHub Actions
* Switches from `bundler/inline` to an optional group in the `Gemfile` because the previous solution didn't work well with rspec
* Adds the converter framework and tests
* Allows loading private converters (see README)
* Switches from multiple CLI tools to a single CLI
* Makes DB connections reusable and adds a new abstraction for the `IntermediateDB`
* `IntermediateDB` acts as an interface for IPC calls when a converter steps runs in parallel (forks). Only the main process writes to the DB.
* Includes a simple example implementation of a converter for now.
