This also enhances the converter step, which stores the Gravatar if it is
set and used. It also fixes the import of uploads which didn't use the
correct user ID.
---------
Co-authored-by: Selase Krakani <849886+s3lase@users.noreply.github.com>
By default, rails makes timestamp columns (`created_at` and
`updated_at`) non-nullable, we also have some required core and plugins
columns we wouldn't necessarily want to enforce in the intermediate DB
schema. It'll be better to set the default values for these during
import instead of enforcing these at the converter level.
This change adds support for globally modifying a column’s `nullable`
state, defaulting all `created_at` columns to be `nullable` while
allowing for table level overrides.
---------
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
* It only imports users and emails so far
* It stores mapped IDs and usernames in a SQLite DB. In the future, we might want to copy those into the Discourse DB at the end of a migration.
* The importer is split into steps which can mostly be configured with a simple DSL
* Data that needs to be shared between steps can be stored in an instance of the `SharedData` class
* Steps are automatically sorted via their defined dependencies before they are executed
* Common logic for finding unique names (username, group name) is extracted into a helper class
* If possible, steps try to avoid loading already imported data (via `mapping.ids` table)
* And steps should select the `discourse_id` instead of the `original_id` of mapped IDs via SQL
* Splits the existing script into multiple classes
* Adds command for generating IntermediateDB schema (`migrations/bin/cli schema generate`)
* Changes the syntax of the IntermediateDB schema config
* Adds validation for the schema config
* It uses YAML schema aka JSON schema to validate the config file
* It generates the SQL schema file and Ruby classes for storing data in the IntermediateDB
* Updates GitHub Action for migrations
* Rubocop: Always `EnforcedShorthandSyntax` for hashes in the `migrations` directory
* Automatically load all available converter steps
* Enable YJIT at runtime, if available
* Progressbar shows skipped records and other small improvements
* Remove unused `report_progress_in_percent` option from step
* Remove `use_custom_progress_increment` option from the step because we can figure it out by looking at the progress
* Introduce `StepTracker` to for logging warnings and errors and tracking step progress
* Make it easier to log warnings and errors in all methods of `Step` without the need to pass around a `stats` object
* DEV: Implement uploads command entrypoint
- Setup Thor UploadsCommand for CLI
- First pass at modularizing various parts of the exising `uploads_import` script
* DEV: First attempt at modularizing missing uploads fixer task
Move missing upload fix to a dedicated uploads task implementation unit
* DEV: First attempt at modularizing missing uploads uploader task
Move uploader to a dedicated uploads task implementation unit
* DEV: First attempt at modularizing missing uploads optimizer task
Move optimizer to a dedicated uploads task implementation unit
* DEV: Various follow up fixes to get optimization working
- Start threads early
- Improve "log" message formatting
- Add missing `copy_to_tempfile` method on "uploader" task
* DEV: Refactor a bit more
Deduplicate and move most of threading premitives to base task as-is
* DEV: Remove redundant condition in uploads db migration
* DEV: More deduplication
Move task retry logic to base class and tidy up other implementation
details carried over from the existing script
* Updates GitHub Actions
* Switches from `bundler/inline` to an optional group in the `Gemfile` because the previous solution didn't work well with rspec
* Adds the converter framework and tests
* Allows loading private converters (see README)
* Switches from multiple CLI tools to a single CLI
* Makes DB connections reusable and adds a new abstraction for the `IntermediateDB`
* `IntermediateDB` acts as an interface for IPC calls when a converter steps runs in parallel (forks). Only the main process writes to the DB.
* Includes a simple example implementation of a converter for now.
* Moves existing files around. All essential scripts are in `migrations/bin`, and non-essential scripts like benchmarks are in `migrations/scripts`
* Dependabot configuration for migrations-tooling (disabled for now)
* Updates test configuration for migrations-tooling
* Shorter configuration for intermediate DB for now. We will add the rest table by table.
* Adds a couple of benchmark scripts
* RSpec setup especially for migrations-tooling and the first tests
* Adds sorting/formatting to the `generate_schema` script
