mirror of
https://github.com/discourse/discourse.git
synced 2026-09-05 04:40:41 -05:00
MT: Split the migrations tooling into separate gems (#40492)
Previously, the migrations tooling was a single flat `migrations/` tree, autoloaded by one global Zeitwerk loader and driven by a Thor CLI, so each planned next step had nowhere clean to land. This change splits it into four `path:`-referenced gems — `migrations-core`, `migrations-tooling`, `migrations-converters`, and `migrations-importer` — served by a single Samovar-based `disco` binary, without rewriting any domain logic. ### Why now The DSL refactor that replaced the IntermediateDB YAML config just landed, which is the cheapest moment to do this. Everything queued behind it — column coverage verification, the `discourse-migrations` validation plugin, the transformer framework, and private converter isolation — either has nowhere clean to land in the flat tree or would have to be retrofitted into a gem layout later. Doing the split now, while it's still a pure move (suite green, no domain logic touched), is far cheaper than after another round of features has built on the flat layout. ### What changes - **Four gems under `migrations/`**, all `path:`-referenced from the root `Gemfile` (nothing is published to RubyGems): `core` (CLI framework, UI, DB infrastructure, IntermediateDB, and the conversion framework), `tooling` (schema DSL and `schema` commands), `converters` (implementations and source adapters), and `importer` (row and uploads import). - **A single CLI binary:** `migrations/bin/cli` (Thor) becomes `disco` (Samovar), with each gem registering its own commands. Same surface — `convert`, `import`, `upload`, `schema generate|validate|…` — and Rails is still booted lazily. - **Isolated test suites:** each gem runs its own no-Rails specs in a new CI job, while the existing job keeps running the Rails-integration specs.
This commit is contained in:
Gerhard Schlager
@@ -21,6 +21,28 @@ permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
gems:
|
||||
if: github.event_name == 'pull_request' || github.repository != 'discourse/discourse-private-mirror'
|
||||
name: Gems
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
- uses: ruby/setup-ruby@v1
|
||||
with:
|
||||
ruby-version: "3.4"
|
||||
|
||||
- name: RSpec (isolated, no Rails)
|
||||
run: |
|
||||
for gem in core tooling converters importer; do
|
||||
echo "== migrations-$gem =="
|
||||
(cd "migrations/$gem" && bundle install --jobs $(($(nproc) - 1)) && bundle exec rspec --tag ~rails)
|
||||
done
|
||||
|
||||
tests:
|
||||
if: github.event_name == 'pull_request' || github.repository != 'discourse/discourse-private-mirror'
|
||||
name: Tests
|
||||
@@ -80,13 +102,13 @@ jobs:
|
||||
- name: Validate IntermediateDB schema
|
||||
run: |
|
||||
# Show structured differences between DSL config and database
|
||||
migrations/bin/cli schema diff --db=intermediate_db || true
|
||||
migrations/bin/disco schema diff --db intermediate_db || true
|
||||
|
||||
# Regenerate schema from DSL config
|
||||
migrations/bin/cli schema generate --db=intermediate_db
|
||||
migrations/bin/disco schema generate --db intermediate_db
|
||||
|
||||
# Check if generated output matches what's committed
|
||||
if [ ! -z "$(git status --porcelain migrations/db/intermediate_db_schema/ migrations/lib/database/intermediate_db/ migrations/lib/database/intermediate_db/enums/)" ]; then
|
||||
if [ ! -z "$(git status --porcelain migrations/core/db/intermediate_db_schema/ migrations/core/lib/migrations/database/intermediate_db/)" ]; then
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo "IntermediateDB schema is not up to date."
|
||||
@@ -97,19 +119,25 @@ jobs:
|
||||
echo "structure changed without regenerating."
|
||||
echo ""
|
||||
echo "To fix:"
|
||||
echo " 1. Run: migrations/bin/cli schema diff"
|
||||
echo " 1. Run: migrations/bin/disco schema diff"
|
||||
echo " to see what changed between config and database"
|
||||
echo " 2. Update config files in migrations/config/schema/intermediate_db/"
|
||||
echo " - New tables: migrations/bin/cli schema add <table>"
|
||||
echo " - Ignore tables: migrations/bin/cli schema ignore <table> --reason '...'"
|
||||
echo " 2. Update config files in migrations/tooling/config/schema/intermediate_db/"
|
||||
echo " - New tables: migrations/bin/disco schema add <table>"
|
||||
echo " - Ignore tables: migrations/bin/disco schema ignore <table> --reason '...'"
|
||||
echo " - New columns: add to the table's include list or ignore with reason"
|
||||
echo " 3. Run: migrations/bin/cli schema generate"
|
||||
echo " 3. Run: migrations/bin/disco schema generate"
|
||||
echo " 4. Commit the updated config and generated files"
|
||||
echo ""
|
||||
echo "Generated file diff:"
|
||||
git -c color.ui=always diff -- migrations/db/intermediate_db_schema/ migrations/lib/database/intermediate_db/ migrations/lib/database/intermediate_db/enums/
|
||||
git -c color.ui=always diff -- migrations/core/db/intermediate_db_schema/ migrations/core/lib/migrations/database/intermediate_db/
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: RSpec
|
||||
run: bin/rspec --default-path migrations/spec
|
||||
- name: RSpec (Rails integration)
|
||||
env:
|
||||
MIGRATIONS_RAILS: "1"
|
||||
run: |
|
||||
for gem in core tooling converters importer; do
|
||||
echo "== migrations-$gem (Rails integration) =="
|
||||
(cd "migrations/$gem" && BUNDLE_GEMFILE="$GITHUB_WORKSPACE/Gemfile" bundle exec rspec --tag rails)
|
||||
done
|
||||
|
||||
@@ -277,19 +277,10 @@ gem "iso8601"
|
||||
gem "rrule"
|
||||
|
||||
group :migrations, optional: true do
|
||||
gem "extralite-bundle", require: "extralite"
|
||||
|
||||
# auto-loading
|
||||
gem "zeitwerk"
|
||||
|
||||
# databases
|
||||
gem "trilogy"
|
||||
|
||||
# CLI
|
||||
gem "ruby-progressbar"
|
||||
|
||||
# non-cryptographic hashing algorithm for generating placeholder IDs
|
||||
gem "digest-xxhash"
|
||||
gem "migrations-core", path: "migrations/core"
|
||||
gem "migrations-tooling", path: "migrations/tooling"
|
||||
gem "migrations-converters", path: "migrations/converters"
|
||||
gem "migrations-importer", path: "migrations/importer"
|
||||
end
|
||||
|
||||
gem "dry-initializer", "~> 3.1"
|
||||
|
||||
+77
-10
@@ -1,3 +1,52 @@
|
||||
PATH
|
||||
remote: migrations/converters
|
||||
specs:
|
||||
migrations-converters (0.0.1)
|
||||
activesupport
|
||||
colored2
|
||||
i18n
|
||||
migrations-core
|
||||
pg
|
||||
zeitwerk
|
||||
|
||||
PATH
|
||||
remote: migrations/core
|
||||
specs:
|
||||
migrations-core (0.0.1)
|
||||
activesupport
|
||||
colored2
|
||||
digest-xxhash
|
||||
extralite-bundle
|
||||
i18n
|
||||
lru_redux
|
||||
oj
|
||||
ruby-progressbar
|
||||
samovar
|
||||
zeitwerk
|
||||
|
||||
PATH
|
||||
remote: migrations/importer
|
||||
specs:
|
||||
migrations-importer (0.0.1)
|
||||
activerecord
|
||||
activesupport
|
||||
colored2
|
||||
i18n
|
||||
migrations-core
|
||||
pg
|
||||
zeitwerk
|
||||
|
||||
PATH
|
||||
remote: migrations/tooling
|
||||
specs:
|
||||
migrations-tooling (0.0.1)
|
||||
activerecord
|
||||
activesupport
|
||||
colored2
|
||||
i18n
|
||||
migrations-core
|
||||
zeitwerk
|
||||
|
||||
GEM
|
||||
remote: https://rubygems.org/
|
||||
specs:
|
||||
@@ -125,6 +174,10 @@ GEM
|
||||
colored2 (4.0.3)
|
||||
concurrent-ruby (1.3.6)
|
||||
connection_pool (2.5.5)
|
||||
console (1.36.0)
|
||||
fiber-annotation
|
||||
fiber-local (~> 1.1)
|
||||
json
|
||||
cose (1.3.1)
|
||||
cbor (~> 0.5.9)
|
||||
openssl-signature_algorithm (~> 1.0)
|
||||
@@ -189,6 +242,10 @@ GEM
|
||||
ffi (1.17.4-x86_64-darwin)
|
||||
ffi (1.17.4-x86_64-linux-gnu)
|
||||
ffi (1.17.4-x86_64-linux-musl)
|
||||
fiber-annotation (0.2.0)
|
||||
fiber-local (1.1.0)
|
||||
fiber-storage
|
||||
fiber-storage (1.0.1)
|
||||
fspath (3.1.2)
|
||||
globalid (1.3.0)
|
||||
activesupport (>= 6.1)
|
||||
@@ -286,6 +343,7 @@ GEM
|
||||
net-imap
|
||||
net-pop
|
||||
net-smtp
|
||||
mapping (1.1.3)
|
||||
matrix (0.4.3)
|
||||
maxminddb (0.1.22)
|
||||
memory_profiler (1.1.0)
|
||||
@@ -617,6 +675,9 @@ GEM
|
||||
nokogiri (>= 1.6.0)
|
||||
ruby2_keywords (0.0.5)
|
||||
rubyzip (3.3.0)
|
||||
samovar (2.4.1)
|
||||
console (~> 1.0)
|
||||
mapping (~> 1.0)
|
||||
sanitize (7.0.0)
|
||||
crass (~> 1.0.2)
|
||||
nokogiri (>= 1.16.8)
|
||||
@@ -683,8 +744,6 @@ GEM
|
||||
tokenizers (0.6.4-x86_64-darwin)
|
||||
tokenizers (0.6.4-x86_64-linux)
|
||||
tokenizers (0.6.4-x86_64-linux-musl)
|
||||
trilogy (2.12.5)
|
||||
bigdecimal
|
||||
tsort (0.2.0)
|
||||
ttfunk (1.8.0)
|
||||
bigdecimal (~> 3.1)
|
||||
@@ -776,7 +835,6 @@ DEPENDENCIES
|
||||
debug (>= 1.0.0)
|
||||
diffy
|
||||
digest
|
||||
digest-xxhash
|
||||
discourse-emojis
|
||||
discourse-fonts
|
||||
discourse-seed-fu
|
||||
@@ -787,7 +845,6 @@ DEPENDENCIES
|
||||
ed25519
|
||||
email_reply_trimmer
|
||||
excon
|
||||
extralite-bundle
|
||||
fabrication
|
||||
faker
|
||||
faraday
|
||||
@@ -818,6 +875,10 @@ DEPENDENCIES
|
||||
memory_profiler
|
||||
message_bus
|
||||
messageformat-wrapper
|
||||
migrations-converters!
|
||||
migrations-core!
|
||||
migrations-importer!
|
||||
migrations-tooling!
|
||||
mini_mime
|
||||
mini_racer
|
||||
mini_scheduler
|
||||
@@ -879,7 +940,6 @@ DEPENDENCIES
|
||||
ruby-lsp-rails
|
||||
ruby-lsp-rspec
|
||||
ruby-prof
|
||||
ruby-progressbar
|
||||
ruby-rc4
|
||||
ruby-readability
|
||||
rubyzip
|
||||
@@ -898,7 +958,6 @@ DEPENDENCIES
|
||||
thor
|
||||
tiktoken_ruby
|
||||
tokenizers
|
||||
trilogy
|
||||
ttfunk
|
||||
tty-prompt
|
||||
tzinfo-data
|
||||
@@ -907,7 +966,6 @@ DEPENDENCIES
|
||||
webmock
|
||||
yaml-lint
|
||||
yard
|
||||
zeitwerk
|
||||
zendesk_api
|
||||
|
||||
CHECKSUMS
|
||||
@@ -943,7 +1001,7 @@ CHECKSUMS
|
||||
bootsnap (1.24.5) sha256=36b677448524d279b470469aabd5dff4a980e3fa4931a0df68da4a500eb1b6c4
|
||||
builder (3.3.0) sha256=497918d2f9dca528fdca4b88d84e4ef4387256d984b8154e9d5d3fe5a9c8835f
|
||||
bullet (8.1.1) sha256=3ddecdf3787c64a3f1e5b316256840bc034d60c1070b1b16d3fe5b5932d42b87
|
||||
bundler (4.0.11) sha256=5bcec0fb78302e48d02ee46f10ee6e6942be647ba5b44a6d1ddfda9a240ce785
|
||||
bundler (4.0.12) sha256=7f8b757d28dfb636e7b24fba2344ac6dd13b5b24f4b46d62573d483f211825ac
|
||||
capybara (3.40.0) sha256=42dba720578ea1ca65fd7a41d163dd368502c191804558f6e0f71b391054aeef
|
||||
capybara-playwright-driver (0.5.9) sha256=4c17fed20817b7e1bde2cfc8186e7bf5cd72017ce1aa695e35130f8e600e7282
|
||||
cbor (0.5.10.2) sha256=df5104f7a62c881123e6505441b1e276208be1771540c2cc3b1de8a210a7c52c
|
||||
@@ -954,6 +1012,7 @@ CHECKSUMS
|
||||
colored2 (4.0.3) sha256=63e1038183976287efc43034f5cca17fb180b4deef207da8ba78d051cbce2b37
|
||||
concurrent-ruby (1.3.6) sha256=6b56837e1e7e5292f9864f34b69c5a2cbc75c0cf5338f1ce9903d10fa762d5ab
|
||||
connection_pool (2.5.5) sha256=e54ff92855753df1fd7c59fa04a398833355f27dd14c074f8c83a05f72a716ad
|
||||
console (1.36.0) sha256=45599ea906cf80a73d8941f03abf873fe66a6a954e0bac5bc1c01e2cdc406f07
|
||||
cose (1.3.1) sha256=d5d4dbcd6b035d513edc4e1ab9bc10e9ce13b4011c96e3d1b8fe5e6413fd6de5
|
||||
cppjieba_rb (0.4.4) sha256=319a7ab57b6ec28a8d1b223487ecd114432f1930d7740db2f99c1991e6c8faaf
|
||||
crack (1.0.1) sha256=ff4a10390cd31d66440b7524eb1841874db86201d5b70032028553130b6d4c7e
|
||||
@@ -996,6 +1055,9 @@ CHECKSUMS
|
||||
ffi (1.17.4-x86_64-darwin) sha256=aa70390523cf3235096cf64962b709b4cfbd5c082a2cb2ae714eb0fe2ccda496
|
||||
ffi (1.17.4-x86_64-linux-gnu) sha256=9d3db14c2eae074b382fa9c083fe95aec6e0a1451da249eab096c34002bc752d
|
||||
ffi (1.17.4-x86_64-linux-musl) sha256=3fdf9888483de005f8ef8d1cf2d3b20d86626af206cbf780f6a6a12439a9c49e
|
||||
fiber-annotation (0.2.0) sha256=7abfadf1d119f508867d4103bf231c0354d019cc39a5738945dec2edadaf6c03
|
||||
fiber-local (1.1.0) sha256=c885f94f210fb9b05737de65d511136ea602e00c5105953748aa0f8793489f06
|
||||
fiber-storage (1.0.1) sha256=f48e5b6d8b0be96dac486332b55cee82240057065dc761c1ea692b2e719240e1
|
||||
fspath (3.1.2) sha256=b5ac9bafb97e2c8f8f9e303cd98ebd484be76fe9aa588bc4d01c6d99e78c9d75
|
||||
globalid (1.3.0) sha256=05c639ad6eb4594522a0b07983022f04aa7254626ab69445a0e493aa3786ff11
|
||||
goldiloader (6.0.0) sha256=613db1b28e9964291255a67a521f04d481b6afd6b9ca56ea1a612fd86e9a89c7
|
||||
@@ -1045,12 +1107,17 @@ CHECKSUMS
|
||||
lru_redux (1.1.0) sha256=ee71d0ccab164c51de146c27b480a68b3631d5b4297b8ffe8eda1c72de87affb
|
||||
lz4-ruby (0.3.3) sha256=011be5ee230cfddc8308d4e2e0b05300c7bc755a887de799377ca6c5b6aede89
|
||||
mail (2.9.0) sha256=6fa6673ecd71c60c2d996260f9ee3dd387d4673b8169b502134659ece6d34941
|
||||
mapping (1.1.3) sha256=2274931d20ecd46eaafdd1e00c58cc7472133b213bcac335cc7733d3c75f4da2
|
||||
matrix (0.4.3) sha256=a0d5ab7ddcc1973ff690ab361b67f359acbb16958d1dc072b8b956a286564c5b
|
||||
maxminddb (0.1.22) sha256=50933be438fbed9dceabef4163eab41884bd8830d171fdb8f739bee769c4907e
|
||||
memory_profiler (1.1.0) sha256=79a17df7980a140c83c469785905409d3027ca614c42c086089d128b805aa8f8
|
||||
message_bus (4.5.2) sha256=f1381bce05d9560e5ea7dfa56e44dfdf0080bda0ac02f4011eb662bb7dd9e778
|
||||
messageformat-wrapper (1.1.0) sha256=ecea879626e412d1bc841c457dacfcbb1a62cf88ca83573e4ea34bb371f160bc
|
||||
method_source (1.1.0) sha256=181301c9c45b731b4769bc81e8860e72f9161ad7d66dd99103c9ab84f560f5c5
|
||||
migrations-converters (0.0.1)
|
||||
migrations-core (0.0.1)
|
||||
migrations-importer (0.0.1)
|
||||
migrations-tooling (0.0.1)
|
||||
mime-types (3.7.0) sha256=dcebf61c246f08e15a4de34e386ebe8233791e868564a470c3fe77c00eed5e56
|
||||
mime-types-data (3.2026.0414) sha256=461c4c655373a44bd6c5fe54bcf5b7776026ea96e808144b1ec465c4b99148cc
|
||||
mini_mime (1.1.5) sha256=8681b7e2e4215f2a159f9400b5816d85e9d8c6c6b491e96a12797e798f8bccef
|
||||
@@ -1182,6 +1249,7 @@ CHECKSUMS
|
||||
ruby-readability (0.7.3) sha256=bd213fab037118cc15d999e167a8ea2cc6b689dc78c033e78fb36a2a37efc241
|
||||
ruby2_keywords (0.0.5) sha256=ffd13740c573b7301cf7a2e61fc857b2a8e3d3aff32545d6f8300d8bae10e3ef
|
||||
rubyzip (3.3.0) sha256=a372fc67892a4f8c0bc8ec906b720353d8e48807a64b2e63adf99b1e3583a034
|
||||
samovar (2.4.1) sha256=c3b91dd0580771e3bc600621c1111f29542529dcffafaac3b6bf068b3f309e80
|
||||
sanitize (7.0.0) sha256=269d1b9d7326e69307723af5643ec032ff86ad616e72a3b36d301ac75a273984
|
||||
sass-embedded (1.99.0-aarch64-linux-gnu) sha256=a46615b0295ca7bd979b9ce79f6b9f1d26881736400188bd6fd5c4b7c9b46473
|
||||
sass-embedded (1.99.0-aarch64-linux-musl) sha256=eaa6d56968909d1d54073c46e21c13fd5bbcb5af609d7e4fbe6b196426ca8e49
|
||||
@@ -1225,7 +1293,6 @@ CHECKSUMS
|
||||
tokenizers (0.6.4-x86_64-darwin) sha256=102b143e71e59bc7ff29e5920f356ad74bbc941ae2d4e3a495ee92311ace7394
|
||||
tokenizers (0.6.4-x86_64-linux) sha256=cdec6e5b007d73de094cdc7e097e4ab9a545f37f9c30e376a53800c3a552a447
|
||||
tokenizers (0.6.4-x86_64-linux-musl) sha256=588031377e3f586b66687fc9dc59f901dea47d863733d871bf6801657e312c72
|
||||
trilogy (2.12.5) sha256=9d930b1cd7b28ac6bb11c0f975ab040206073a767f1b7672f154339fc2c0cd71
|
||||
tsort (0.2.0) sha256=9650a793f6859a43b6641671278f79cfead60ac714148aabe4e3f0060480089f
|
||||
ttfunk (1.8.0) sha256=a7cbc7e489cc46e979dde04d34b5b9e4f5c8f1ee5fc6b1a7be39b829919d20ca
|
||||
tty-color (0.6.0) sha256=6f9c37ca3a4e2367fb2e6d09722762647d6f455c111f05b59f35730eeb24332a
|
||||
@@ -1256,4 +1323,4 @@ RUBY VERSION
|
||||
ruby 3.4.7p58
|
||||
|
||||
BUNDLED WITH
|
||||
4.0.11
|
||||
4.0.12
|
||||
|
||||
@@ -8,3 +8,6 @@
|
||||
/private/
|
||||
|
||||
/tmp/
|
||||
|
||||
# Each gem resolves its own lockfile for the isolated spec suite (not committed)
|
||||
/*/Gemfile.lock
|
||||
|
||||
+7
-48
@@ -1,50 +1,9 @@
|
||||
# Migrations Tooling - AI Agent Guide
|
||||
# Migrations Tooling — Agent Guide
|
||||
|
||||
## Running Tests
|
||||
Start with **[README.md](README.md)** — it is the single source of truth for this
|
||||
project: gem layout and namespaces, the `disco` CLI, converters, the schema DSL,
|
||||
and the install / test / lint workflow.
|
||||
|
||||
Tests must be run from the project root with `--default-path migrations/spec`:
|
||||
|
||||
```bash
|
||||
bin/rspec --default-path migrations/spec
|
||||
bin/rspec --default-path migrations/spec migrations/spec/lib/database/schema/dsl/
|
||||
bin/rspec --default-path migrations/spec migrations/spec/path/to/file_spec.rb
|
||||
```
|
||||
|
||||
## CLI
|
||||
|
||||
The CLI binary is at `migrations/bin/cli`:
|
||||
|
||||
```bash
|
||||
migrations/bin/cli help
|
||||
migrations/bin/cli schema generate
|
||||
migrations/bin/cli schema validate
|
||||
migrations/bin/cli schema diff
|
||||
```
|
||||
|
||||
## Schema DSL
|
||||
|
||||
The schema DSL lives in `migrations/lib/database/schema/dsl/`. Config files are in `migrations/config/schema/`.
|
||||
|
||||
Key files:
|
||||
- `table_builder.rb` - DSL for defining table configs
|
||||
- `schema_resolver.rb` - Resolves DSL config + DB introspection into final schema
|
||||
- `conventions_builder.rb` - Global column conventions (renames, type overrides)
|
||||
- `generator.rb` - Generates SQL, models, and enums from resolved schema
|
||||
- `validator.rb` - Validates DSL config
|
||||
- `resolved_schema_validator.rb` - Validates resolved schema before generation
|
||||
|
||||
## Linting
|
||||
|
||||
```bash
|
||||
bin/lint path/to/file
|
||||
bin/lint --fix path/to/file
|
||||
```
|
||||
|
||||
Uses both rubocop and syntax_tree. Always lint changed files.
|
||||
|
||||
## Gems
|
||||
|
||||
```bash
|
||||
bundle config set --local with migrations
|
||||
bundle install
|
||||
```
|
||||
This file is reserved for **agent-specific** guidance that does not belong in the
|
||||
README (conventions, gotchas, do/don't notes for automated contributors). There is
|
||||
none yet; add it here as it comes up.
|
||||
|
||||
+56
-5
@@ -1,15 +1,47 @@
|
||||
# Migrations Tooling
|
||||
|
||||
The `migrations/` directory is split into four path-referenced gems:
|
||||
|
||||
- `core/` — `Migrations::*`: CLI framework, UI, SQLite schemas, DB infrastructure,
|
||||
IntermediateDB models, and the conversion framework (`Migrations::Conversion::*`).
|
||||
- `tooling/` — `Migrations::Tooling::*`: the schema DSL, `disco schema` commands, benchmarks.
|
||||
- `converters/` — `Migrations::Converters::*`: public converter implementations + source adapters.
|
||||
- `importer/` — `Migrations::Importer::*`: the row importer and the uploads importer.
|
||||
|
||||
All four are wired into the root `Gemfile` via `path:` in the optional `:migrations` group.
|
||||
|
||||
## Command line interface
|
||||
|
||||
The single binary is `migrations/bin/disco` (commands register dynamically via
|
||||
`Migrations::CLI::Registry`). Run it without arguments — or with `--help` — for the
|
||||
authoritative, always-current list of commands:
|
||||
|
||||
```bash
|
||||
./bin/cli help
|
||||
migrations/bin/disco --help
|
||||
```
|
||||
|
||||
Rails is booted lazily: only commands that declare `requires_rails!` (import, upload, schema)
|
||||
load the Discourse app.
|
||||
|
||||
## Converters
|
||||
|
||||
Public converters are stored in `lib/converters/`.
|
||||
If you need to run a private converter, put its code into a subdirectory of `private/converters/`
|
||||
Public converters live in `converters/lib/migrations/converters/`. To run a private
|
||||
(closed-source) converter, put its code in a subdirectory of `private/converters/`
|
||||
(or point `MIGRATIONS_PRIVATE_CONVERTERS_PATH` at it).
|
||||
|
||||
## Schema DSL
|
||||
|
||||
The schema DSL lives in `migrations/tooling/lib/migrations/tooling/schema/dsl/`. Config sources
|
||||
are in `migrations/tooling/config/schema/`. Generated artifacts (SQL, models, enums) are written
|
||||
into `migrations/core/`.
|
||||
|
||||
Key files:
|
||||
- `table_builder.rb` - DSL for defining table configs
|
||||
- `schema_resolver.rb` - Resolves DSL config + DB introspection into final schema
|
||||
- `conventions_builder.rb` - Global column conventions (renames, type overrides)
|
||||
- `generator.rb` - Generates SQL, models, and enums from resolved schema
|
||||
- `validator.rb` - Validates DSL config
|
||||
- `resolved_schema_validator.rb` - Validates resolved schema before generation
|
||||
|
||||
## Development
|
||||
|
||||
@@ -28,8 +60,27 @@ bundle update --group migrations
|
||||
|
||||
### Running tests
|
||||
|
||||
You need to execute `rspec` in the root of the project.
|
||||
Each gem has an isolated, no-Rails suite, run from the gem directory:
|
||||
|
||||
```bash
|
||||
bin/rspec --default-path migrations/spec
|
||||
cd migrations/core && bundle exec rspec
|
||||
cd migrations/tooling && bundle exec rspec
|
||||
cd migrations/converters && bundle exec rspec
|
||||
cd migrations/importer && bundle exec rspec
|
||||
```
|
||||
|
||||
Specs that need a booted Rails environment are tagged `:rails`. They are excluded by default and
|
||||
run from the host app's bundle:
|
||||
|
||||
```bash
|
||||
cd migrations/<gem> && BUNDLE_GEMFILE=../../Gemfile MIGRATIONS_RAILS=1 bundle exec rspec --tag rails
|
||||
```
|
||||
|
||||
### Linting
|
||||
|
||||
```bash
|
||||
bin/lint path/to/file
|
||||
bin/lint --fix path/to/file
|
||||
```
|
||||
|
||||
Uses both rubocop and syntax_tree. Always lint changed files.
|
||||
|
||||
@@ -1,95 +0,0 @@
|
||||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
require_relative "../migrations"
|
||||
|
||||
require "colored2"
|
||||
require "thor"
|
||||
|
||||
module Migrations
|
||||
configure_zeitwerk
|
||||
enable_i18n
|
||||
apply_global_config
|
||||
|
||||
module CLI
|
||||
class Application < Thor
|
||||
remove_command :tree
|
||||
|
||||
desc "convert [FROM]", "Convert a file"
|
||||
option :settings, type: :string, desc: "Path of settings file", banner: "path"
|
||||
option :reset, type: :boolean, desc: "Reset database before converting data"
|
||||
option :only,
|
||||
type: :string,
|
||||
desc: "Run only specified steps (comma-separated)",
|
||||
banner: "step1,step2"
|
||||
option :skip,
|
||||
type: :string,
|
||||
desc: "Skip specified steps (comma-separated)",
|
||||
banner: "step1,step2"
|
||||
def convert(converter_type)
|
||||
modified_options = modify_step_options
|
||||
::Migrations::CLI::ConvertCommand.new(converter_type, modified_options).execute
|
||||
end
|
||||
|
||||
desc "import", "Import a file"
|
||||
option :reset, type: :boolean, desc: "Reset MappingsDB before importing data"
|
||||
option :only,
|
||||
type: :string,
|
||||
desc: "Run only specified steps (comma-separated)",
|
||||
banner: "step1,step2"
|
||||
option :skip,
|
||||
type: :string,
|
||||
desc: "Skip specified steps (comma-separated)",
|
||||
banner: "step1,step2"
|
||||
def import
|
||||
modified_options = modify_step_options
|
||||
::Migrations::CLI::ImportCommand.new(modified_options).execute
|
||||
end
|
||||
|
||||
desc "upload", "Upload media uploads"
|
||||
option :settings,
|
||||
type: :string,
|
||||
desc: "Uploads settings file path",
|
||||
default: "./migrations/config/upload.yml",
|
||||
aliases: "-s",
|
||||
banner: "path"
|
||||
option :fix_missing, type: :boolean, desc: "Fix missing uploads"
|
||||
option :optimize, type: :boolean, desc: "Optimize uploads"
|
||||
def upload
|
||||
::Migrations::CLI::UploadCommand.new(options).execute
|
||||
end
|
||||
|
||||
desc "schema [COMMAND]", "Manage database schema"
|
||||
subcommand "schema", ::Migrations::CLI::SchemaSubCommand
|
||||
|
||||
def self.exit_on_failure?
|
||||
true
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def modify_step_options
|
||||
modified_options = options.dup
|
||||
modified_options[:only] = split_step_class_names(options[:only])
|
||||
modified_options[:skip] = split_step_class_names(options[:skip])
|
||||
modified_options
|
||||
end
|
||||
|
||||
def split_step_class_names(class_names)
|
||||
class_names.presence&.split(",")&.map { |name| name.strip.demodulize.underscore } || []
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
if defined?(RubyVM::YJIT)
|
||||
RubyVM::YJIT.enable
|
||||
else
|
||||
warn "WARNING: Performance degraded: RubyVM::YJIT is not available".yellow
|
||||
end
|
||||
|
||||
# rubocop:disable Discourse/NoChdir
|
||||
Dir.chdir(File.expand_path("../..", __dir__)) do
|
||||
::Migrations::CLI::ExceptionHandler.handle_and_exit { ::Migrations::CLI::Application.start }
|
||||
end
|
||||
# rubocop:enable Discourse/NoChdir
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
# Convenience launcher for the migrations CLI.
|
||||
#
|
||||
# The real executable is the `disco` binary shipped by the migrations-core gem
|
||||
# (migrations/core/bin/disco), which is also runnable via `bundle exec disco`.
|
||||
# This thin wrapper exposes it at the familiar migrations/bin/ location; all of
|
||||
# the logic lives in Migrations::CLI::Bootstrap.
|
||||
exec(File.expand_path("../core/bin/disco", __dir__), *ARGV)
|
||||
@@ -1,5 +0,0 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
Migrations::Database::Schema.table :badge_groupings do
|
||||
include_all
|
||||
end
|
||||
@@ -1,5 +0,0 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
Migrations::Database::Schema.table :tag_groups do
|
||||
include_all
|
||||
end
|
||||
@@ -0,0 +1,2 @@
|
||||
--require spec_helper
|
||||
--format documentation
|
||||
@@ -0,0 +1,13 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
source "https://rubygems.org"
|
||||
|
||||
gem "migrations-core", path: "../core"
|
||||
|
||||
gemspec
|
||||
|
||||
group :test do
|
||||
gem "rspec"
|
||||
gem "rspec-multi-mock"
|
||||
gem "mocha"
|
||||
end
|
||||
@@ -0,0 +1,4 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require "migrations-core"
|
||||
require_relative "migrations/converters"
|
||||
@@ -0,0 +1,99 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require "zeitwerk"
|
||||
|
||||
module Migrations
|
||||
module Converters
|
||||
# Directories under the gem's converters root that are framework
|
||||
# infrastructure rather than converter implementations.
|
||||
NON_CONVERTER_DIRS = %w[adapter cli].freeze
|
||||
|
||||
def self.root_path
|
||||
@root_path ||= File.expand_path("../..", __dir__)
|
||||
end
|
||||
|
||||
def self.converters_path
|
||||
@converters_path ||= File.join(__dir__, "converters")
|
||||
end
|
||||
|
||||
# Where private (closed-source) converters live in the host application.
|
||||
def self.private_converters_path
|
||||
@private_converters_path ||=
|
||||
ENV["MIGRATIONS_PRIVATE_CONVERTERS_PATH"].presence ||
|
||||
File.join(Migrations.host_app_root, "migrations", "private", "converters")
|
||||
end
|
||||
|
||||
def self.all
|
||||
@all_converters ||=
|
||||
begin
|
||||
public_paths = Dir[File.join(converters_path, "*")]
|
||||
private_paths = Dir[File.join(private_converters_path, "*")]
|
||||
non_converter_paths = NON_CONVERTER_DIRS.map { |d| File.join(converters_path, d) }
|
||||
all_paths = (public_paths - non_converter_paths) + private_paths
|
||||
|
||||
all_paths.each_with_object({}) do |path, hash|
|
||||
next unless File.directory?(path)
|
||||
|
||||
name = File.basename(path).downcase
|
||||
existing_path = hash[name]
|
||||
|
||||
raise <<~MSG if existing_path
|
||||
Duplicate converter name found: #{name}
|
||||
* #{existing_path}
|
||||
* #{path}
|
||||
MSG
|
||||
|
||||
hash[name] = path
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def self.names
|
||||
all.keys.sort
|
||||
end
|
||||
|
||||
def self.path_of(converter_name)
|
||||
converter_name = converter_name.downcase
|
||||
path = all[converter_name]
|
||||
raise "Could not find a converter named '#{converter_name}'" unless path
|
||||
path
|
||||
end
|
||||
|
||||
def self.default_settings_path(converter_name)
|
||||
local_settings_path = File.join(path_of(converter_name), "settings.local.yml")
|
||||
return local_settings_path if File.exist?(local_settings_path)
|
||||
|
||||
File.join(path_of(converter_name), "settings.yml")
|
||||
end
|
||||
|
||||
def self.loader
|
||||
@loader ||=
|
||||
begin
|
||||
loader = Zeitwerk::Loader.new
|
||||
loader.log! if ENV["DEBUG"]
|
||||
loader.inflector.inflect("db" => "DB", "id" => "ID", "cli" => "CLI")
|
||||
loader.push_dir(converters_path, namespace: Converters)
|
||||
loader.ignore(File.join(converters_path, "register.rb"))
|
||||
|
||||
# Each converter directory collapses all of its subdirectories into a
|
||||
# single namespace, so that e.g. `discourse/steps/users.rb` defines
|
||||
# `Migrations::Converters::Discourse::Users`. This is required by
|
||||
# `Migrations::Conversion::Base#steps`, which discovers steps via the
|
||||
# converter module's constants.
|
||||
all.each_value do |converter_path|
|
||||
Dir[File.join(converter_path, "**", "*")].each do |subdir|
|
||||
loader.collapse(subdir) if File.directory?(subdir)
|
||||
end
|
||||
end
|
||||
|
||||
loader
|
||||
end
|
||||
end
|
||||
|
||||
def self.setup_loader
|
||||
loader.setup
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
Migrations::Converters.setup_loader
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
require "pg"
|
||||
|
||||
module Migrations
|
||||
module Database
|
||||
module Converters
|
||||
module Adapter
|
||||
class Postgres
|
||||
def initialize(settings)
|
||||
@@ -0,0 +1,66 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module Converters
|
||||
module CLI
|
||||
class ConvertCommand < Migrations::CLI::Command
|
||||
class Error < StandardError
|
||||
include Migrations::CLI::PresentableError
|
||||
end
|
||||
|
||||
self.description = "Convert a source dump into the IntermediateDB"
|
||||
|
||||
options do
|
||||
option "-h/--help", "Print out help."
|
||||
option "--settings <path>", "Path of the settings file."
|
||||
option "--reset", "Reset the database before converting data."
|
||||
option "--only <steps>",
|
||||
"Run only the specified steps (comma-separated).",
|
||||
default: [],
|
||||
type: STEP_LIST
|
||||
option "--skip <steps>",
|
||||
"Skip the specified steps (comma-separated).",
|
||||
default: [],
|
||||
type: STEP_LIST
|
||||
end
|
||||
|
||||
one :converter_type, "The converter to run (e.g. discourse)."
|
||||
|
||||
def call
|
||||
return print_usage if @options[:help]
|
||||
|
||||
type = converter_type.downcase
|
||||
validate_converter_type!(type)
|
||||
|
||||
settings = load_settings(type)
|
||||
|
||||
Database.reset!(settings[:intermediate_db][:path]) if @options[:reset]
|
||||
|
||||
converter = "migrations/converters/#{type}/converter".camelize.constantize
|
||||
converter.new(settings).run(only_steps: @options[:only], skip_steps: @options[:skip])
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def validate_converter_type!(type)
|
||||
names = Converters.names
|
||||
return if names.include?(type)
|
||||
|
||||
raise Error, <<~MSG
|
||||
Unknown converter name: #{type}
|
||||
Valid names are: #{names.join(", ")}
|
||||
MSG
|
||||
end
|
||||
|
||||
def load_settings(type)
|
||||
settings_path = @options[:settings] || Converters.default_settings_path(type)
|
||||
settings_path = File.expand_path(settings_path, Dir.pwd)
|
||||
|
||||
raise Error, "Settings file not found: #{settings_path}" unless File.exist?(settings_path)
|
||||
|
||||
YAML.safe_load(File.read(settings_path), symbolize_names: true)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
+2
-2
@@ -3,10 +3,10 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class Converter < Base::Converter
|
||||
class Converter < Conversion::Base
|
||||
def initialize(settings)
|
||||
super
|
||||
@source_db = Database::Adapter::Postgres.new(settings[:source_db])
|
||||
@source_db = Adapter::Postgres.new(settings[:source_db])
|
||||
end
|
||||
|
||||
def step_args(step_class)
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class BadgeGroupings < Base::ProgressStep
|
||||
class BadgeGroupings < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class Badges < Base::ProgressStep
|
||||
class Badges < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def execute
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class Categories < Base::ProgressStep
|
||||
class Categories < Conversion::ProgressStep
|
||||
SEEDED_CATEGORY_SETTINGS = %w[
|
||||
uncategorized_category_id
|
||||
meta_category_id
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class CategoryCustomFields < Base::ProgressStep
|
||||
class CategoryCustomFields < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class CategoryModerationGroups < Base::ProgressStep
|
||||
class CategoryModerationGroups < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class CategoryUsers < Base::ProgressStep
|
||||
class CategoryUsers < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class GroupUsers < Base::ProgressStep
|
||||
class GroupUsers < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class Groups < Base::ProgressStep
|
||||
class Groups < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def execute
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class MutedUsers < Base::ProgressStep
|
||||
class MutedUsers < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class PermalinkNormalizations < Base::Step
|
||||
class PermalinkNormalizations < Conversion::Step
|
||||
attr_accessor :source_db
|
||||
|
||||
def execute
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class SiteSettings < Base::ProgressStep
|
||||
class SiteSettings < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def execute
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class TagGroupMemberships < Base::ProgressStep
|
||||
class TagGroupMemberships < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class TagGroupPermissions < Base::ProgressStep
|
||||
class TagGroupPermissions < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class TagGroups < Base::ProgressStep
|
||||
class TagGroups < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class TagUsers < Base::ProgressStep
|
||||
class TagUsers < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class Tags < Base::ProgressStep
|
||||
class Tags < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class TopicAllowedGroups < Base::ProgressStep
|
||||
class TopicAllowedGroups < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class TopicAllowedUsers < Base::ProgressStep
|
||||
class TopicAllowedUsers < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class TopicTags < Base::ProgressStep
|
||||
class TopicTags < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class TopicUsers < Base::ProgressStep
|
||||
class TopicUsers < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class Topics < Base::ProgressStep
|
||||
class Topics < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class UserAssociatedAccounts < Base::ProgressStep
|
||||
class UserAssociatedAccounts < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class UserEmails < Base::ProgressStep
|
||||
class UserEmails < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class UserFieldOptions < Base::ProgressStep
|
||||
class UserFieldOptions < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class UserFieldValues < Base::ProgressStep
|
||||
class UserFieldValues < Conversion::ProgressStep
|
||||
USER_FIELD_PREFIX = "user_field_"
|
||||
|
||||
attr_accessor :source_db
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class UserFields < Base::ProgressStep
|
||||
class UserFields < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class UserOptions < Base::ProgressStep
|
||||
class UserOptions < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class UserSuspensions < Base::ProgressStep
|
||||
class UserSuspensions < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def max_progress
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
module Migrations
|
||||
module Converters
|
||||
module Discourse
|
||||
class Users < Base::ProgressStep
|
||||
class Users < Conversion::ProgressStep
|
||||
attr_accessor :source_db
|
||||
|
||||
def execute
|
||||
@@ -0,0 +1,7 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
Migrations::CLI::Registry.register(
|
||||
name: "convert",
|
||||
command_class: "Migrations::Converters::CLI::ConvertCommand",
|
||||
description: "Convert a source dump into the IntermediateDB",
|
||||
)
|
||||
@@ -0,0 +1,18 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
Gem::Specification.new do |s|
|
||||
s.name = "migrations-converters"
|
||||
s.version = "0.0.1"
|
||||
s.summary = "Discourse migrations: public converter implementations"
|
||||
s.authors = ["Discourse Team"]
|
||||
s.required_ruby_version = ">= 3.4"
|
||||
|
||||
s.files = Dir["lib/**/*"]
|
||||
|
||||
s.add_dependency "migrations-core"
|
||||
s.add_dependency "activesupport"
|
||||
s.add_dependency "colored2"
|
||||
s.add_dependency "i18n"
|
||||
s.add_dependency "pg"
|
||||
s.add_dependency "zeitwerk"
|
||||
end
|
||||
+23
-24
@@ -2,11 +2,12 @@
|
||||
|
||||
RSpec.describe Migrations::Converters do
|
||||
let(:root_path) { Dir.mktmpdir }
|
||||
let(:core_path) { File.join(root_path, "lib", "converters") }
|
||||
let(:converters_path) { File.join(root_path, "converters") }
|
||||
let(:private_path) { File.join(root_path, "private", "converters") }
|
||||
|
||||
before do
|
||||
allow(Migrations).to receive(:root_path).and_return(root_path)
|
||||
allow(described_class).to receive(:converters_path).and_return(converters_path)
|
||||
allow(described_class).to receive(:private_converters_path).and_return(private_path)
|
||||
reset_memoization(described_class, :@all_converters)
|
||||
end
|
||||
after do
|
||||
@@ -14,29 +15,27 @@ RSpec.describe Migrations::Converters do
|
||||
reset_memoization(described_class, :@all_converters)
|
||||
end
|
||||
|
||||
def create_converters(core_names: [], private_names: [])
|
||||
core_names.each { |dir| FileUtils.mkdir_p(File.join(core_path, dir)) }
|
||||
def create_converters(public_names: [], private_names: [])
|
||||
public_names.each { |dir| FileUtils.mkdir_p(File.join(converters_path, dir)) }
|
||||
private_names.each { |dir| FileUtils.mkdir_p(File.join(private_path, dir)) }
|
||||
end
|
||||
|
||||
describe ".all" do
|
||||
subject(:all) { described_class.all }
|
||||
|
||||
it "returns all the converters except for 'base'" do
|
||||
create_converters(core_names: %w[base foo bar])
|
||||
it "excludes the framework infrastructure directories" do
|
||||
create_converters(public_names: described_class::NON_CONVERTER_DIRS + %w[foo bar])
|
||||
|
||||
expect(all).to eq(
|
||||
{ "foo" => File.join(core_path, "foo"), "bar" => File.join(core_path, "bar") },
|
||||
)
|
||||
expect(all.keys).to contain_exactly("foo", "bar")
|
||||
end
|
||||
|
||||
it "returns converters from core and private directory" do
|
||||
create_converters(core_names: %w[base foo bar], private_names: %w[baz qux])
|
||||
it "returns converters from the gem and the private directory" do
|
||||
create_converters(public_names: %w[foo bar], private_names: %w[baz qux])
|
||||
|
||||
expect(all).to eq(
|
||||
{
|
||||
"foo" => File.join(core_path, "foo"),
|
||||
"bar" => File.join(core_path, "bar"),
|
||||
"foo" => File.join(converters_path, "foo"),
|
||||
"bar" => File.join(converters_path, "bar"),
|
||||
"baz" => File.join(private_path, "baz"),
|
||||
"qux" => File.join(private_path, "qux"),
|
||||
},
|
||||
@@ -44,7 +43,7 @@ RSpec.describe Migrations::Converters do
|
||||
end
|
||||
|
||||
it "raises an error if there a duplicate names" do
|
||||
create_converters(core_names: %w[base foo bar], private_names: %w[foo baz qux])
|
||||
create_converters(public_names: %w[foo bar], private_names: %w[foo baz qux])
|
||||
|
||||
expect { all }.to raise_error(StandardError, /Duplicate converter name found: foo/)
|
||||
end
|
||||
@@ -54,7 +53,7 @@ RSpec.describe Migrations::Converters do
|
||||
subject(:names) { described_class.names }
|
||||
|
||||
it "returns a sorted array of converter names" do
|
||||
create_converters(core_names: %w[base foo bar], private_names: %w[baz qux])
|
||||
create_converters(public_names: %w[adapter foo bar], private_names: %w[baz qux])
|
||||
|
||||
expect(names).to eq(%w[bar baz foo qux])
|
||||
end
|
||||
@@ -62,39 +61,39 @@ RSpec.describe Migrations::Converters do
|
||||
|
||||
describe ".path_of" do
|
||||
it "returns the path of a converter" do
|
||||
create_converters(core_names: %w[base foo bar])
|
||||
create_converters(public_names: %w[adapter foo bar])
|
||||
|
||||
expect(described_class.path_of("foo")).to eq(File.join(core_path, "foo"))
|
||||
expect(described_class.path_of("foo")).to eq(File.join(converters_path, "foo"))
|
||||
end
|
||||
|
||||
it "raises an error if there is no converter" do
|
||||
create_converters(core_names: %w[base foo bar])
|
||||
create_converters(public_names: %w[adapter foo bar])
|
||||
|
||||
expect { described_class.path_of("baz") }.to raise_error(
|
||||
StandardError,
|
||||
"Could not find a converter named 'baz'",
|
||||
)
|
||||
expect { described_class.path_of("base") }.to raise_error(
|
||||
expect { described_class.path_of("adapter") }.to raise_error(
|
||||
StandardError,
|
||||
"Could not find a converter named 'base'",
|
||||
"Could not find a converter named 'adapter'",
|
||||
)
|
||||
end
|
||||
end
|
||||
|
||||
describe ".default_settings_path" do
|
||||
it "returns the path of the default settings file" do
|
||||
create_converters(core_names: %w[foo bar])
|
||||
create_converters(public_names: %w[foo bar])
|
||||
|
||||
expect(described_class.default_settings_path("foo")).to eq(
|
||||
File.join(core_path, "foo", "settings.yml"),
|
||||
File.join(converters_path, "foo", "settings.yml"),
|
||||
)
|
||||
expect(described_class.default_settings_path("bar")).to eq(
|
||||
File.join(core_path, "bar", "settings.yml"),
|
||||
File.join(converters_path, "bar", "settings.yml"),
|
||||
)
|
||||
end
|
||||
|
||||
it "raises an error if there is no converter" do
|
||||
create_converters(core_names: %w[foo bar])
|
||||
create_converters(public_names: %w[foo bar])
|
||||
|
||||
expect { described_class.default_settings_path("baz") }.to raise_error(
|
||||
StandardError,
|
||||
@@ -0,0 +1,5 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require_relative "../../core/spec/spec_setup"
|
||||
|
||||
MigrationsSpecSetup.call(gem: "migrations-converters", spec_dir: __dir__)
|
||||
@@ -0,0 +1,2 @@
|
||||
--require spec_helper
|
||||
--format documentation
|
||||
@@ -0,0 +1,11 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
source "https://rubygems.org"
|
||||
|
||||
gemspec
|
||||
|
||||
group :test do
|
||||
gem "rspec"
|
||||
gem "rspec-multi-mock"
|
||||
gem "mocha"
|
||||
end
|
||||
Executable
+34
@@ -0,0 +1,34 @@
|
||||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../../Gemfile", __dir__)
|
||||
|
||||
require "bundler"
|
||||
Bundler.setup(:default, :migrations)
|
||||
|
||||
require "colored2"
|
||||
|
||||
require "migrations-core"
|
||||
require "migrations-tooling"
|
||||
require "migrations-converters"
|
||||
require "migrations-importer"
|
||||
|
||||
Migrations.enable_i18n
|
||||
Migrations.apply_global_config
|
||||
|
||||
# Each gem registers its top-level commands into Migrations::CLI::Registry.
|
||||
require "migrations/tooling/register"
|
||||
require "migrations/converters/register"
|
||||
require "migrations/importer/register"
|
||||
|
||||
if defined?(RubyVM::YJIT)
|
||||
RubyVM::YJIT.enable
|
||||
else
|
||||
warn "WARNING: Performance degraded: RubyVM::YJIT is not available".yellow
|
||||
end
|
||||
|
||||
# rubocop:disable Discourse/NoChdir
|
||||
Dir.chdir(Migrations.host_app_root) do
|
||||
Migrations::CLI::ExceptionHandler.handle_and_exit { Migrations::CLI::Bootstrap.run(ARGV) }
|
||||
end
|
||||
# rubocop:enable Discourse/NoChdir
|
||||
-11
@@ -18,14 +18,3 @@ en:
|
||||
converter:
|
||||
default_step_title: "Converting %{type}"
|
||||
max_progress_calculation: "Calculating items took %{duration}"
|
||||
|
||||
importer:
|
||||
default_step_title: "Importing %{type}"
|
||||
loading_required_data: "Loading required data..."
|
||||
done: "Done. Total runtime: %{runtime}"
|
||||
site_setting_log_message: "Updated by import script"
|
||||
fallback_names:
|
||||
user: "user"
|
||||
group: "group"
|
||||
category: "Category"
|
||||
badge: "Badge"
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
-- This file is auto-generated from the IntermediateDB schema. To make changes,
|
||||
-- update the configuration files in "migrations/config/schema/" and then run
|
||||
-- `migrations/bin/cli schema generate` to regenerate this file.
|
||||
-- update the configuration files in "migrations/tooling/config/schema/" and then run
|
||||
-- `migrations/bin/disco schema generate` to regenerate this file.
|
||||
|
||||
CREATE TABLE badge_groupings
|
||||
(
|
||||
@@ -0,0 +1,3 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require_relative "migrations"
|
||||
@@ -0,0 +1,107 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require "active_support"
|
||||
require "active_support/core_ext"
|
||||
require "zeitwerk"
|
||||
|
||||
module Migrations
|
||||
class NoSettingsFound < StandardError
|
||||
end
|
||||
|
||||
# Each gem registers its `config/locales` directory so that `enable_i18n`
|
||||
# loads the union of all translations (and an isolated gem still sees its own).
|
||||
def self.locale_load_paths
|
||||
@locale_load_paths ||= []
|
||||
end
|
||||
|
||||
def self.register_locale_path(dir)
|
||||
locale_load_paths << dir if locale_load_paths.exclude?(dir)
|
||||
end
|
||||
|
||||
# Root of the `migrations-core` gem. Other gems expose their own root.
|
||||
def self.root_path
|
||||
@root_path ||= File.expand_path("..", __dir__)
|
||||
end
|
||||
|
||||
# Root of the host Discourse application (the repository root). Used to lazily
|
||||
# boot Rails and to discover private converters.
|
||||
def self.host_app_root
|
||||
@host_app_root ||= File.expand_path("../../..", __dir__)
|
||||
end
|
||||
|
||||
def self.load_rails_environment(quiet: false)
|
||||
message = "Loading Rails environment..."
|
||||
print message if !quiet
|
||||
|
||||
rails_root = host_app_root
|
||||
# rubocop:disable Discourse/NoChdir
|
||||
Dir.chdir(rails_root) do
|
||||
ENV["DISCOURSE_DEV_ALLOW_HTTPS"] = "1" # suppress warning
|
||||
require File.join(rails_root, "config/environment")
|
||||
rescue LoadError => e
|
||||
$stderr.puts e.message
|
||||
raise
|
||||
end
|
||||
# rubocop:enable Discourse/NoChdir
|
||||
|
||||
if !quiet
|
||||
print "\r"
|
||||
print " " * message.length
|
||||
print "\r"
|
||||
end
|
||||
end
|
||||
|
||||
def self.loader
|
||||
@loader ||=
|
||||
begin
|
||||
loader = Zeitwerk::Loader.new
|
||||
loader.log! if ENV["DEBUG"]
|
||||
configure_inflections(loader)
|
||||
loader.push_dir(File.join(__dir__, "migrations"), namespace: Migrations)
|
||||
configure_collapses(loader)
|
||||
loader
|
||||
end
|
||||
end
|
||||
|
||||
def self.configure_inflections(loader)
|
||||
loader.inflector.inflect(
|
||||
{
|
||||
"cli" => "CLI",
|
||||
"id" => "ID",
|
||||
"intermediate_db" => "IntermediateDB",
|
||||
"mappings_db" => "MappingsDB",
|
||||
"uploads_db" => "UploadsDB",
|
||||
},
|
||||
)
|
||||
end
|
||||
|
||||
# Collapse `common/` into the root `Migrations` namespace, matching the previous
|
||||
# flat layout (e.g. `common/enum.rb` => `Migrations::Enum`). Nested directories
|
||||
# such as `common/set_store/` keep contributing a namespace segment
|
||||
# (`Migrations::SetStore::*`).
|
||||
def self.configure_collapses(loader)
|
||||
loader.collapse(File.join(__dir__, "migrations", "common"))
|
||||
end
|
||||
|
||||
def self.setup_loader
|
||||
loader.setup
|
||||
end
|
||||
|
||||
def self.enable_i18n
|
||||
require "i18n"
|
||||
|
||||
locale_load_paths.each { |dir| I18n.load_path += Dir[File.join(dir, "**", "migrations.*.yml")] }
|
||||
I18n.backend.load_translations
|
||||
|
||||
# always use English for now
|
||||
I18n.default_locale = :en
|
||||
I18n.locale = :en
|
||||
end
|
||||
|
||||
def self.apply_global_config
|
||||
Regexp.timeout = 2
|
||||
end
|
||||
end
|
||||
|
||||
Migrations.register_locale_path(File.join(Migrations.root_path, "config", "locales"))
|
||||
Migrations.setup_loader
|
||||
@@ -0,0 +1,64 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require "samovar"
|
||||
|
||||
module Migrations
|
||||
module CLI
|
||||
# Builds the Samovar command tree from the registry, selects the command for
|
||||
# the given argv, lazily boots Rails when the selected command requires it,
|
||||
# and dispatches.
|
||||
module Bootstrap
|
||||
def self.run(argv)
|
||||
top = build_top_command_class.new(normalize_option_args(argv))
|
||||
|
||||
leaf = deepest_command(top)
|
||||
if leaf.class.respond_to?(:requires_rails?) && leaf.class.requires_rails?
|
||||
Migrations.load_rails_environment(quiet: true)
|
||||
end
|
||||
|
||||
top.call
|
||||
rescue Samovar::Error => e
|
||||
e.command.print_usage
|
||||
exit(1)
|
||||
end
|
||||
|
||||
# Samovar expects `--opt value`; the previous Thor-based CLI also accepted
|
||||
# `--opt=value`. Split the `=` form into two tokens so existing muscle
|
||||
# memory keeps working. Only long options are touched; `--`, short flags,
|
||||
# and bare positional values pass through unchanged.
|
||||
def self.normalize_option_args(argv)
|
||||
argv.flat_map do |arg|
|
||||
if arg.start_with?("--") && (index = arg.index("=")) && index > 2
|
||||
[arg[0...index], arg[(index + 1)..]]
|
||||
else
|
||||
arg
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# Walks the nested `command` chain to the deepest selected sub-command.
|
||||
def self.deepest_command(command)
|
||||
node = command
|
||||
node = node.command while node.respond_to?(:command) && node.command
|
||||
node
|
||||
end
|
||||
|
||||
def self.build_top_command_class
|
||||
commands = Registry.command_classes
|
||||
|
||||
Class.new(Command) do
|
||||
self.description = "Discourse migration tools"
|
||||
nested :command, commands
|
||||
|
||||
def call
|
||||
if @command
|
||||
@command.call
|
||||
else
|
||||
print_usage
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,83 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require "samovar"
|
||||
|
||||
module Migrations
|
||||
module CLI
|
||||
# Base class for all `disco` commands. Commands that need a booted Rails
|
||||
# environment declare `requires_rails!`; the binary boots Rails only after
|
||||
# such a command has been selected, keeping help and Rails-free commands
|
||||
# fast.
|
||||
class Command < Samovar::Command
|
||||
# Coerces a comma-separated `--only`/`--skip` value into a list of
|
||||
# normalized step names. Shared as the `type:` for those options.
|
||||
STEP_LIST = ->(value) do
|
||||
value.to_s.split(",").map { |name| name.strip.demodulize.underscore }
|
||||
end
|
||||
|
||||
def self.requires_rails!
|
||||
@requires_rails = true
|
||||
end
|
||||
|
||||
def self.requires_rails?
|
||||
return true if @requires_rails == true
|
||||
superclass.respond_to?(:requires_rails?) && superclass.requires_rails?
|
||||
end
|
||||
|
||||
# Samovar parses each declaration once, front-to-back, and only matches
|
||||
# options at the front of the remaining input — so by default options must
|
||||
# precede positionals (`convert --only x discourse`). Hoist any recognized
|
||||
# option flags (and their values) to the front so they may also appear
|
||||
# after positionals (`convert discourse --only x`), the way the previous
|
||||
# Thor-based CLI allowed. Everything after a `--` separator is left as-is.
|
||||
def parse(input)
|
||||
# Reorder in place: Samovar consumes the input array by reference (nested
|
||||
# commands rely on it being emptied), so we must mutate it rather than
|
||||
# pass a copy.
|
||||
input.replace(hoist_options(input))
|
||||
super(input)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def hoist_options(input)
|
||||
options = self.class.table.merged[:options]
|
||||
return input unless options
|
||||
|
||||
takes_value = {}
|
||||
options.each do |option|
|
||||
option.flags.each do |flag|
|
||||
[flag.prefix, *Array(flag.alternatives)].each do |prefix|
|
||||
takes_value[prefix] = !flag.boolean?
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
flags = []
|
||||
positionals = []
|
||||
index = 0
|
||||
|
||||
while index < input.size
|
||||
token = input[index]
|
||||
|
||||
if token == "--"
|
||||
positionals.concat(input[index..])
|
||||
break
|
||||
elsif takes_value.key?(token)
|
||||
flags << token
|
||||
if takes_value[token] && index + 1 < input.size
|
||||
flags << input[index + 1]
|
||||
index += 1
|
||||
end
|
||||
else
|
||||
positionals << token
|
||||
end
|
||||
|
||||
index += 1
|
||||
end
|
||||
|
||||
flags + positionals
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
+1
-1
@@ -8,7 +8,7 @@ module Migrations
|
||||
rescue ClassFilter::UnknownClassNamesError => e
|
||||
handle_unknown_class_names_error(e)
|
||||
exit(1)
|
||||
rescue Database::Schema::ConfigError, Database::Schema::GenerationError => e
|
||||
rescue PresentableError => e
|
||||
puts e.message.red
|
||||
exit(1)
|
||||
rescue => e
|
||||
@@ -0,0 +1,10 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module CLI
|
||||
# Errors including this module are shown to the user as a clean, red message
|
||||
# (no backtrace) by the ExceptionHandler.
|
||||
module PresentableError
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,38 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module CLI
|
||||
# Central registry of top-level `disco` commands. Each gem requires its
|
||||
# `register.rb` at startup, which pushes its commands here. The Samovar
|
||||
# command tree is built from this registry just before argv is parsed.
|
||||
module Registry
|
||||
Entry = Struct.new(:name, :command_class, :description, keyword_init: true)
|
||||
|
||||
def self.entries
|
||||
@entries ||= {}
|
||||
end
|
||||
|
||||
def self.register(name:, command_class:, description: nil)
|
||||
name = name.to_s
|
||||
raise "A `disco` command named '#{name}' is already registered" if entries.key?(name)
|
||||
entries[name] = Entry.new(name:, command_class:, description:)
|
||||
end
|
||||
|
||||
def self.reset!
|
||||
@entries = {}
|
||||
end
|
||||
|
||||
# Resolves the registered command classes (stored as strings to keep Rails
|
||||
# lazy) into a name => Class hash, ordered by registration name.
|
||||
def self.command_classes
|
||||
entries
|
||||
.sort_by { |name, _| name }
|
||||
.each_with_object({}) do |(name, entry), hash|
|
||||
klass = entry.command_class
|
||||
klass = klass.to_s.constantize if klass.is_a?(String) || klass.is_a?(Symbol)
|
||||
hash[name] = klass
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,89 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module Conversion
|
||||
class Base
|
||||
attr_accessor :settings
|
||||
|
||||
def initialize(settings)
|
||||
@settings = settings
|
||||
end
|
||||
|
||||
def run(only_steps: [], skip_steps: [])
|
||||
if respond_to?(:setup)
|
||||
puts "Initializing..."
|
||||
setup
|
||||
end
|
||||
|
||||
create_database
|
||||
|
||||
filter_steps(steps, only_steps, skip_steps).each do |step_class|
|
||||
step = create_step(step_class)
|
||||
before_step_execution(step)
|
||||
execute_step(step)
|
||||
after_step_execution(step)
|
||||
end
|
||||
rescue SignalException
|
||||
STDERR.puts "\nAborted"
|
||||
exit(1)
|
||||
ensure
|
||||
Database::IntermediateDB.close
|
||||
end
|
||||
|
||||
def steps
|
||||
step_class = Step
|
||||
current_module = self.class.name.deconstantize.constantize
|
||||
|
||||
current_module
|
||||
.constants
|
||||
.map { |c| current_module.const_get(c) }
|
||||
.select { |klass| klass.is_a?(Class) && klass < step_class }
|
||||
.sort_by(&:to_s)
|
||||
end
|
||||
|
||||
def before_step_execution(step)
|
||||
# do nothing
|
||||
end
|
||||
|
||||
def execute_step(step)
|
||||
executor =
|
||||
if step.is_a?(ProgressStep)
|
||||
ProgressStepExecutor
|
||||
else
|
||||
StepExecutor
|
||||
end
|
||||
|
||||
executor.new(step).execute
|
||||
end
|
||||
|
||||
def after_step_execution(step)
|
||||
# do nothing
|
||||
end
|
||||
|
||||
def step_args(step_class)
|
||||
{}
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def create_database
|
||||
db_path = File.expand_path(settings[:intermediate_db][:path], Migrations.root_path)
|
||||
Database.migrate(db_path, migrations_path: Database::INTERMEDIATE_DB_SCHEMA_PATH)
|
||||
|
||||
db = Database.connect(db_path)
|
||||
Database::IntermediateDB.setup(db)
|
||||
end
|
||||
|
||||
def create_step(step_class)
|
||||
default_args = { settings: }
|
||||
|
||||
args = default_args.merge(step_args(step_class))
|
||||
step_class.new(StepTracker.new, args)
|
||||
end
|
||||
|
||||
def filter_steps(step_classes, only_steps, skip_steps)
|
||||
ClassFilter.filter(step_classes, only: only_steps, skip: skip_steps)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,32 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module Conversion
|
||||
class ParallelJob
|
||||
def initialize(step)
|
||||
@step = step
|
||||
@tracker = step.tracker
|
||||
|
||||
@offline_connection = Database::OfflineConnection.new
|
||||
|
||||
ForkManager.after_fork_child { Database::IntermediateDB.setup(@offline_connection) }
|
||||
end
|
||||
|
||||
def run(item)
|
||||
@tracker.reset_stats!
|
||||
@offline_connection.clear!
|
||||
|
||||
begin
|
||||
@step.process_item(item)
|
||||
rescue StandardError => e
|
||||
@tracker.log_error("Failed to process item", exception: e, details: item)
|
||||
end
|
||||
|
||||
[@offline_connection.parametrized_insert_statements, @tracker.stats]
|
||||
end
|
||||
|
||||
def cleanup
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,29 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module Conversion
|
||||
class ProgressStep < Step
|
||||
def max_progress
|
||||
nil
|
||||
end
|
||||
|
||||
def items
|
||||
raise NotImplementedError
|
||||
end
|
||||
|
||||
def process_item(item)
|
||||
raise NotImplementedError
|
||||
end
|
||||
|
||||
class << self
|
||||
def run_in_parallel(value)
|
||||
@run_in_parallel = !!value
|
||||
end
|
||||
|
||||
def run_in_parallel?
|
||||
@run_in_parallel == true
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,127 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require "etc"
|
||||
require "colored2"
|
||||
|
||||
module Migrations
|
||||
module Conversion
|
||||
class ProgressStepExecutor
|
||||
WORKER_COUNT = Etc.nprocessors - 1 # leave 1 CPU free to do other work
|
||||
MIN_PARALLEL_ITEMS = WORKER_COUNT * 10
|
||||
MAX_QUEUE_SIZE = WORKER_COUNT * 100
|
||||
PRINT_RUNTIME_AFTER_SECONDS = 5
|
||||
|
||||
def initialize(step)
|
||||
@step = step
|
||||
end
|
||||
|
||||
def execute
|
||||
@max_progress = calculate_max_progress
|
||||
|
||||
puts @step.class.title
|
||||
@step.execute
|
||||
|
||||
if execute_in_parallel?
|
||||
execute_parallel
|
||||
else
|
||||
execute_serially
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def execute_in_parallel?
|
||||
@step.class.run_in_parallel? && (@max_progress.nil? || @max_progress > MIN_PARALLEL_ITEMS)
|
||||
end
|
||||
|
||||
def execute_serially
|
||||
job = SerialJob.new(@step)
|
||||
|
||||
with_progressbar do |progressbar|
|
||||
@step.items.each do |item|
|
||||
stats = job.run(item)
|
||||
progressbar.update(
|
||||
increment_by: stats.progress,
|
||||
warning_count: stats.warning_count,
|
||||
error_count: stats.error_count,
|
||||
)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def execute_parallel
|
||||
worker_output_queue = SizedQueue.new(MAX_QUEUE_SIZE)
|
||||
work_queue = SizedQueue.new(MAX_QUEUE_SIZE)
|
||||
|
||||
workers = start_workers(work_queue, worker_output_queue)
|
||||
writer_thread = start_db_writer(worker_output_queue)
|
||||
push_work(work_queue)
|
||||
|
||||
workers.each(&:wait)
|
||||
worker_output_queue.close
|
||||
writer_thread.join
|
||||
end
|
||||
|
||||
def calculate_max_progress
|
||||
start_time = Time.now
|
||||
max_progress = @step.max_progress
|
||||
duration = Time.now - start_time
|
||||
|
||||
if duration > PRINT_RUNTIME_AFTER_SECONDS
|
||||
message =
|
||||
I18n.t(
|
||||
"converter.max_progress_calculation",
|
||||
duration: DateHelper.human_readable_time(duration),
|
||||
)
|
||||
puts " #{message}"
|
||||
end
|
||||
|
||||
max_progress
|
||||
end
|
||||
|
||||
def with_progressbar
|
||||
ExtendedProgressBar.new(max_progress: @max_progress).run { |progressbar| yield progressbar }
|
||||
end
|
||||
|
||||
def start_db_writer(worker_output_queue)
|
||||
Thread.new do
|
||||
Thread.current.name = "writer_thread"
|
||||
|
||||
with_progressbar do |progressbar|
|
||||
while (parametrized_insert_statements, stats = worker_output_queue.pop)
|
||||
parametrized_insert_statements.each do |sql, parameters|
|
||||
Database::IntermediateDB.insert(sql, *parameters)
|
||||
end
|
||||
|
||||
progressbar.update(
|
||||
increment_by: stats.progress,
|
||||
warning_count: stats.warning_count,
|
||||
error_count: stats.error_count,
|
||||
)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def start_workers(work_queue, worker_output_queue)
|
||||
workers = []
|
||||
|
||||
Process.warmup
|
||||
|
||||
ForkManager.batch_forks do
|
||||
WORKER_COUNT.times do |index|
|
||||
job = ParallelJob.new(@step)
|
||||
workers << Worker.new(index, work_queue, worker_output_queue, job).start
|
||||
end
|
||||
end
|
||||
|
||||
workers
|
||||
end
|
||||
|
||||
def push_work(work_queue)
|
||||
@step.items.each { |item| work_queue.push(item) }
|
||||
work_queue.close
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,27 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module Conversion
|
||||
class SerialJob
|
||||
def initialize(step)
|
||||
@step = step
|
||||
@tracker = step.tracker
|
||||
end
|
||||
|
||||
def run(item)
|
||||
@tracker.reset_stats!
|
||||
|
||||
begin
|
||||
@step.process_item(item)
|
||||
rescue StandardError => e
|
||||
@tracker.log_error("Failed to process item", exception: e, details: item)
|
||||
end
|
||||
|
||||
@tracker.stats
|
||||
end
|
||||
|
||||
def cleanup
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,45 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module Conversion
|
||||
class Step
|
||||
IntermediateDB = Database::IntermediateDB
|
||||
Enums = Database::IntermediateDB::Enums
|
||||
|
||||
attr_accessor :settings
|
||||
attr_reader :tracker
|
||||
|
||||
# inside of Step it might make more sense to access it as `step` instead of `tracker`
|
||||
alias step tracker
|
||||
|
||||
def initialize(tracker, args = {})
|
||||
@tracker = tracker
|
||||
|
||||
args.each do |arg, value|
|
||||
setter = :"#{arg}="
|
||||
public_send(setter, value) if respond_to?(setter, true)
|
||||
end
|
||||
end
|
||||
|
||||
def execute
|
||||
# do nothing
|
||||
end
|
||||
|
||||
class << self
|
||||
def title(
|
||||
value = (
|
||||
getter = true
|
||||
nil
|
||||
)
|
||||
)
|
||||
@title = value unless getter
|
||||
@title.presence ||
|
||||
I18n.t(
|
||||
"converter.default_step_title",
|
||||
type: name&.demodulize&.underscore&.humanize(capitalize: false),
|
||||
)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,16 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module Conversion
|
||||
class StepExecutor
|
||||
def initialize(step)
|
||||
@step = step
|
||||
end
|
||||
|
||||
def execute
|
||||
puts @step.class.title
|
||||
@step.execute
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,7 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module Conversion
|
||||
StepStats = Struct.new(:progress, :warning_count, :error_count)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,44 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
module Migrations
|
||||
module Conversion
|
||||
class StepTracker
|
||||
attr_reader :stats
|
||||
|
||||
def initialize
|
||||
@stats = StepStats.new
|
||||
reset_stats!
|
||||
end
|
||||
|
||||
def reset_stats!
|
||||
@stats.progress = 1
|
||||
@stats.warning_count = 0
|
||||
@stats.error_count = 0
|
||||
end
|
||||
|
||||
def progress=(value)
|
||||
@stats.progress = value
|
||||
end
|
||||
|
||||
def log_info(message, details: nil)
|
||||
log(Database::IntermediateDB::LogEntry::INFO, message, details:)
|
||||
end
|
||||
|
||||
def log_warning(message, exception: nil, details: nil)
|
||||
@stats.warning_count += 1
|
||||
log(Database::IntermediateDB::LogEntry::WARNING, message, exception:, details:)
|
||||
end
|
||||
|
||||
def log_error(message, exception: nil, details: nil)
|
||||
@stats.error_count += 1
|
||||
log(Database::IntermediateDB::LogEntry::ERROR, message, exception:, details:)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def log(type, message, exception: nil, details: nil)
|
||||
Database::IntermediateDB::LogEntry.create(type:, message:, exception:, details:)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,104 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require "oj"
|
||||
|
||||
module Migrations
|
||||
module Conversion
|
||||
class Worker
|
||||
OJ_SETTINGS = { mode: :object, class_cache: true, symbol_keys: true }
|
||||
|
||||
def initialize(index, input_queue, output_queue, job)
|
||||
@index = index
|
||||
@input_queue = input_queue
|
||||
@output_queue = output_queue
|
||||
@job = job
|
||||
|
||||
@threads = []
|
||||
@mutex = Mutex.new
|
||||
@data_processed = ConditionVariable.new
|
||||
end
|
||||
|
||||
def start
|
||||
parent_input_stream, parent_output_stream = IO.pipe
|
||||
fork_input_stream, fork_output_stream = IO.pipe
|
||||
|
||||
worker_pid =
|
||||
start_fork(
|
||||
parent_input_stream,
|
||||
parent_output_stream,
|
||||
fork_input_stream,
|
||||
fork_output_stream,
|
||||
)
|
||||
|
||||
fork_output_stream.close
|
||||
parent_input_stream.close
|
||||
|
||||
start_input_thread(parent_output_stream, worker_pid)
|
||||
start_output_thread(fork_input_stream)
|
||||
|
||||
self
|
||||
end
|
||||
|
||||
def wait
|
||||
@threads.each(&:join)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def start_fork(
|
||||
parent_input_stream,
|
||||
parent_output_stream,
|
||||
fork_input_stream,
|
||||
fork_output_stream
|
||||
)
|
||||
ForkManager.fork do
|
||||
Process.setproctitle("worker_process#{@index}")
|
||||
|
||||
parent_output_stream.close
|
||||
fork_input_stream.close
|
||||
|
||||
Oj.load(parent_input_stream, OJ_SETTINGS) do |data|
|
||||
result = @job.run(data)
|
||||
Oj.to_stream(fork_output_stream, result, OJ_SETTINGS)
|
||||
end
|
||||
rescue SignalException
|
||||
exit(1)
|
||||
ensure
|
||||
@job.cleanup
|
||||
end
|
||||
end
|
||||
|
||||
def start_input_thread(output_stream, worker_pid)
|
||||
@threads << Thread.new do
|
||||
Thread.current.name = "worker_#{@index}_input"
|
||||
|
||||
begin
|
||||
while (data = @input_queue.pop)
|
||||
Oj.to_stream(output_stream, data, OJ_SETTINGS)
|
||||
@mutex.synchronize { @data_processed.wait(@mutex) }
|
||||
end
|
||||
ensure
|
||||
output_stream.close
|
||||
Process.waitpid(worker_pid)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def start_output_thread(input_stream)
|
||||
@threads << Thread.new do
|
||||
Thread.current.name = "worker_#{@index}_output"
|
||||
|
||||
begin
|
||||
Oj.load(input_stream, OJ_SETTINGS) do |data|
|
||||
@output_queue.push(data)
|
||||
@mutex.synchronize { @data_processed.signal }
|
||||
end
|
||||
ensure
|
||||
input_stream.close
|
||||
@mutex.synchronize { @data_processed.signal }
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
# This file is auto-generated from the IntermediateDB schema. To make changes,
|
||||
# update the configuration files in "migrations/config/schema/" and then run
|
||||
# `migrations/bin/cli schema generate` to regenerate this file.
|
||||
# update the configuration files in "migrations/tooling/config/schema/" and then run
|
||||
# `migrations/bin/disco schema generate` to regenerate this file.
|
||||
|
||||
module Migrations
|
||||
module Database
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
# This file is auto-generated from the IntermediateDB schema. To make changes,
|
||||
# update the configuration files in "migrations/config/schema/" and then run
|
||||
# `migrations/bin/cli schema generate` to regenerate this file.
|
||||
# update the configuration files in "migrations/tooling/config/schema/" and then run
|
||||
# `migrations/bin/disco schema generate` to regenerate this file.
|
||||
|
||||
module Migrations
|
||||
module Database
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
# This file is auto-generated from the IntermediateDB schema. To make changes,
|
||||
# update the configuration files in "migrations/config/schema/" and then run
|
||||
# `migrations/bin/cli schema generate` to regenerate this file.
|
||||
# update the configuration files in "migrations/tooling/config/schema/" and then run
|
||||
# `migrations/bin/disco schema generate` to regenerate this file.
|
||||
|
||||
module Migrations
|
||||
module Database
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
# This file is auto-generated from the IntermediateDB schema. To make changes,
|
||||
# update the configuration files in "migrations/config/schema/" and then run
|
||||
# `migrations/bin/cli schema generate` to regenerate this file.
|
||||
# update the configuration files in "migrations/tooling/config/schema/" and then run
|
||||
# `migrations/bin/disco schema generate` to regenerate this file.
|
||||
|
||||
module Migrations
|
||||
module Database
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user