Skip to content

Migration Engine: Copy the remaining required categories to SQL - #6156

Open
warwickschroeder wants to merge 2 commits into
migration-engine-7from
migration-engine-8
Open

warwickschroeder wants to merge 2 commits into
migration-engine-7from
migration-engine-8

Conversation

@warwickschroeder

Copy link
Copy Markdown
Contributor

--start-data-migration copies the last four required categories from RavenDB to SQL: pending integration events, custom checks, failed error imports with their bodies, and group comments. With them, the offline copy covers all sixteen required categories.

What this adds

  • Pending integration events, custom checks, failed error imports and group comments are copied by --start-data-migration.
  • A failed import body over MaxBodySizeToStore goes to body storage before the transaction opens, and only for the row the insert keeps.
  • The RavenDB source flags results of the persister's six own checks, and the target skips flagged rows as SourceStorageCheck, a harmless reason, matched on id and category, so a customer or audit check reusing an id is copied.
  • Group comment ids that differ only in trailing spaces, failed imports of the same message, and licensing endpoints the target folds are reported as merges naming both rows and the one kept; on SQL Server the merge check now follows the column's full collation, accents and width included.
  • Every row the copy stores differently from the source logs a warning naming it: a message type cut to 450 characters, a dropped or collapsed failure group, a failed import with no exception or body, and a group comment the retention sweep will remove. The category summary counts changed and merged rows apart from already present.
  • A retry claim is copied only where RavenDB still held one, so a retry by id behaves as it did before the move.
  • The startup warning explains each harmless skip reason in its own sentence, and the failure summary marks harmless reasons , harmless.
  • Licensing throughput is added to the stored day, folded under the target's own key collation, so the RavenDB source no longer groups endpoints by SQL Server's rules; a retry of the category deletes its copied rows first.
  • A reset that deletes copied rows is refused once ServiceControl has opened on the target.
  • Docs: the overview and system design cover the four categories, the new merges and skip reason, the throughput add, the log line for each data change, and a waiting MessageFailed event that is never sent; the instructions say where merged and changed rows are reported, and docs/README.md needed no change.

Tests

  • ServiceControl.Migration.AcceptanceTests: each new category end to end on SQL Server and PostgreSQL, the category summary separating changed and merged rows, a copied event dispatched once after a retry, the persister check skip by id and category, imported bodies replayed through the product's import path, and blank group comments left out as harmless.
  • ServiceControl.Persistence.Tests/EFCore/Migration: the four writers and their row mappers, their skips, already present rows and merges, that custom checks and failed imports have no text column with a length limit, throughput days summed within and across batches, the retry reset's delete and its refusal after ServiceControl opened, a logged change or merge for each row stored differently, and retry claims copied only where held.
  • ServiceControl.Persistence.Tests.RavenDB/DataMigration: the four readers, the persister checks flagged by id and category and matching the checks the persister registers, throughput read one endpoint per batch, and each retry-issued message flagged with whether RavenDB held its claim.
  • ServiceControl.Persistence.Tests.SqlServer: throughput endpoints that differ only by accent are summed under an accent-insensitive key collation and reported as a merge.
  • ServiceControl.Persistence.Tests.PostgreSql: a NUL character in a custom check or failed import is skipped as malformed and the rest copy.
  • ServiceControl.UnitTests/Migration: every skip reason is exactly one of harmless, permanent or retryable, each harmless reason has its own explanation, and the refusal marks harmless reasons.

…ntegration events, custom checks, failed error imports with their bodies, and group comments. With them, the offline copy covers all sixteen required categories.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants