DATAMIMIC — Governed Test Data for Regulated Enterprises

This repository contains the DATAMIMIC Community Edition (CE). MIT-licensed, Python-native, MCP-ready.

CE is fully usable standalone for deterministic synthetic data generation and model-based pseudonymization. The Enterprise Platform adds governed workflows, PII scanning, role-based access, audit logging, scheduling, multi-system execution, and the operational layer for governed enterprise workflows.

👉 Enterprise Platform: datamimic.io  |  📘 Docs: docs.datamimic.io  |  📅 Book a strategy call: datamimic.io/contact

🤖 AI agent? Install the DATAMIMIC plugin for Codex or Claude Code, or read the consumer skill. Repository contributors also use AGENTS.md.

---

Current development architecture — explore the observed structure, target and differences with ArchKeel.

---

What is DATAMIMIC?

DATAMIMIC CE is the open-source deterministic data engine at the core of the DATAMIMIC Enterprise Platform. It is usable standalone for synthetic data generation and model-based pseudonymization in local, CI, or agent-driven workflows.

The Enterprise Platform adds the governed workflows, scanners, dashboards, and execution layer that regulated enterprises require for production-scale test-data operations.

Available in CE (this repo):

• Generate fully synthetic, deterministic datasets — model-driven, no source data required
• Pseudonymize staging/QA exports — seeded, repeatable or run-specific (non-seeded) field transformation; PII fields identified and modeled manually in the XML pipeline
• Execute single-system pipelines against PostgreSQL · MySQL · Oracle · MS SQL · SQLite · MongoDB · CSV · JSON · XML · XLSX · DbUnit · fixed-width (.fcw)
• Model behavior — weighted state machines, composite multi-field references, control flow (<while>, <assert>), and a scriptable memstore for staged aggregation
• Emit provenance — execution logs plus a SHA-256 content hash on generate_domain facade outputs, so two runs can be checked for identical output
• Guide agents — machine-readable capabilities, progressive reference queries, and one canonical CLI project Build; an optional MCP adapter exposes the same authoring service

The Enterprise Platform adds:

• PII scanner — probability-scored field suggestions with configurable thresholds; users confirm mappings in DataWorkbench
• Multi-system execution — Oracle / MongoDB / Kafka in coordinated workflows with referential integrity
• Industry message templates — EDIFACT / SWIFT MT / HL7 v2.x / HL7 FHIR generated as deterministic test/training artefacts
• Governance layer — role-based dashboards, audit trails, approval flows, reusable enterprise templates, scheduler
• Performance core — Rust fastpath, ML/auto-regressive engine for complex distributions, keyset and manifest building, optimised distributed execution
• On-premise / air-gapped deployment — podman-compose or Helm, with consulting-led rollout

Deployed in regulated EU banking environments for deterministic test data across Oracle, MongoDB, and Kafka pipelines. Reference customers available under NDA — see also datamimic.io case studies.

---

AI agents: author, verify, and run data models

The agent plugin prepares CE in your project environment when needed. For manual installation, use python -m pip install datamimic-ce from PyPI. These instructions describe CE 5.0.

XML is the persisted executable project: a datamimic.xml entry and the steps it includes. Expectations and role declarations live beside it in acceptance/build.json. DM JSON is a transient agent representation of one XML file.

Need	Command
Discover the running contracts	datamimic authoring schema / datamimic authoring schema --name build_request_v5
Discover CE vocabulary	datamimic authoring capabilities / datamimic authoring capabilities --full
Inspect one element	datamimic authoring capability --domain element --name generate
Read focused guidance	datamimic authoring reference --topic assembly --name model
Compile/decompile supported syntax	datamimic authoring compile - / datamimic authoring decompile model.xml
Prove a project	datamimic authoring build <project>
Inspect existing XML safely	datamimic authoring lint model.xml / datamimic authoring dry-run model.xml
Execute the persisted project	datamimic run datamimic.xml

Authoring commands default to JSON and accept files or - for stdin. A ready compile/decompile proves syntax. A ready Build also requires complete bounded capture and the declared acceptance checks. Include every business requirement in acceptance/build.json; request smoke/replay through its verification fields when needed. A ready Build lists every project file it read with its digest. Target isolation does not prove delivery to an external database.

Prompt to paste into an agent

Create the dataset I describe with the DATAMIMIC plugin's generate-test-data
skill and the project CLI. Write the project as datamimic.xml plus
acceptance/build.json with expectations for my stated requirements and identifier/FK
roles where needed, then run datamimic authoring build on the project directory.
Repair the reported paths before retrying. Stop on outcome=ready and return the
project path with the bounded verification evidence. Execute the XML only if I
requested real execution.

Worked projects cover nested foreign keys, memstore readback and time series. Return a human-readable model review with the XML and verification evidence.

Optional MCP adapter

python -m pip install "datamimic-ce[mcp]" adds the same application as typed MCP tools: Build, compile/decompile, discovery, lint and bounded execution. Registration and request mapping are in the MCP quickstart.

---

CE vs Enterprise Platform

CE and EE are not the same engine with a feature flag. They share the DSL, while EE is an independently optimised execution engine built for enterprise-scale throughput and operational control.

Engine comparison

Capability	Community Edition (CE)	Enterprise Platform (EE)
Deterministic data generation	✅	✅
Deterministic seeding in the DSL (<setup rngSeed>)	✅ entities, generators and script expressions	Separate engine; not covered by the CE test matrix
Pseudonymization — seeded	✅ manual model	✅ scanner-assisted, user-confirmed model
Pseudonymization — non-seeded	✅ manual model	✅ scanner-assisted, user-confirmed model
Python API + XML pipelines	✅	✅
Domain models: Finance, Healthcare, Demographics	✅	✅
Time-series generation (<generate start/end/interval>, ISO 8601, prefix-stable)	✅	✅
MCP server for AI agent integration	✅	✅
CLI + local execution	✅	✅
Scale	millions of records via Python multiprocessing (and optional Ray)	designed for billion-record workloads — Rust fastpath, optimised multi-process execution, and keyset/manifest building on top of the shared Ray distribution layer
PII scanner	❌	✅ probability-scored field suggestions, configurable threshold, user confirmation in DataWorkbench
Runtime configuration profiles	❌	✅ Performance · Balanced · Flexibility
Memory management	standard	optimised for high-volume batch and streaming
Logging granularity	flat execution log	configurable: minimal · standard · deep nested tracing
Nested structure evaluation	basic	deep nested generation with extended condition + ruleset evaluation
Importer / exporter logging	❌	per-stage logging for importers and exporters
Error handling	structured catalog, typed errors and guidance	structured catalog with recovery strategies
Rust fastpath	❌	performance-critical paths in Rust
Keyset and manifest building	❌	reads live DB schemas to build coordinated multi-table generation plans
ML / auto-regressive engine	❌	combine statistical models with conditions, rulesets, validators for complex distributions

Platform capabilities (EE only)

Capability	EE
Multi-user collaboration	✅
Role-based access control (RBAC)	✅
Audit logs + provenance dashboards	✅
PII scanner — probability scoring, threshold-based suggestions for user-confirmed field mapping	✅
DataWorkbench — visual field mapping and pseudonymization model builder	✅
Reusable enterprise template library	✅
Scheduled execution + task runner	✅
CI/CD pipeline integration (Tosca, Jenkins, GitLab)	✅
Multi-system execution: Oracle, MongoDB, Kafka	✅
Template engine: schema-aware editors for EDIFACT, SWIFT MT, HL7 v2.x, and HL7 FHIR — customer-uploadable specs, further industry formats built per engagement on the same framework	✅
Audit-evidence artefacts for GDPR Art. 30 records, PCI DSS 4.0.1 Req. 6.5.5 (test data) reviews, and HIPAA Security Rule reviews	✅
On-premise deployment + air-gapped environments	✅
LSP-powered IDE tooling for DSL authoring	✅

👉 Explore the Enterprise Platform  |  Book a platform demo

---

EE runtime profiles

The EE core supports three runtime configuration profiles, selectable per execution context:

Profile	Optimises for	Typical use case
Performance	Maximum throughput via Rust fastpath, optimised multi-process execution, and Ray-based distribution	Bulk generation for PostgreSQL, Oracle, and Kafka, designed for billion-record workloads
Balanced	Throughput + full audit logging	Standard enterprise pipeline runs with compliance requirements
Flexibility	Deep nested evaluation, extended condition and ruleset processing	Complex domain models with ML engine combinations, multi-level referential structures

Logging depth is independently configurable per profile — from minimal (throughput-optimised) to full nested tracing across importers, exporters, and generation stages.

---

EE template engine

The EE template engine generates industry-standard financial messages from DATAMIMIC models. The workbench parses uploaded message samples, auto-detects the message type, and validates edits against the registered spec version in real time.

Capabilities

• Spec-aware form editing — segments and elements rendered as structured forms with mandatory/optional indicators, per-field value suggestions, and inline custom-extension support
• Strict validation against baked spec versions, with segment- and element-level error reporting
• Advisory mode when a spec is unregistered or in draft — editing stays enabled, validation continues as guidance
• Round-trip between the structured form view and the authoritative template text — no fidelity loss
• Download / adjust / upload your own spec — customers can extend or override the baked spec catalogue without waiting for a release
• Live structure tree + preview for every edit
• File auto-detection — upload an existing message, the editor identifies the type and loads the matching spec

Format coverage

Format	Coverage
UN/EDIFACT	Schema-aware form editor; spec versions and subsets per engagement
SWIFT MT	Schema-aware form editor; categories and SR versions per engagement
HL7 v2.x	Schema-aware form editor; versions per engagement
HL7 FHIR	Schema-aware form editor for FHIR resources (Patient, Observation, Encounter, …); profiles per engagement
Further industry formats (ISO 20022 / MX, vertical dialects)	Built into the editor catalogue per customer engagement, on the same framework

Customers can extend the spec catalogue between releases by downloading, adjusting, and uploading their own spec files directly.

Generated messages are traceable to their source model and syntactically valid against the registered spec. They are intended for test and training environments only — they are not network-validated and must not be transmitted on production SWIFTNet or EDI networks. See the SWIFT CSP note below.

---

Who is DATAMIMIC for?

Enterprise Platform (EE)

Role	What DATAMIMIC solves
QA / Test Manager	Eliminate manual test data requests. Self-service, governed, always ready.
Business Analyst	Define data requirements in business-readable models — no scripting needed.
Platform / DevOps Engineer	Integrate deterministic test data generation into CI/CD and scheduled pipelines.
Compliance / Audit	Full audit trail for every generation run. Regulator-ready logs, no production data exposure.
Enterprise Architect	One governed standard across Oracle, MongoDB, Kafka, flat files, and custom systems.

Community Edition (CE)

Developers and data engineers who need deterministic synthetic data generation or model-based pseudonymization in local environments, CI pipelines, or agent-driven workflows. PII field identification is manual in CE; the EE DataWorkbench assists review and users confirm mappings.

---

Why deterministic generation matters

Most test data tools produce random output. That breaks regression tests, audit trails, and cross-team reproducibility.

How it works in CE:

• Set <setup rngSeed="N"> and seeded generation replays on the tested runtime profiles, including generator, entity, and script-expression paths. Seeded date-derived values use the deterministic clock anchor; without a seed, generation is not replay-stable.
• distribution="ordered" reads a source in file order; distribution="random" shuffles it, reproducibly with a seed.
• The Python facade generate_domain(...) returns determinism_proof.content_hash: the same hash means the same data.
• The committed replay model covers every registered literal generator except the database-backed SequenceTableGenerator, plus the supported dynamic script-global families. It is replayed in two separate processes on each CI run (test_determinism_seed_scenarios).

The CE seeded-runtime guarantee is scoped to the tested profiles. CI runs seven x64 cells: ubuntu-22.04 with Python 3.11, 3.12, and 3.13 in Etc/UTC; Python 3.11 in America/Los_Angeles; and windows-latest with Python 3.11 in UTC and Pacific Standard Time. These six cells use uv sync --resolution highest; a representative ubuntu-22.04/Python 3.11/Etc/UTC cell uses uv sync --resolution lowest-direct. Each cell records its exact interpreter and installed package versions in the uploaded artifact and job log. Other Python versions and dependency resolutions are outside this matrix. uv.lock is intentionally untracked. The committed goldens and cross-cell fan-in, rather than production dependency pinning, define the byte-identical gate.

EE behavior is outside this CE evidence.

from datamimic_ce.domains.facade import generate_domain

request = {
    "domain": "person",
    "version": "v1",
    "count": 1,
    "seed": "regression-suite-42",       # same seed → replayable on the tested profiles
    "locale": "en_US",
    "clock": "2025-01-01T00:00:00Z"      # fixed clock = stable time context
}

response = generate_domain(request)
# response["determinism_proof"]["content_hash"] is stable for the same CE version, request,
# seed, and clock within the tested compatibility boundary.

Direct service use is deterministic for repeated runs with the same seeded RNG and configuration:

import random
from datamimic_ce.domains.finance.services import CreditCardService

# Same seeded Random → same CreditCard fields for the same configuration.
card_a = CreditCardService(rng=random.Random(42)).generate()
card_b = CreditCardService(rng=random.Random(42)).generate()
assert card_a.bic == card_b.bic and card_a.card_number == card_b.card_number

Determinism evidence — CE

The CI gate compares seeded runtime manifests and committed goldens across the seven matrix cells above. Its coverage is:

Scope	Evidence
Facade API	4/4 registered domains
Entities	23/23 entities, selected attributes
Literal generators	34/35; database-backed SequenceTableGenerator is excluded from this byte hash and covered by external-service DSL tests
Seeded Safe Globals	27 representative paths across datetime, fake, math, numpy, pandas, random, and uuid
Rejected entropy	5 committed negative DSL descriptors
Encoding	1 canonical UTF-8 probe

Arbitrary exporter bytes/encodings and Enterprise Platform behavior are outside this CE gate.

---

How DATAMIMIC differs from Faker and generic generators

	Faker / Random generators	DATAMIMIC CE	DATAMIMIC EE
Reproducible output	❌	✅	separate engine; outside CE evidence
Domain-aware relationships	❌	✅	✅
Business logic constraints	❌	✅	✅
Per-output provenance hash	❌	✅	✅
Source data pseudonymization	❌	✅ manual	✅ scanner-assisted, user-confirmed
PII field detection	❌	❌	✅ probability-scored suggestions
Enterprise governance layer	❌	❌	✅
Multi-system execution	❌	❌	✅
Role-based workflows	❌	❌	✅
Designed for regulated-industry deployment (governance, audit, RBAC)	❌	❌	✅

# Faker — broken relationships
from faker import Faker
fake = Faker()
patient_age = fake.random_int(1, 99)
conditions  = [fake.word()]
# "25-year-old with Alzheimer's" — meaningless for any real test

# DATAMIMIC — domain-aware, deterministic with a seed
import random
from datamimic_ce.domains.healthcare.services import PatientService
patient = PatientService(rng=random.Random(42)).generate()
print(f"{patient.full_name}, {patient.age}, {patient.conditions}")
# Age-appropriate, domain-consistent — and identical every run with a fixed seed

---

Quickstart — Community Edition

python -m pip install datamimic-ce

For database and MongoDB connections, prefer connection profiles. The guide explains client references, profile selection, and how profile values override XML attributes.

Healthcare domain

import random
from datamimic_ce.domains.healthcare.services import PatientService

patient = PatientService(rng=random.Random(42)).generate()
print(patient.full_name, patient.age, patient.conditions)
# Age-appropriate conditions, demographically realistic; deterministic with a seed

Finance domain

import random
from datamimic_ce.domains.finance.services import BankAccountService

account = BankAccountService(rng=random.Random(42)).generate()
print(account.account_number, account.balance)
# Balance-consistent, locale-correct; reproducible with a seed

Pseudonymization — CE (manual model)

DATAMIMIC supports two pseudonymization modes with different repeatability properties:

Mode	How	Compliance note	Use case
Seeded (rngSeed set)	Deterministic on the tested profiles	May form part of a pseudonymisation approach. Whether the result qualifies as pseudonymised under GDPR Art. 4(5) depends on whether attribution requires additional information that is kept separately and protected.	Regression testing, stable CI/CD pipelines
Non-seeded (no rngSeed)	Not replay-stable	—	One-time data delivery

Note on GDPR: Pseudonymisation does not make data anonymous or establish GDPR compliance. Under Recital 26, anonymity depends on whether a person remains identifiable considering means reasonably likely to be used.

In CE, PII fields are identified and modeled manually in the XML pipeline. The source is a pipe-separated project data file; the scenario tests_ce/scenarios/s28_pseudonymize_source runs this model:

<setup rngSeed="42">
  <generate name="customers" source="data/entity/customers.ent.csv" target="CSV" distribution="ordered">
    <!-- rngSeed makes seeded runs produce the same pseudonyms; remove it for a new
         mapping per run. distribution="ordered" keeps source row N mapped to the
         same pseudonym. -->
    <variable name="p"   entity="Person"      dataset="DE" />
    <variable name="acc" entity="BankAccount" dataset="DE" />

    <key name="customer_id" script="customer_id" />
    <key name="first_name" script="p.given_name" />
    <key name="last_name"  script="p.family_name" />
    <key name="email"      script="p.email" />
    <key name="iban"       script="acc.iban" />
    <key name="birth_date" script="p.birthdate" />
  </generate>
</setup>

Built-in converters can additionally transform a key's value — e.g. replace the original with a keyed hash, or partially mask it. Hash derives its HMAC key from <setup rngSeed>; the same input + seed + algorithm + output format + optional salt gives the same token across runs. Without a seed each run gets a new random key:

<key name="email" script="p.email" converter="Hash('sha256','hex')" />
<key name="iban"  script="acc.iban" converter="MiddleMask(8, 4)" />

For pseudonymization, keep a random key outside the descriptor and separate from the generation seed. Reference a runtime property in the converter:

<key name="email" script="p.email"
     converter="Hash('sha256','hex',key=pseudonym_key.encode('utf-8'))" />

Supply it from an access-controlled environment variable:

# Generate once, store securely, and reuse for stable tokens.
export DATAMIMIC_PSEUDONYM_KEY="$(python -c 'import secrets; print(secrets.token_hex(32))')"
datamimic run datamimic.xml --param-env pseudonym_key=DATAMIMIC_PSEUDONYM_KEY

--param NAME=VALUE and --param-env NAME=ENV_NAME are repeatable and override properties from conf/environment.env.properties. Property names must be Python identifiers; duplicate names are rejected. Use --param-env for secrets because command-line values can appear in process listings and shell history. Runtime properties remain available to descriptor scripts; use trusted descriptors and control access to the key and source data.

Hash keys must contain at least 32 bytes before adding an optional salt. Length alone does not prove entropy: use a cryptographic random source, rather than a password, date or generation seed. Rotating the external key changes tokens for the same input; retain the prior key securely only when old mappings must be reproduced. An external key keeps tokens stable when rngSeed changes and leaves seeded data generation unchanged. Existing Hash(...) calls without an explicit key retain their seed-derived behavior and its weak-seed limitations.

Available converters (13): Mask, MiddleMask(start, end), CutLength(n), Substring(start, end), JavaHash, RemoveNoneOrEmptyElement, Hash(type, format[, salt]), DateFormat(fmt), Append, UpperCase, LowerCase, Date2Timestamp, Timestamp2Date.

datamimic run ./pseudonymize-customers/datamimic.xml

Use a controlled export or staging input. Do not point this workflow at a live production source.

With rngSeed: the same descriptor, seed, and ordered source rows reproduce the same replacement sequence; this is not a subject-keyed mapping across reordered exports.

Without rngSeed: random/entity replacements use a new run-specific mapping, and hash converters use a new random key; outputs are not replay-stable.

In the Enterprise Platform (EE): the DataWorkbench PII scanner assists source-schema review by assigning probability scores and flagging candidates above a configurable threshold. Users confirm the fields and model before execution.

<setup rngSeed="42">
  <generate name="patients" count="1000" target="CSV">
    <variable name="patient" entity="Patient" dataset="US" ageMin="60" ageMax="80" />
    <key name="full_name"   script="patient.full_name" />
    <key name="age"         script="patient.age" />
    <array name="conditions" script="patient.conditions" />
  </generate>
</setup>

datamimic run ./patient-scenario/datamimic.xml

Time-series generation — CE

Any <generate> becomes a time-series loop when given strict ISO 8601 start/end/interval attributes. Per iteration the script context exposes a ts namespace:

Variable	Type	Meaning
ts.now	datetime	Current tick
ts.step	int	Position within one series (0..N-1)
ts.series	int	Which series this row belongs to (0..count-1)

Output column names — including whether to even emit a timestamp or series-id column — are entirely the user's choice via <key>. The primitive is domain-agnostic; the same DSL covers IoT readings, financial ticks, log streams, smart meters, anything time-indexed.

<setup>
  <!-- Stock ticks: three symbols, 5-min interval, 30-min window (writes ticks.csv) -->
  <generate name="ticks" count="3"
            start="2026-01-01T09:30:00+00:00"
            end="2026-01-01T10:00:00+00:00"
            interval="PT5M"
            target="CSV">
    <key name="timestamp" script="ts.now.isoformat()"/>
    <key name="symbol"    script="['AAPL','MSFT','GOOG'][ts.series]"/>
    <key name="price"     script="100 + ts.step * 0.25"/>
  </generate>

  <!-- Sensor with diurnal seasonality, single series (count defaults to 1; writes readings.csv) -->
  <generate name="readings"
            start="2026-01-01T00:00:00+00:00"
            end="2026-01-08T00:00:00+00:00"
            interval="PT1H"
            target="CSV">
    <key name="timestamp" script="ts.now.isoformat()"/>
    <key name="value"     script="20 - 10 * math.cos(ts.now.hour * math.pi / 12)"/>
  </generate>
</setup>

Guarantees:

• Prefix-stable by construction — the first N ticks of series 0 are byte-identical regardless of total window length, because each row's ts.now is a pure function of start + interval * step.
• Loop order is contiguous per series — series 0's full sequence, then series 1's, etc. Makes downstream grouping trivial.
• Strict ISO 8601 — start/end via datetime.fromisoformat (Z-suffix supported); interval via the isodate library (PT1H, PT15M, PT5S, P1D, P1W, P1DT12H, fractional seconds for sub-second precision). Resolution: PT0.001S = 1 ms, PT0.000001S = 1 µs (Python datetime.timedelta microsecond floor; sub-µs intervals and constant-length-undefined units like months/years are rejected with a clear error).
• count is orthogonal, not overloaded — it means "outer-loop iterations of this <generate>" in all modes (same as nested <generate count=…>). In time-series mode each outer iteration is one series of N ticks, so total rows = count × ticks_per_series. Default count="1" keeps single-series fixtures terse.
• Naming caveat — a <key name="ts"> output column would shadow the namespace (current_product overrides current_variables in script scope), and a <variable name="ts"> is rejected at parse time. Use a different name, e.g. timestamp for the column.

Composes with the existing <variable> mechanism for multi-source merges (e.g. join each tick with a sensor-metadata CSV via <variable source="meta.csv" cyclic="True"> inside the same <generate>), with <key condition="..."> filtering, and with <nestedKey> sub-scopes — the ts namespace is visible everywhere a <key script> runs. See tests_ce/integration_tests/test_timeseries/ for committed DSL fixtures + proofs (including pagination invariance).

---

Where CE fits on its own

Most teams adopt CE for one of three reasons. EE is not required for any of them.

1. Reproducible test data for CI/CD pipelines. Pin a seed against the generate_domain facade — or hand a seeded random.Random to any domain service — and you get replay-identical output within the tested compatibility boundary above. Both layers are replay-tested on every CI run by tests_ce/architecture/. Regression tests stop being flaky because the input data is stable across runs.

from datamimic_ce.domains.facade import generate_domain

response = generate_domain({
    "domain": "person", "version": "v1", "count": 1,
    "seed": "ci-pipeline-42", "locale": "en_US",
    "clock": "2026-01-01T00:00:00Z",
})
# Same CE version + request + seed + clock → same output within the tested boundary.

2. Deterministic data backend for AI agents and LLM tooling. The CLI and Python API are the baseline surfaces for seeded, verifiable generation. The optional MCP adapter exposes the canonical Build, codec, discovery, lint, and bounded-run authoring operations. Domain-facade outputs include a determinism_proof.content_hash, so Python callers can re-execute within the tested compatibility boundary and check whether the data is identical — useful for agent regression tests and any workflow where the data an agent saw must be reconstructable.

3. Pseudonymization of staging and QA exports. Manual model in CE (XML pipeline), no scanner license required. Seeded mode for stable regression test data; non-seeded mode for one-time deliveries that are not replay-stable. See the Pseudonymization section above.

---

Where DATAMIMIC fits in your compliance program

DATAMIMIC provides test-data and execution evidence that can support compliance work. It does not replace your DPO, your CISO, or your auditor. The following are pointers for where DATAMIMIC outputs commonly slot into established programs:

CE's reproducible, rule-based artefacts are within the tested boundary above. EE adds multi-system audit evidence and role-based dashboards; the CE matrix does not verify EE execution.

Regulation / standard	Where DATAMIMIC contributes
DORA (Reg. 2022/2554) — Arts. 24–25, digital operational resilience testing	Reproducible test data can support scenario-based, compatibility, performance, and end-to-end testing
ISO/IEC 27701:2025 — Privacy Information Management System (PIMS)	Synthetic data can reduce PII use in non-production; model and execution evidence can support privacy-management processes
HIPAA Security Rule — 45 CFR Part 164, Subpart C (US Covered Entity / Business Associate scope)	Synthetic healthcare data can reduce the need to copy ePHI into dev/test; safeguards still apply where ePHI exists
GDPR — Arts. 4(5), 25, 32	May contribute to a pseudonymisation approach and related privacy-by-design and security measures; whether requirements are met depends on the dataset, additional information, and safeguards
PCI DSS 4.0.1 — Req. 6.5.5 (live PANs not used in pre-production except where the environment is part of the CDE and protected by applicable PCI DSS requirements)	Synthetic card-number test data can reduce the need to use live PANs in pre-production

These mappings show where DATAMIMIC can contribute; they are not attestations and do not establish that a dataset or processing activity satisfies the reference.

---

Architecture

The 5.0 refactoring target is documented in the shared CE/EE core structure, including nested boundaries and the source-to-target map. It is a target, not a claim that the current implementation conforms; see the verification status.

CE and EE share the DATAMIMIC DSL. The execution layer is separate: CE is a Python execution engine using multiprocessing (with optional Ray for distribution); EE is an independently-optimised execution engine with a Rust fastpath, ML/auto-regressive generation, keyset and manifest building from live schemas, and execution designed for billion-record workloads.

╔══════════════════════════════════════════════════════════════════╗
║              DATAMIMIC ENTERPRISE PLATFORM (EE)                  ║
║                                                                  ║
║  ┌──────────────────────────────────────────────────────────┐    ║
║  │  PLATFORM LAYER                                          │    ║
║  │  UI · RBAC · Governance · Audit Dashboards               │    ║
║  │  DataWorkbench · PII Scanner · Pseudonymization Builder  │    ║
║  │  Scheduler · Task Runner · CI/CD · Template Engine       │    ║
║  └──────────────────────────────────────────────────────────┘    ║
║                                                                  ║
║  ┌──────────────────────────────────────────────────────────┐    ║
║  │  EE CORE  (separately maintained, more advanced than CE) │    ║
║  │                                                          │    ║
║  │  Rust fastpath for performance-critical paths            │    ║
║  │  ML / auto-regressive engine for complex distributions   │    ║
║  │  Keyset and manifest building from live DB schemas       │    ║
║  │  Execution designed for billion-record workloads         │    ║
║  │  Runtime profiles: Performance · Balanced · Flexibility  │    ║
║  │  Deep nested evaluation · Conditions · Rulesets          │    ║
║  │  Structured error catalog · Per-stage execution logging  │    ║
║  └──────────────────────────────────────────────────────────┘    ║
╚══════════════════════════════════════════════════════════════════╝

╔══════════════════════════════════════════════════════════════════╗
║              DATAMIMIC COMMUNITY EDITION (CE)  — this repo       ║
║                                                                  ║
║  Determinism Kit · Domain Services · Schema Validators           ║
║  Synthetic Generation · Pseudonymization (manual model)          ║
║  Python API · XML Pipelines · CLI · MCP Server                   ║
╚══════════════════════════════════════════════════════════════════╝

         ↓              ↓              ↓              ↓
    PostgreSQL       Oracle         MongoDB      CSV / JSON / XML

EE adds Kafka, EDIFACT, SWIFT MT, HL7 v2.x, and HL7 FHIR as additional targets — see Supported systems below. Both editions share the DATAMIMIC DSL.

---

Supported systems

System	CE	EE	Notes
PostgreSQL	✅	✅	EE adds schema introspection and referential integrity
MySQL	✅	✅
Oracle	✅	✅	EE production-validated in regulated banking environments
MS SQL Server	✅	✅
SQLite	✅	✅	Lightweight CI/CD fixtures
MongoDB	✅	✅	EE adds nested document generation
CSV / JSON / XML	✅	✅	Flat file pipelines
XLSX	✅	✅	Spreadsheet read + write (first row = header)
DbUnit XML	✅	✅	.dbunit.xml dataset read + write
Fixed-width (.fcw)	✅	✅	Self-describing column files, read + write
Apache Kafka	—	✅	Real-time streaming, payment scenarios
HL7 v2.x	—	✅	Test/training output via template engine
HL7 FHIR	—	✅	Test/training output via template engine
EDIFACT / SWIFT MT	—	✅	Test/training output only; does not satisfy SWIFT CSCF v2025 secure-zone controls (1.1 environment protection, 1.4 internet restriction). Generated messages must not be transmitted from a CSP-attested secure zone.

---

CE domains

Domain	Services available
Healthcare	Patient, Doctor, Hospital, MedicalDevice, MedicalProcedure
Finance	Bank, BankAccount, CreditCard, Transaction
Insurance	InsuranceCompany, InsuranceProduct, InsurancePolicy, InsuranceCoverage
E-commerce	Order, Product
Public sector	AdministrationOffice, EducationalInstitution, PoliceOfficer
Demographics	Person (DE / US / VN locale packs), Address, City, Country
Common	Company

All services are versioned and seeded; facade outputs carry a content hash that shows whether two generations are identical. Domain services can be used directly via constructor injection, or driven through the higher-level generate_domain({...}) facade for seed/locale/clock/count parameterisation (currently supports person, address, patient, doctor at v1).

---

CLI reference

# Typed discovery and project Build
datamimic authoring schema --name build_request_v5
datamimic authoring capabilities
datamimic authoring capability --domain element --name generate
datamimic authoring reference --topic assembly --name model
datamimic authoring build my-scenario

# Inspect/edit persisted XML through its transient DM JSON view
datamimic authoring decompile my-scenario/datamimic.xml
datamimic authoring lint my-scenario/datamimic.xml
datamimic authoring dry-run my-scenario/datamimic.xml

# Normal production execution and project tools
datamimic run my-scenario/datamimic.xml
datamimic init my-scenario
datamimic demo list
datamimic demo create demo-healthcare
datamimic info
datamimic version

---

Documentation

Resource	Link
Full documentation	docs.datamimic.io
MCP quickstart	docs/mcp_quickstart.md
Developer guide	docs/developer_guide.md
Enterprise platform	datamimic.io
GitHub Discussions	Discussions
Issue tracker	Issues
Email support	support@rapiddweller.com

---

Contributing

See CONTRIBUTING.md. CE is MIT licensed and community contributions are welcome.

The CE engine is the foundation. If you are building integrations, domain extensions, or MCP tooling on top of DATAMIMIC, we want to hear from you.

---

License

MIT — see LICENSE.

The DATAMIMIC Enterprise Platform (EE) is a commercial product. Contact us for licensing.

---

DATAMIMIC — Deterministic, governed test data for regulated enterprises.

datamimic.io  |  Book a demo  |  LinkedIn