Back to case studies
Spec-Driven Generation of Streaming Data Pipelines
Client name withheld · IT & Services
Accuracy, direct jobs
85–90% (template)
100%
Human-approved decisions
15 on the hardest job
Residual drift
zero
For a business SaaS data platform: twelve Codex skills turn a data-contract pipeline spec into a versioned design, then into Java Flink jobs, and review the code against both the design and the hand-written production jobs.
What changed
Before
Generation
One template reused
60–65% accurate overall
After
Specify
Versioned SDD
from the data contract
Decide
Decision log
human approvals
Generate
Java Flink jobs
Kafka → DuckDB → S3
Verify
Code vs SDD
Code vs production
accuracy %
A data contract becomes a versioned design, then a trace matrix, then code, and the code is reviewed against both the design and the production jobs.
What we built
The platform's ten pipelines split into three runtime families, and reusing one pipeline as a template scored 85 to 90% for the first family and 60 to 65% overall. So twelve Codex skills turn a resolved data-contract pipeline spec into a versioned SDD, then into Java Flink jobs that read CDC events from Kafka, keep state in DuckDB and write tenant-partitioned Parquet to S3.
Key Features
Seven artifacts per version: constitution, source, spec, plan, tasks, acceptance and decision log, with every runtime value in one canonical config.
A decision log as the only place for questions and approvals. Each open decision carries an options analysis, and a recommendation is never an approval.
Staged prompts: each stage declares its read-only, read-write and prohibited paths and runs as a fresh context.
A full trace matrix before any code edit, then Spotless, Maven test and package.
Independent reviews of the code against the SDD and against the production code.
Results
Two direct current-state jobs matched the hand-written production jobs at 100% on behavior and output shape.
28 entity classes generated.
A five-source tenant recompute job needed 15 decisions, each with an options analysis and an explicit approval, before code generation.
Drift reconciliation passed both human gates with zero residual drift.
Stack
Next step
Let’s look at your SDLC together.
45 minutes with a senior engineer: your setup, the risks, what to fix first. No sales team.