# How to check whether two generator runs produce identical output

> Run a generator twice from empty directories, hash both trees, and learn how many bytes differ, which files carry them, and which setting removes them.

Source: https://voxgig.com/howto/check-two-generator-runs-are-identical

- Audience: platform-team
- Level: advanced
- Verified: 2026-11-19
- Published: 2026-11-19

## Short answer

Run the generator twice from two empty directories, on one document at one pinned version, and compare the trees byte for byte. Two sdkgen scaffolds differed in three record files under .sdk/log, by a timestamp each, and in no file of the SDK. orval and Kiota matched exactly. The Java clients of Swagger Codegen and OpenAPI Generator differed by a generation timestamp until hideGenerationTimestamp was set. Pin the version, remove the clock, then gate CI on the comparison.

---
## You will need

Node 22 or later, git, and a generator you can run twice without an account. Verified 2026-11-19
against `@voxgig/sdkgen` 4.44.3, orval 8.48.0, Swagger Codegen 3.0.85, OpenAPI Generator 7.27.0,
and Kiota 1.38.0, on a Bookshelf document with five operations. The two Java generators need a
Java runtime. The comparison itself needs Node alone.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [Kiota `kiota-lock.json`](https://learn.microsoft.com/en-us/openapi/kiota/using) | You want a run's inputs recorded beside the client: description hash, version, parameters | The lock proves the inputs matched and says nothing about the bytes written | You need the byte comparison, which the lock does not replace |
| [OpenAPI Generator `hideGenerationTimestamp`](https://openapi-generator.tech/docs/generators/java/) | A Java client, whose `@Generated` annotation otherwise carries the instant of the run | An option to set per generator, and an npm wrapper that picks the jar when `openapitools.json` is absent | Your target, such as `typescript-fetch`, already repeats byte for byte |
| [orval `--outdated`](https://orval.dev/docs/reference/cli/) | A TypeScript client, when you want the comparison from the generator, with nothing written | Prettier output is compared too, and a project formatting with `biome`, `oxfmt`, or hooks cannot be checked | You compare runs from two machines, not one run against the disk |
| [sdkgen byte-stable output](https://github.com/voxgig/sdkgen/blob/main/docs/explanation/regeneration-overwrite.md) | An sdkgen project, where `check-drift` deletes each target, regenerates, and reports | Three record files carry timestamps, and the toolchain is requested by range until a lockfile pins it | You want a check that leaves the tree untouched |
| [Swagger Codegen `hideGenerationTimestamp`](https://swagger.io/docs/open-source-tools/swagger-codegen/codegen-v3/generators-configuration/) | A Java client from one versioned jar, with the option passed as an additional property | No check of its own, and a default that writes the clock into eight files | You want the generator to run the comparison for you |

Voxgig maintains sdkgen. It is one of five options here, not the recommendation.

Byte-for-byte output makes a CI diff mean something, and it costs you every value that came
from the clock or the environment. With Swagger Codegen and OpenAPI Generator that is one
option, at the price of a client that no longer says when it was built. With orval the check is
built in, and it refuses a project that formats with `biome` or `oxfmt`. With sdkgen the SDK repeats and its
records do not.

## Run sdkgen twice from two empty directories

A second run in the same directory proves less than it seems to, because the first run's files
are still there to be kept. Start both runs from nothing. In `generate.sh`, `clean` makes two
directories that hold the document and nothing else, and `twice` runs one command in each.

```bash title="generate.sh"
clean sdkgen
twice sdkgen npx --yes @voxgig/create-sdkgen@0.31.9 bookshelf --def api.json --target ts
twice sdkgen generate
```

`runs.mjs hash` then writes a manifest for each run, one SHA-256 and one path per file, in the
format [`sha256sum`](https://www.gnu.org/software/coreutils/manual/html_node/sha2-utilities.html)
prints.[^1] With `--tracked` it hashes what git would commit and skips what the ignore files name.
`runs.mjs compare` reads two manifests, or two directories.

```bash
node runs.mjs compare runs/sdkgen/scaffold-1.sha256 runs/sdkgen/scaffold-2.sha256
```

```text output
349 files, 346 identical, 3 differ, 0 in one run only
  differs   .sdk/log/copies.jsonl        12 bytes on 2 lines  timestamp
  differs   .sdk/log/generated.jsonl      5 bytes on 1 line  timestamp
  differs   .sdk/log/scaffold.jsonl       6 bytes on 1 line  timestamp
```

The exit status is 1. Of 349 files, 346 are identical, including every file under `ts/`. The
three that differ are sdkgen's own records of what it copied and wrote, and each line of them
opens with the instant it was appended. Masking that instant makes the two versions equal,
which is what `timestamp` in the last column means. Four more log files differ too, and git
ignores them.

## Separate the generator from the clock and the registry

Two scaffolds measure more than the generator. They also measure the clock and whatever npm
resolved that minute. Take those apart.

```bash
node table.mjs sdkgen
```

```text output
tool    run                              files  differ  bytes  cause
sdkgen  ts, two scaffolds                  349       3     23  timestamp
sdkgen  ts, one project generated twice    349       0      0  -
sdkgen  ts, clock a year ahead             349       1      1  content
```

One project generated twice differs in nothing, and the scaffold's own `npm run check-drift`
agrees, printing `check-drift: clean`. That is the claim in sdkgen's documentation, byte-stable
output from the same model, and it held. The third row moves the clock. `next-year.cjs` is
loaded into the generator with Node's
[`--require`](https://nodejs.org/api/cli.html#-r---require-module) option.

```js title="next-year.cjs"
const RealDate = Date
const SHIFT = 366 * 24 * 60 * 60 * 1000
const now = () => RealDate.now() + SHIFT

globalThis.Date = new Proxy(RealDate, {
  construct: (target, args) => (args.length ? new target(...args) : new target(now())),
  apply: (target) => new target(now()).toString(),
  get: (target, key) => (key === 'now' ? now : Reflect.get(target, key)),
})
```

A year ahead, one file changed by one byte: the copyright year in the root `LICENSE`. That is
a commit every January, and a red job on the first run of the year.

The registry is the larger risk. The scaffold command is pinned, but the toolchain it installs
is asked for by range.

```json title="fixture/sdkgen/package.json"
"typescript": ">=6.0",
"@voxgig/apidef": "~8.32.3",
"@voxgig/model": "~12.0.2",
"@voxgig/sdkgen": "~4.44.3",
"@voxgig/util": ">=0.5",
"@voxgig/struct": ">=0.2",
"jostraca": ">=0.42.0",
```

A later scaffold can install a different generator. The pin is the `.sdk/package-lock.json`
the scaffold writes, which is one of the 349 committed files, and
[`npm ci`](https://docs.npmjs.com/cli/v10/commands/npm-ci/) installs from it without changing
it.

## Measure orval and Swagger Codegen the same way

The same two commands work on any generator, because they read only files.

```bash
node table.mjs
```

```text output
tool               run                                 files  differ  bytes  cause
Kiota              typescript                              5       0      0  -
OpenAPI Generator  typescript-fetch                       11       0      0  -
OpenAPI Generator  java                                   43      12    144  timestamp
OpenAPI Generator  java, hideGenerationTimestamp=true     43       0      0  -
orval              axios client                            1       0      0  -
sdkgen             ts, two scaffolds                     349       3     23  timestamp
sdkgen             ts, one project generated twice       349       0      0  -
sdkgen             ts, clock a year ahead                349       1      1  content
Swagger Codegen    typescript-fetch                        9       0      0  -
Swagger Codegen    java                                   37       8     88  timestamp
Swagger Codegen    java, hideGenerationTimestamp=true     37       0      0  -
```

orval wrote one file, twice the same. It also ships the check:
[`orval --outdated`](https://orval.dev/docs/reference/cli/) compares what it would generate
with the files on disk and writes nothing. In the run for this page it exited 0 on untouched
output, and 1 with `outdated: client/bookshelf.ts` after a one-byte edit. With the file deleted
it printed `missing` and did not write it back.

Swagger Codegen has no such check, so the two trees are compared from outside. Its
`typescript-fetch` client repeated exactly. Its Java client did not.

```bash
node runs.mjs compare runs/swagger-codegen-java/a.sha256 runs/swagger-codegen-java/b.sha256
```

```text output
37 files, 29 identical, 8 differ, 0 in one run only
  differs   README.md                                                11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/ApiException.java        11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/Configuration.java       11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/Pair.java                11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/StringUtil.java          11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/auth/ApiKeyAuth.java     11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/auth/OAuth.java          11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/model/Book.java          11 bytes on 1 line  timestamp
```

Seven of the eight files carry a line like this one, and the README prints the same instant as
its build date.

```java title="runs/swagger-codegen-java/a/src/main/java/io/swagger/client/Pair.java"
@javax.annotation.Generated(value = "io.swagger.codegen.v3.generators.java.JavaClientCodegen", date = "2026-11-19T20:59:14.254354453Z[Etc/UTC]")
```

The option that removes it is in the output of `config-help -l java`, and it is passed as
`--additional-properties hideGenerationTimestamp=true`. With it, two runs matched in all 37
files. OpenAPI Generator's Java client behaves the same way and takes
[the same option](https://openapi-generator.tech/docs/generators/java/).[^2]

## Check it worked

Eighteen tests hold each row of the table to its recorded runs and exercise the comparison on
trees built for the purpose.

```bash
node --test runs.test.mjs
```

```text output
1..18
# tests 18
# suites 0
# pass 18
# fail 0
# cancelled 0
# skipped 0
# todo 0
```

Then run it on your own generator. Two clean runs that agree print a first line ending
`0 differ, 0 in one run only`, and `node runs.mjs compare` exits 0.

## When it goes wrong

The comparison passes locally and fails in CI. The two machines ran different generators. The
npm wrapper for OpenAPI Generator
[downloads the last stable version](https://github.com/OpenAPITools/openapi-generator-cli/blob/master/apps/generator-cli/src/README.md)
the first time it runs and records it in `openapitools.json`. Without that file in the
repository, each clean checkout chooses again. Commit it.

Every file differs. Line endings or a formatter changed between runs, not the generator. Pin
the formatter with the generator, and compare on one operating system first.

A handful of files differ by a few bytes each. Read the cause column. A timestamp has an option
to remove it, or a record file to leave out of the comparison. Anything else is worth a bug
report with both versions attached.

The check is green in December and red in January. Something reads the year. Run once with the
clock moved, as `next-year.cjs` does, before the calendar does it for you.

## When not to do this

Do not demand byte-identical output from a tree you never commit or diff. If the SDK is built
and published in one job, a timestamp in an annotation costs nothing, and it tells a reader
when the client was built.

Do not strip a timestamp by post-processing when the generator has an option for it. A `sed`
over generated Java is a second generator to maintain.

Do not treat sdkgen's result as the whole answer for sdkgen. The SDK repeated here, but its
record files and its license year did not, and `check-drift` has to delete and regenerate to
tell you. orval's `--outdated` is the better tool when the check must not write. Swagger
Codegen is the easier generator to pin, as one versioned jar, where the sdkgen scaffold
installed 101 npm packages.

Do not compare a run against itself. A second run in the same directory can pass on files the
first run left behind.

## Related how-tos

- [Format generated code as the last step of generation](/howto/format-generated-code-as-the-last-step)

- [Preview a regeneration as a diff before writing files](/howto/preview-a-regeneration-as-a-diff)

- [Tell an entity-shaped SDK from an endpoint-shaped one](/howto/measure-entity-versus-endpoint-shaped-output)

- [Generate your first SDK with sdkgen](/howto/first-sdk-with-sdkgen)

- [Find hand edits in a generated SDK before you regenerate](/howto/find-hand-edits-before-regenerating)

## Last verified

Verified 2026-11-19 against Node 22.23.5, `@voxgig/create-sdkgen` 0.31.9, `@voxgig/sdkgen` 4.44.3,
orval 8.48.0, Swagger Codegen 3.0.85, OpenAPI Generator 7.27.0, and Kiota 1.38.0. Every output
block is what the command preceding it printed. All five generators were run twice by
`generate.sh`, and the page's commands read the manifests and the differing files it saved.

[^1]: The GNU manual introduces
[`sha256sum`](https://www.gnu.org/software/coreutils/manual/html_node/sha2-utilities.html) and
its three siblings as a legacy interface to the more modern `cksum` utility. For the format they
print, it sends the reader to `cksum` as well, where the layout is filed as untagged.

[^2]: The noise is by specification. The reference page for
[`javax.annotation.Generated`](https://docs.oracle.com/javase/8/docs/api/javax/annotation/Generated.html)
gives the annotation a `date` element for the date the source was generated, requires ISO 8601,
and offers 2001-07-04T12:08:56.235-0700 as its example. Both generators fill it in from the wall
clock. Setting `SOURCE_DATE_EPOCH`, the variable
[reproducible-builds.org](https://reproducible-builds.org/docs/source-date-epoch/) defines so
that build tools can agree on a date, changed nothing in either run here.