Benchmarks

The benchmark/ directory holds two reproducible SBCL harnesses. Both emit human-readable progress to standard error and machine-readable TSV to standard output, with full provenance (host, SBCL version, pinned sources, and execution order) embedded in the output itself.

Benchmark details

This page summarizes how to run the harnesses and read their output. The environment variables, TSV schema, and correctness gates are documented in benchmark/README.md.

The two harnesses

• benchmark/run.lisp measures cl-json-kit's own string and stream reader/writer APIs.
• benchmark/competitors.lisp compares its string DOM APIs against Jzon, Jonathan, JSOWN, and Yason.

Running them

The library's own reader/writer throughput:

sbcl --noinform --disable-debugger --script benchmark/run.lisp > results.tsv

The comparison against other libraries — the competitor libraries are provided by the default nix develop shell:

nix develop --command sbcl --noinform --disable-debugger \
  --script benchmark/competitors.lisp > competitor-results.tsv

A fast smoke test with no warmup and a single sample:

BENCH_WARMUP=0 BENCH_ITERATIONS=1 \
  nix develop --command sbcl --noinform --disable-debugger \
  --script benchmark/competitors.lisp > competitor-smoke.tsv

Environment variables

Both harnesses accept these:

Variable	Default	Allowed values	Meaning
BENCH_OPERATIONS	all	all, parse, stringify	Which operation family to measure.
BENCH_SEED	20260724	integer ≥ 0	Seed for the per-round case shuffle.
BENCH_WARMUP	2	integer ≥ 0	Number of warmup rounds.
BENCH_ITERATIONS	5	integer ≥ 1	Measured samples per case.

benchmark/run.lisp also accepts CL_JSON_KIT_ROOT to point at a specific checkout; without it, the script loads the checkout that contains it.

What is measured

run.lisp measures three ~1 MiB strings with different escape densities, integer and floating-point JSON arrays, an integer-valued JSON object, and the public read-json stream API. Reader throughput uses the JSON input length; writer throughput uses the original Lisp string length (not the escaped output length).

competitors.lisp compares in two modes:

• native rows keep each library's configured DOM and API, so they compare the libraries' configured native work — not identical DOM construction.
• canonical parse rows normalize every result to hash-table objects with string keys, vector arrays, and the shared scalar markers, so they compare a common result contract but include per-library normalization cost.

Correctness gates come first

Before any timing, every selected case must pass a validator: reader validators check the root type, exact element or member count, and every expected value; writer validators reparse the produced JSON and compare the complete source string.

cl-json-kit is the stringify oracle

Stringify gates reparse each result with cl-json-kit and compare against the source payload. cl-json-kit is therefore the oracle, not an independent JSON conformance implementation — a shared interpretation or defect could pass. These fixtures are not a complete JSON conformance suite.

Scope of measurements

These measurements apply to the recorded corpus, operation and mode, pinned source state, SBCL version, settings, execution order, and host. They do not measure parser-core performance when canonical parse normalization is included.

The TSV artifact records the workload and environment needed to reproduce a result. Interpret each result with those recorded conditions.