Evals

Numbers, not vibes

A model is only as good as what you can prove about it. We build reproducible eval suites and run the same ones across every accelerator, so a claim about quality or speed comes with a receipt. No fabricated numbers live here — results appear as the runs finish.

What we measure

Quality

Does the quantized model still get the answer right? Task accuracy held against a higher-precision reference, so we know what 4-bit actually costs.

Throughput

Prefill and decode tokens per second under a fixed prompt and sampler — the number that decides whether a model is usable in the loop.

Footprint

Peak memory across context lengths — how much headroom is left, and how long a context the machine can actually hold.

Cost

Tokens per second per watt, and per dollar of hardware. The honest denominator when comparing a desk to a datacenter.

The matrix

Every suite, run on every platform, holding the model and settings fixed. Cells fill in as passes complete.

Suite	DGX Spark	ASUS GX10	DGX Station	Dell GB300
Quality vs. reference	—	—	—	—
Prefill throughput	—	—	—	—
Decode throughput	—	—	—	—
Peak memory	—	—	—	—
Tokens / watt	—	—	—	—

Measuring in progress — check back as suites land.

How we keep it honest

One harness, every box. The same eval code runs on each platform — differences come from the hardware, not the measurement.
Fixed prompts and sampler. Temperature, context, and the prompt set are pinned so runs are comparable and repeatable.
Precision held to account. Quantized builds are always scored against a higher-precision reference, so the trade-off is a measured number, not a guess.
Nothing published until it’s run. Empty cells stay empty. A number here means the pass actually happened on that machine.