What it does
The simulator runs the operations a decoder actually performs, matrix multiply, normalization, softmax, attention and sampling, and records for each one the bytes read, the bytes written, the floating point operations, the wall time, the lane it was assigned to, and a latency estimate derived from a hardware profile.
Profiles ship for a theoretical device and for three real ones, so the same trace can be replayed against different peak throughput and memory bandwidth figures. The output is a JSONL trace, a JSON summary and a Markdown report.
Calibration is the point
A synthetic estimate on its own is a number nobody should believe. The simulator runs a real local model through Ollama on the same machine, measures decode tokens per second, and compares that against its own estimate to produce a calibration factor, a memory bound score and a bottleneck classification.
When the real measurement is skipped, the report marks every number as a synthetic estimate rather than quietly presenting the two as equivalent.
In the code
def run_matmul(self, a: np.ndarray, b: np.ndarray, lane: str = "matmul_lane") -> np.ndarray:
if a.shape[-1] != b.shape[0]:
raise ValueError(f"matmul shape mismatch: {a.shape} x {b.shape}")
start = time.perf_counter()
y = matmul(a, b)
end = time.perf_counter()
m = int(np.prod(a.shape[:-1]))
k = int(a.shape[-1])
n = int(b.shape[-1])
flops = 2 * m * n * k
bytes_read = a.nbytes + b.nbytes
bytes_written = y.nbytes
est = self.profile.estimate_latency_ms(flops, bytes_read + bytes_written)
self.profiler.record("MATMUL", bytes_read, bytes_written, start, end,
flops=flops, estimated_lpu_ms=est, lane=lane)
return yThe value is in what gets recorded rather than in what gets computed. Bytes moved and floating point work are tracked separately for every operation, which is exactly the pair you need to say whether a workload is memory bound or compute bound, and that classification is the only interesting question about inference hardware. Because the estimate comes from a swappable profile rather than from constants baked into the operation, the same recorded trace can be asked what it would cost on a laptop, on a mid-range consumer card, or on a device that does not exist yet. The lane label is how the scheduler story gets told later: it makes the trace answer which units would have been busy, not just how long the whole thing took.
How this differs from the ordinary version
It refuses to launder estimates as measurements
The report separates what was measured from what was estimated, and says so on the page. When the live inference baseline is skipped, everything is labelled synthetic. That is unusual for hardware concept work and it is the reason the numbers are worth reading.
Calibration is against your machine, not a datasheet
The baseline comes from running a real local model through Ollama on the same box, so the calibration factor is relative to hardware you can inspect rather than to a vendor claim.
Software first, silicon later
Treating the unit as a runtime means the interesting questions, lane assignment, key-value cache pressure, where the bottleneck sits, are answerable now and answerable repeatably, without waiting for a chip.
In the field
Buying decisions before buying
Anyone deciding what to run a local model on is choosing between memory bandwidth and raw throughput without knowing which one their workload is limited by. A trace that reports bytes moved and floating point work per operation answers that in an afternoon.
Questions
- Are the latency numbers real?
- The Ollama baseline is a real measurement on the machine that ran it. The simulator latency is an estimate from a hardware profile, and the report labels it as such rather than presenting the two as the same kind of number.
- What hardware profiles ship with it?
- A theoretical device at 100 TOPS and 512 GB/s, plus profiles for an M1 laptop, an RTX 3060 Ti and an RTX 2070 Super.