Compiler benchmark

Every pair of twelve circuits (four QEC memory experiments, six Clifford circuits, two small non-Clifford ones) and the nine reference devices. A compiler is ranked by how much faster its programs run than the reference compiler's, over the pairs both compile; a single wrong program makes an entry ineligible.

The reference compiler, pair by pair

Each cell is one pair: the round time in milliseconds (T_jones) where qccdc's program is valid; wrong where it breaks one of the device's rules (hover for which); refused where it cannot compile the pair. Every one of these is a place another compiler can do better.

circuit	chain	cyclone_base	cyclone_dual_loop	deck_unit_cell	grid9x9	h2_racetrack	ladder_2x72	ring144_24v	stationary_chain
rep3_r3_z memory, 5 qubits	4.18	4.43	4.60	5.86	5.86	4.09	5.11	19.59	4.89
rep5_r5_z memory, 9 qubits	6.91	8.66	7.50	13.30	13.30	6.65	10.82	96.67	14.85
surface3_r3_z memory, 17 qubits	refused	41.33	refused	39.06	39.06	69.39	79.28	117.31	32.41
steane_r3_z memory, 16 qubits	refused	62.98	refused	57.44	57.44	89.17	100.14	106.83	40.45
bell2 clifford, 2 qubits	0.87	0.87	0.87	0.99	0.99	0.87	0.87	1.33	0.87
ghz8 clifford, 8 qubits	4.65	4.39	4.65	5.45	5.45	4.65	4.85	7.42	1.48
ghz16 clifford, 16 qubits	9.69	9.21	9.69	11.45	11.45	9.69	9.99	15.54	3.73
ghz32 clifford, 32 qubits	19.77	19.67	19.77	23.41	23.41	19.77	20.37	50.51	refused
bv6 clifford, 7 qubits	2.82	2.65	2.81	3.25	3.25	2.81	2.92	4.36	2.92
clifford12 clifford, 12 qubits	refused	58.09	refused	60.16	60.16	61.49	73.78	90.23	40.13
qft6 general, 6 qubits	refused	15.02	17.23	17.82	17.82	17.07	19.11	25.04	10.94
adder3 general, 10 qubits	20.48	18.82	30.82	28.80	28.80	19.53	23.93	40.23	16.76

The contract

A compiler is a directory with qccd-compiler.json. The grader runs it once per pair, with no network, a time and memory limit, and nothing it can reach but the pair's files:

<entry> --circuit circuit.qasm --device device.arch.json --expanded device.expanded.json --out DIR

`DIR/program.tsir.json`	required on exit 0: native gates R, VZ, MS; meta.qubit_map
`DIR/certificate.qcert.json`	optional; reported, never required
exit 0	compiled
exit 4	refused (costs coverage, not eligibility)
exit other	crash

{
 "kind": "qccd.compiler",
 "version": 1,
 "name": "my-compiler",
 "about": "starts as the reference compiler (qccdc + cooling): make it your own",
 "runtime": "python3",
 "entry": [
  "python3",
  "<your script>"
 ],
 "build": null
}

Runtimes: python3, native. Cooling is the compiler's decision; the heating budget (rule R7) is checked like every other rule.

Submit a compiler

qccd bench init-compiler my-compiler      # a working one to start from: the reference
qccd bench run --compiler my-compiler       # grade it on the public pairs, on your machine
qccd bench publish my-compiler --server https://qccd.academy/official

The run grades every pair locally, exactly as the server will (the hidden pairs aside), and your agent can read the report (qccd_get_bench). Running a compiler runs your code, so it happens in your shell, never through the agent's tools. Publishing needs you at the terminal: it shows the compiler's digest and asks you to type it back. On the server the compiler runs in a separate sandbox container with no network and no secrets.