Evaluation

A design is judged by numbers: how long its programme runs, how hot it makes its ions, how much hardware it takes, and how often the logical qubit it stores is lost. On this website every one of those numbers is a term of a small language, printed on this page, evaluated on three texts: the device, the programme, and the evaluation. Nothing else is read, so every number on the leaderboard can be recomputed from the files it came from, and traced to the line of text that defines it.

The language is the Rules page’s, extended. A rule is a formula that holds or fails; an evaluation adds named numbers (metric), the assertions a design must meet to be ranked (require), and noise (channel): a clause that says which Pauli error goes where in the circuit a run becomes. The noisy circuit the Goal 1 board samples is the programme’s stabilizer translation with those clauses applied. The grader reads these texts: every number it publishes — the metrics, the hardware budget, the noisy circuit and its error budget — is computed by the interpreter of the language from the text on this page, and the same interpreter runs in every workspace. The Design page reads the same texts in the browser with a second interpreter, held to the first metric by metric: every number it shows (cost, steps, runtime, heat, DACs, the noise each budget line books) is a metric of these texts, and its Report checks itself against the grader’s values.

• Three texts and one function
• The evaluation language
• The words of an evaluation
• The state a programme leaves
• Speed: how long a programme runs
• Heat: what the motion costs
• Hardware: electrodes, switches and DACs
• From a run to a stim circuit
• The noise model, qccd-noise@2
• A Goal 1 entry, recomputed
• The noiseless check
• The logical error rate and its interval
• Reading the Goal 1 board
• What is proved, decided, computed and estimated
• Writing your own evaluation

Three texts and one function

The evaluation language

An evaluation is a text file. Its terms and formulas are the rule language’s, unchanged — quantifiers over finite sets, comparisons, arithmetic, count, sum, max, min and set builders — with one more term, if C then a else b. What it adds are items:

file      = { item } ;
item      = "evaluation" STRING | "title" STRING | "about" STRING
          | "include" STRING | "clock" STRING | "rules" STRING | "rank" NAME
          | "param" NAME "=" expr
          | "define" NAME [ "(" NAME { "," NAME } ")" ] "=" expr
          | metric | requirement | channel ;
metric    = "metric" NAME STRING { "unit" STRING | "better" ( "low" | "high" ) | "clock" STRING
                                 | "group" STRING | "about" STRING | "source" STRING }
            "=" expr ;
requirement = "require" ID STRING { "about" STRING | "source" STRING } formula ;
channel   = "channel" NAME STRING { "about" STRING | "source" STRING }
            ( "before" | "after" ) binders [ "where" formula ] ":"
            NOISE "(" expr ")" "on" expr [ "counted" "as" budget ] ;
budget    = NAME | NAME "=" expr { "," NAME "=" expr } ;
NOISE     = "X_ERROR" | "Y_ERROR" | "Z_ERROR" | "DEPOLARIZE1" | "DEPOLARIZE2" ;
primary   = ... the rule language's primaries ...
          | "if" formula "then" expr "else" expr ;

evaluation "goal1"
title "Goal 1: the best QEC memory under 200 ions"
about "Every curated memory experiment under 200 ions, compiled for a design, run under qccd-noise@2 on the cited Helios-class physics, and ranked by its logical error rate per round per logical qubit."
include "board"
include "qccd-noise@2"
clock "quantinuum"
rules "standard"
rank ler_per_round_per_logical

# the release's budget: its fixed physics, which a design may not change
param max_dacs = 128
param max_junctions = 200
param max_ions = 200

require G1 "Every rule of the standard rule set holds"
  about "The programme is judged by the Rules page's standard set, rule by rule."
  rules_hold()

require G2 "No more than 200 ions"
  about "The board's own filter: an entry over it is graded, and not listed."
  n_ions <= max_ions

require G3 "With its noise taken out, the circuit is the experiment"
  about "Every detector and every logical observable has a fixed value, and it is 0: the value the experiment's own circuit gives it."
  noiseless_ok()

The words of an evaluation

Every word a term may use, with what it means. A word the rule language also has means the same thing here. The instants are the evaluation’s own: the start and the end of every step, because a number like the hottest an ion gets needs the state at the end of a step, before a cooling resets it.

The device

The device's wiring

The programme

Instants and the state

The device's laws

The experiment

The sample

Arithmetic

The state a programme leaves

Every metric reads the run, and the run is the replay: the semantics of the hardware language, the same one the rules judge. It is worth writing out, because every number below inherits it.

p.init({"d0": "T0_0v", "d1": "T0_1h"})
p.shuttle("d0", ["T0_0v", "J0_1", "T0_1h"])
p.cool()
p.gate("CX", [["d0", "d1"]])

Two clocks appear on this page. The small examples run on the website’s default device document; Goal 1 runs on the cited Helios-class physics every one of its releases fixes (clock "quantinuum"), and a design may not change it.

Speed: how long a programme runs

Time is the first thing a design spends. Under qccd-noise@2 every microsecond an ion is not operated on is memory error, and the coherent part grows with the square of each wait, so a slow schedule pays twice. The speed metrics read the clock and the steps:

metric T = time(final()) / 1000

metric T_jones = time(final()) / 1000

metric T_transport = time(final()) / 1000

metric T_quantinuum = time(final()) / 1000

metric time_transport = sum(duration(k) for k in steps where kind(k) = "simd") / 1000

metric time_gates = sum(duration(k) for k in steps where kind(k) = "gate") / 1000

metric time_cooling = sum(duration(k) for k in steps where kind(k) = "cool") / 1000

metric time_spam = sum(duration(k) for k in steps where kind(k) in {"measure", "reset"}) / 1000

metric instructions = count(k in steps)

metric two_ion_gates = count(g in two_qubit_gates)

metric transport_instructions = count(k in steps where is_transport(k))

metric ions_per_instruction = if transport_instructions = 0 then 0
    else sum(loop_riders(k) for k in steps where is_transport(k)) / transport_instructions

metric cost = count(h in hops) + count(h in hops where degree(to(h)) >= 3) + sum(size(entails(o)) for o in moves)

metric depth = sum(depth(k) for k in steps)

define is_transport(k) = kind(k) = "simd" and (rotation(k) or loop_riders(k) > 0)

ions_per_instruction is how much of the machine one waveform moves: on a broadcast-wired device a rotation moves every ion on its loop for the price of one instruction, which is why the rings run their rounds in few instructions.

p.init({"d0": "C3", "d1": "C0"})
p.shuttle("d0", ["C3", "C2", "C1", "C0"])
p.cool()
p.gate("CX", [["d0", "d1"]])
p.measure(["d0", "d1"])

p.init({"d0": "T0_0v", "d1": "T0_1h"})
p.shuttle("d0", ["T0_0v", "J0_1", "T0_1h"])
p.cool()
p.gate("CX", [["d0", "d1"]])

p.init({"d0": "S0", "d1": "S4"})
p.simd("shuttle", [["d0", "S0", "S1"], ["d1", "S4", "S5"]])
p.simd("shuttle", [["d0", "S1", "S2"], ["d1", "S5", "S6"]])

p.fill()
p.rotate(2)

Heat: what the motion costs

Every operation on a trapped-ion machine is paid for in motional quanta. A two-ion gate shares a motional mode between its ions, and a hot mode blurs the spin-dependent force that entangles them, so the gate’s error grows with the heat it meets (rule R16), and rule R7 refuses a gate above the device’s budget. The device’s law is part of the metrics file:

define eps0 = 1 - primitive("ms_gate", "fidelity_at_n0")
define kappa = slope("ms_gate")
define Nref = chain_reference("ms_gate")
define chain_factor(N) = max(max(N, 2) / ln(max(N, 2)), Nref / ln(Nref))
define chain_ratio(N) = chain_factor(N) / chain_factor(Nref)
define ms_error(n, N) = if Nref = none or chain_ratio(N) = 1 then eps0 + kappa * max(n, 0)
    else (eps0 - kappa / 2) + kappa / 2 * chain_ratio(N) * (2 * max(n, 0) + 1)
define gate_nbar(g) = max(nbar(i, start(g)) for i in ions(g))
define gate_chain(g) = occupancy(site(g), start(g))

ε(n̄, N) = ε₀ + κ·n̄ in a chain no longer than the reference; in a longer one, ε = (ε₀ − κ/2) + (κ/2)·r(N)·(2n̄ + 1), where r(N) is Murali’s laser-instability factor N/ln N relative to the reference length (arXiv:2004.04706, Eq. 1): the two forms are the same function at the reference, and a longer chain is never better than it.

metric peak_nbar = max(max(nbar(i, end(k)) for i in ions) for k in steps)

metric gate_nbar_max = if count(g in two_qubit_gates) = 0 then 0 else max(gate_nbar(g) for g in two_qubit_gates)

metric gate_nbar_mean = if count(g in two_qubit_gates) = 0 then 0
    else sum(gate_nbar(g) for g in two_qubit_gates) / count(g in two_qubit_gates)

metric junction_crossings = count(h in hops where degree(to(h)) >= 3)

metric transport_quanta = deposited("shuttle") + deposited("junction") + deposited("split_merge")

metric total_quanta = deposited("shuttle") + deposited("junction") + deposited("split_merge") + deposited("gate")
    + deposited("anomalous")

metric n_coolings = count(o in coolings)

metric gate_error_sum = sum(ms_error(gate_nbar(g), gate_chain(g)) for g in two_qubit_gates)

Hardware: electrodes, switches and DACs

The half of a design that is not time or heat is what it costs to build and drive. Each trap carries DC electrodes that shape its wells; each electrode is wired through a two-way switch; and each needs a digital-to-analogue converter (DAC) to play its waveform — one per electrode when wired directly, which is the first thing that does not scale. Broadcast wiring shares one DAC among every electrode that plays the same role, which holds the count near constant at the price that one cycle carries one motion (rule R22). The floors below are the deck’s array (pp. 19–20) and WISE (arXiv:2305.12773): a device may declare more, never less.

param least_electrodes_per_trap = 24
param least_compensation_per_trap = 8
param most_shims_per_dac = 100
param least_channels_per_site = 32

metric n_sites = count(s in sites)

metric n_junctions = count(n in junctions)

metric n_rails = count(r in rails)

metric places = sum(capacity(n) for n in nodes)

metric electrodes = if channel_map() then map_electrodes() else n_sites * per_trap + n_junctions * per_junction

metric switches = if channel_map() then map_switches() else 2 * (n_sites * per_trap + n_junctions * per_junction)

metric dacs = broadcast_dacs + compensation_dacs

define scheme = wiring("scheme", "direct")
define broadcast_wiring = scheme in {"wise", "broadcast_groups"}
define per_trap = if channel_map() or broadcast_wiring then wiring("electrodes_per_trap", 24)
    else max(wiring("electrodes_per_trap", 24), least_electrodes_per_trap)
define per_junction = if channel_map() or broadcast_wiring then wiring("electrodes_per_junction", 48)
    else max(wiring("electrodes_per_junction", 48),
             budget("junction_electrode_multiplier", 2) * per_trap)
define least_compensation = ceil(n_sites * least_compensation_per_trap / most_shims_per_dac)
define least_driven = min(driven(s) for s in sites where driven(s) > 0)
define broadcast_dacs = if channel_map() then
      (if count(s in sites where driven(s) > 0) > 0 and least_driven < least_channels_per_site
       then ceil(shared_channels() * least_channels_per_site / least_driven)
       else shared_channels())
    else if broadcast_wiring then max(wiring("dacs_dynamic", 0), least_channels_per_site)
    else 0
define compensation_dacs = if channel_map() then max(compensation_channels(), least_compensation)
    else if broadcast_wiring then
      max(ceil(n_sites * wiring("compensation_electrodes_per_trap", 8) / wiring("shim_per_dac", 1)),
          least_compensation)
    else electrodes

evaluation "board"
title "A board's measure: the metrics, and the hardware a design may use"
about "Every task release fixes a hardware budget with its physics: at most so many DACs and so many junctions. A design over it keeps its numbers, and is not ranked. The grader judges every entry's device by these two requirements, with the release's numbers."
include "metrics"

# the device's own budget block, unless the release sets these (the grader does)
param max_dacs = budget("max_dacs", none)
param max_junctions = budget("max_junctions", none)

require B1 "The wiring fits the DAC budget"
  about "The DACs the wiring needs, counted with the physics' floors, are at most the budget's."
  max_dacs = none or dacs <= max_dacs

require B2 "The device fits the junction budget"
  max_junctions = none or n_junctions <= max_junctions

Chip area is not a metric: no grader measures it, so no design is ranked by it.

From a run to a stim circuit

A memory experiment is a circuit, written in OpenQASM, together with its detectors — parities of its measurement results that are 0 when nothing went wrong — and its logical observables. To learn how often the logical qubit is lost on a design, the compiled programme is turned back into a circuit, with noise where the run says it belongs, and sampled.

The noise model, qccd-noise@2

What a schedule costs, so that two machines running one circuit get different logical error rates when they spend different time idle, in transport or beside a readout. Every number is a published measurement on trapped-ion QCCD hardware, or arithmetic on two of them; every channel is a Pauli channel stim samples directly: a coherent rotation enters as its Pauli twirl, leakage as the Pauli error the vendor's own stabilizer emulator charges it as. What the literature does not support is left out rather than given an invented number: no error per shuttle, junction, split or swap (transport costs its time and its heat, which the channels below charge), no ion loss, no two-ion-gate crosstalk.

param memory_rate = 7.9e-3   # = 0.0079
param coherent_rate = 0.058   # = 0.058
param nbar_floor = 0.05   # = 0.05
param crosstalk_near = 2.1e-4   # = 0.00021
param crosstalk_far = (97 * 4.8e-5 - 3 * 2.1e-4) / (97 - 3)   # = 4.283e-05
param crosstalk_reset = 3.5e-6   # = 3.5e-06

define near(k) = {i for i in ions where position(i, start(k)) in {position(j, start(k)) for j in operated(k)}
                         and i not in operated(k)}
define far(k) = {i for i in ions where i not in operated(k) and i not in near(k)}

channel prepare "The first preparation"
  after p in preparations:
    X_ERROR(min(1 / 2, scale * primitive("reset", "error"))) on ions(p) counted as reset

channel memory_coherent "Coherent dephasing over a wait"
  before o in qubit_operations, i in ions(o):
    Z_ERROR(min(1 / 2, scale * pow(sin(pi * coherent_rate * waited(i, o) * 1e-6), 2))) on i

channel readout_z "A wrong readout"
  before m in measurements where basis(m) = "Z":
    X_ERROR(min(1 / 2, scale * (1 - primitive("measure", "fidelity")))) on ions(m) counted as measure

channel readout_x "A wrong readout, in the X basis"
  before m in measurements where basis(m) = "X":
    Z_ERROR(min(1 / 2, scale * (1 - primitive("measure", "fidelity")))) on ions(m) counted as measure

channel ms "The two-ion gate"
  after g in two_qubit_gates:
    DEPOLARIZE2(min(15 / 16, scale * ms_error(gate_heat(g), gate_length(g)))) on ions(g)
      counted as ms_base = ms_error(0, 2),
                 ms_heating = ms_error(gate_heat(g), 2) - ms_error(0, 2),
                 ms_chain = ms_error(gate_heat(g), gate_length(g)) - ms_error(gate_heat(g), 2)

channel gate_1q "A one-ion pulse"
  after g in single_qubit_gates where gate(g) != "VZ" and gate(g) != "I":
    DEPOLARIZE1(min(3 / 4, scale * (1 - primitive("1q_gate", "fidelity")))) on ions(g)

channel reset "A wrong reset"
  after r in resets:
    X_ERROR(min(1 / 2, scale * primitive("reset", "error"))) on ions(r)

channel crosstalk "Readout crosstalk in the trap"
  after k in steps where kind(k) = "measure":
    DEPOLARIZE1(min(3 / 4, scale * crosstalk_near)) on near(k)

channel crosstalk_global "Readout crosstalk elsewhere"
  after k in steps where kind(k) = "measure":
    DEPOLARIZE1(min(3 / 4, scale * crosstalk_far)) on far(k)

channel crosstalk_reset "Reset crosstalk"
  after k in steps where kind(k) = "reset":
    DEPOLARIZE1(min(3 / 4, scale * crosstalk_reset)) on far(k) counted as crosstalk_global

channel memory "Memory error while the clock runs"
  after k in steps where duration(k) > 0:
    Z_ERROR(min(1 / 2, scale * memory_rate * duration(k) * 1e-6)) on {i for i in ions where i not in operated(k)}

Read as a list, the model says where every error a trapped-ion machine is measured to make enters the circuit: at the gates, at the readouts and resets, beside a readout, and with the clock. What it does not say is said in its about: no error per rail, junction, split or swap (transport pays in time and heat, which memory, memory_coherent and ms charge), no ion loss, no crosstalk from a two-ion gate.

evaluation "qccd-noise@1"
title "Gate noise and dephasing, from the device's own physics"
about "Deliberately simple: two-ion gates priced by the heat they meet and the chain they sit in, one-ion pulses, readout and reset by their infidelities, and every ion dephasing for as long as each step takes. The first preparation is noiseless, and nothing is charged for crosstalk or for the shape of an idle stretch."
include "metrics"
clock "qccdsim_jones"

# The model's knobs.  The official model is scale 1 with heating, chain length and idling
# priced against the species' own coherence time; a sweep may set them otherwise.
param scale = 1                     # every probability is multiplied by this (0: noiseless)
param heating = true                # price the n-bar a two-ion gate meets
param chain = true                  # price the length of the chain in its trap
param idle = true                   # dephase every ion for as long as each step takes
param t2 = coherence_time()         # seconds; the species' T_coh unless a sweep sets it

define gate_heat(g) = if heating then gate_nbar(g) else 0
define gate_length(g) = if chain then gate_chain(g) else 2

channel readout_z "A wrong readout"
  before m in measurements where basis(m) = "Z":
    X_ERROR(min(1 / 2, scale * (1 - primitive("measure", "fidelity")))) on ions(m) counted as measure

channel readout_x "A wrong readout, in the X basis"
  before m in measurements where basis(m) = "X":
    Z_ERROR(min(1 / 2, scale * (1 - primitive("measure", "fidelity")))) on ions(m) counted as measure

channel ms "The two-ion gate"
  about "The device's gate error law at the heat the pair meets and the chain in its trap."
  after g in two_qubit_gates:
    DEPOLARIZE2(min(15 / 16, scale * ms_error(gate_heat(g), gate_length(g)))) on ions(g)
      counted as ms_base = ms_error(0, 2),
                 ms_heating = ms_error(gate_heat(g), 2) - ms_error(0, 2),
                 ms_chain = ms_error(gate_heat(g), gate_length(g)) - ms_error(gate_heat(g), 2)

channel gate_1q "A one-ion pulse"
  after g in single_qubit_gates where gate(g) != "VZ" and gate(g) != "I":
    DEPOLARIZE1(min(3 / 4, scale * (1 - primitive("1q_gate", "fidelity")))) on ions(g)

channel reset "A wrong reset"
  after r in resets:
    X_ERROR(min(1 / 2, scale * primitive("reset", "error"))) on ions(r)

channel idle "Dephasing while the clock runs"
  about "Pure dephasing of every ion over every step's duration, against the species' coherence time; the appended readout takes one measurement's time."
  after k in steps + readout where idle and duration(k) > 0 and t2 != none:
    Z_ERROR(min(1 / 2, scale * ((1 - exp(-duration(k) * 1e-6 / t2)) / 2))) on ions

channel idle "Dephasing while the clock runs"
  after k in steps + readout where idle and duration(k) > 0 and t2 != none:
    Z_ERROR(min(1 / 2, scale * ((1 - exp(-duration(k) * 1e-6 / t2)) / 2))) on ions

A Goal 1 entry, recomputed

The repetition code’s two-basis memory, three rounds, ten ions, on a twelve-site linear chain: one of the entries on the Goal 1 board, submitted by John Zhuoyang Ye (os_30293bca0826939b). Its bundle — the device, the programme and the certificate — is evaluated here by the text above, and set beside the official report.

The circuit is 627 lines over 10 qubits, with 18 measurements (the programme’s 12 and the appended readout’s 6). The first steps, as the circuit writes them, with what wrote each line; probabilities are shown to four figures, and the whole circuit has them exactly:

R 0 1 2 3 4 5 6 7 8 9                ← translation
X_ERROR(0.0005) 0 1 2 3 4 5 6 7 8 9  ← prepare

(nothing on the qubits)

Z_ERROR(2.37e-06) 0 1 2 3 4 5 6 7 8 9  ← memory

Z_ERROR(1.754e-06) 0 1 2 3 4 5 6 7 8 9  ← memory

Z_ERROR(9.047e-09) 3           ← memory_coherent
Z_ERROR(9.047e-09) 4           ← memory_coherent
Z_ERROR(9.047e-09) 5           ← memory_coherent
Z_ERROR(9.047e-09) 0           ← memory_coherent
SQRT_Y_DAG 3                   ← translation
DEPOLARIZE1(2.5e-05) 3         ← gate_1q
SQRT_Y_DAG 4                   ← translation
DEPOLARIZE1(2.5e-05) 4         ← gate_1q
SQRT_Y_DAG 5                   ← translation
DEPOLARIZE1(2.5e-05) 5         ← gate_1q
SQRT_Y 0                       ← translation
DEPOLARIZE1(2.5e-05) 0         ← gate_1q
Z_ERROR(3.95e-08) 1 2 6 7 8 9  ← memory

S 3                               ← translation
S 3                               ← translation
S 4                               ← translation
S 4                               ← translation
S 5                               ← translation
S 5                               ← translation
Z_ERROR(1.185e-07) 0 1 2 6 7 8 9  ← memory

Z_ERROR(2.37e-06) 0 1 2 3 4 5 6 7 8 9  ← memory

Z_ERROR(3.294e-09) 0               ← memory_coherent
Z_ERROR(2.354e-08) 6               ← memory_coherent
SQRT_XX 0 6                        ← translation
DEPOLARIZE2(0.0008375) 0 6         ← ms
Z_ERROR(5.53e-07) 1 2 3 4 5 7 8 9  ← memory

SQRT_X_DAG 0                         ← translation
DEPOLARIZE1(2.5e-05) 0               ← gate_1q
Z_ERROR(3.95e-08) 1 2 3 4 5 6 7 8 9  ← memory

Z_ERROR(8.3e-13) 6                   ← memory_coherent
SQRT_X_DAG 6                         ← translation
DEPOLARIZE1(2.5e-05) 6               ← gate_1q
Z_ERROR(3.95e-08) 0 1 2 3 4 5 7 8 9  ← memory

Z_ERROR(3.738e-08) 6                      ← memory_coherent
X_ERROR(1e-06) 6                          ← readout_z
M 6                                       ← translation
DEPOLARIZE1(4.283e-05) 0 1 2 3 4 5 7 8 9  ← crosstalk_global
Z_ERROR(9.48e-07) 0 1 2 3 4 5 7 8 9       ← memory

Z_ERROR(8.08e-08) 6                     ← memory_coherent
R 6                                     ← translation
X_ERROR(0.0005) 6                       ← reset
DEPOLARIZE1(3.5e-06) 0 1 2 3 4 5 7 8 9  ← crosstalk_reset
Z_ERROR(7.9e-08) 0 1 2 3 4 5 7 8 9      ← memory

Z_ERROR(8.017e-07) 0  ← memory_coherent
Z_ERROR(1.935e-07) 1  ← memory_coherent
Z_ERROR(5.803e-08) 2  ← memory_coherent
Z_ERROR(4.303e-07) 3  ← memory_coherent
Z_ERROR(5.803e-08) 4  ← memory_coherent
Z_ERROR(1.912e-09) 5  ← memory_coherent
X_ERROR(1e-06) 0      ← readout_z
X_ERROR(1e-06) 1      ← readout_z
X_ERROR(1e-06) 2      ← readout_z
Z_ERROR(1e-06) 3      ← readout_x
Z_ERROR(1e-06) 4      ← readout_x
Z_ERROR(1e-06) 5      ← readout_x
M 0                   ← translation
M 1                   ← translation
M 2                   ← translation
MX 3                  ← translation
MX 4                  ← translation
MX 5                  ← translation

Reading it: the preparation resets every qubit and charges each a reset’s error; a transport step writes no gate, only the memory error of everyone who waited through it; a pulse is its stim gate followed by its infidelity; a two-ion gate is SQRT_XX followed by the gate law at its heat; a readout is preceded by the coherent dephasing of each ion’s wait and the readout flip, and followed by the crosstalk on the ions beside and elsewhere.

The same check on the distance-3 surface code’s entry on a 9×9 grid (os_4c75a8b07249aea2, 3,195 lines, 34 qubits): T 87.68 ms against the report’s 87.68 (equal), 44 DACs against 44 (equal), 0.2169 expected faults against 0.2169 (equal); its circuit.

The noiseless check

The logical error rate and its interval

param z95 = 1.959963984540054

define wilson_centre(k, n) = (k / n + z95 * z95 / (2 * n)) / (1 + z95 * z95 / n)
define wilson_half(k, n) = z95 * sqrt(k / n * (1 - k / n) / n + z95 * z95 / (4 * n * n)) / (1 + z95 * z95 / n)
define wilson_lo(k, n) = if n = 0 then 0 else if k = 0 then 0 else max(0, wilson_centre(k, n) - wilson_half(k, n))
define wilson_hi(k, n) = if n = 0 then 1 else if k = n then 1 else min(1, wilson_centre(k, n) + wilson_half(k, n))
define per_round(p) = if rounds() <= 1 then p else if p >= 1 then 1 else 1 - pow(1 - p, 1 / rounds())

metric ler = failures() / shots()

metric ler_per_round = per_round(ler)

metric ler_per_round_per_logical = ler_per_round / n_logical

metric ler_per_round_lo = per_round(wilson_lo(failures(), shots()))

metric ler_per_round_hi = per_round(wilson_hi(failures(), shots()))

Reading the Goal 1 board

Goal 1 asks for the best quantum memory under 200 ions. Every curated memory experiment is one release, every release is graded by the evaluation above, and every entry of every release is ranked on one list by ler_per_round_per_logical.

metric n_ions = count(i in ions)

metric n_logical = logical_qubits()

metric ions_per_logical_qubit = n_ions / n_logical

An entry is on the board only when it is eligible and places at most 200 ions. Five conditions are requirements of the evaluation, decided on this page’s terms: the wiring fits 128 DACs (B1) and the device 200 junctions (B2), the rules hold (G1), the programme places at most 200 ions (G2, the board’s own filter rather than a stage of grading), and the noiseless check holds (G3). The other stages are about the submission itself, and the grader decides them before any number is computed:

What is proved, decided, computed and estimated

The replay’s heat is an upper bound (rule R15): excitations from successive motions are added, and the interference that can cancel them is not modelled. The noise model’s numbers are measurements on one machine family, cited channel by channel; a design is ranked by them as a machine built to that family would run it.

Writing your own evaluation

An evaluation may start from another (include), change its numbers (param), and add metrics, requirements and channels of its own, in the language above and with the words above, and nothing else. A word the vocabulary does not have is refused with the line it is on.

evaluation "mine"
title "Goal 1, with a price on every junction crossing"
include "goal1"
param junction_error = 1e-4      # per crossing: an assumption, not a measurement

channel junction "A junction crossing dephases the ion"
  after o in moves where size(crossed(o)) > 0:
    Z_ERROR(1 - pow(1 - junction_error, size(crossed(o)))) on ions(o)

metric T_us "Execution time in microseconds" unit "us" better low
  = time(final())

On the 9×9 grid entry above, whose ions cross 455 junctions, the new channel writes 455 lines and adds 0.0455 expected faults to the 0.2169 the board’s model charges.