05Family 05 of 05
Proof
A design is a hypothesis until it is measured on matches it has never seen, by a rule written before the numbers came in.
- 3studies in this family
- 3stages of selection
- 9questions the field has not settled
How belief accumulates
No single test makes a design credible. Belief is built in gates, and each gate proves one thing and nothing more.
Every claim becomes a test
The other four families make claims. This one turns each claim into the test that could refute it, on the same bench.
Three kinds of proof
| Question | What settles it | Study |
|---|---|---|
| Is the design better? | Both official metrics, on whole held-out matches, selected by the published rule | Two metrics, one verdict |
| Does it know when it is unsure? | Three confidences kept apart, and a record of where every value came from | Confidence and provenance |
| What has nobody proved yet? | Nine open questions, each with the experiment that would close it | Open frontiers |
How an architect chooses here
- The unit is the match. Neighbouring frames are near-copies. Splitting a match by frames leaks the answer.
- Write the rule first. Select by the published ranking rule, never by a blend invented after the scores.
- Degrade the labelled data to the target. A design validated with a ball that is always tracked has not met the problem.
- Publish what reproduces. If someone else cannot regenerate the number from public data, it is a claim.
What this family cannot do
Proof cannot rescue a design. It can only tell the truth about one. The designs themselves start on the map.
In this family3
-
How a possession design earns belief when its two official metrics disagree and its fixture cannot be tuned on. The lenses to report, the stages of selection, and the rule that ...
- Evaluation design
- Pareto
- Held-out matches
-
Three different uncertainties hide behind the word confidence: was it seen, is it the same instant, and which state explains it. Provenance keeps them apart and keeps the answer...
- Uncertainty
- Provenance
- Leakage
-
Nine questions possession inference has not settled, each with the experiment that would settle it. This is where the next gains are.
- Open questions
- Experiments
- Field