04Classifiers and decoders
Temporal action segmentation (MS-TCN, ASFormer)
Models built to label every frame of a long sequence without shattering it. Computer vision met the over-segmentation problem first, and its fixes transfer.
- Sequence
- Learned
- Needs labels
- Research code
- Modality
- Per-frame embeddings from video or tracking
- Output
- One label per frame, in coherent segments
- Evidence
- CVPR 2019 (MS-TCN); BMVC 2021 (ASFormer)
- Code and weights
- Official research code; no football weights
- Licence
- MIT (ASFormer); MS-TCN under its repository's own terms
Action segmentation labels every frame of an untrimmed video: pour, stir, cut. Its metrics reward getting segments right, not only frames, and its classic failure is over-segmentation: a model with good frame accuracy that splits one action into a dozen pieces. Replace the kitchen with a pitch and this is the possession problem, spell metric included.
MS-TCN answers with refinement in stages. A first dilated TCN predicts frame labels. Further stages take those predictions as input and predict again, each correcting the last, so later stages see label context instead of raw features. A smoothing loss penalises abrupt changes in the output between adjacent frames.
ASFormer keeps the staged design and swaps convolution for attention: an encoder for the first pass and decoders that refine it, with attention windows arranged so the model stays trainable on sequences of thousands of frames.
Where it wins
- They attack fragmentation directly, in the architecture and in the loss.
- Their segment-level evaluation, edit score and segmental F1, is the closest cousin of spellF1 in the literature.
- They are full-sequence models. A long recording is processed whole, not in clips.
- Input-agnostic. Any per-frame embedding works.
What it costs
- Supervised, and developed on video benchmarks rather than on football tracking.
- Smoothing is a prior with a strength. Too strong, and real one-touch possessions vanish.
- Learned refinement is opaque. When it merges two spells there is no cost to inspect.
- Scores are not calibrated probabilities.
How it plugs into possession
As a lesson first and a model second. The lesson: per-frame accuracy and segment quality are different objectives, and the gap between them must be closed deliberately.
MS-TCN closes it with learned refinement and a smoothing loss. A Viterbi decode closes it with an explicit switching cost and an exact optimum, and needs no training to do so.
With labels in hand, a multi-stage model is a credible emission network. Even then, a structured decoder on top is the cheapest insurance against fragments.
Our read
The clearest statement of the over-segmentation problem. Its learned fix needs labels. The same prior can be imposed exactly with a decoder.