04Classifiers and decoders

Temporal action segmentation (MS-TCN, ASFormer)

Models built to label every frame of a long sequence without shattering it. Computer vision met the over-segmentation problem first, and its fixes transfer.

  • Sequence
  • Learned
  • Needs labels
  • Research code
A first stage over-segments. Later stages read its predictions and merge the fragments.
Modality
Per-frame embeddings from video or tracking
Output
One label per frame, in coherent segments
Evidence
CVPR 2019 (MS-TCN); BMVC 2021 (ASFormer)
Code and weights
Official research code; no football weights
Licence
MIT (ASFormer); MS-TCN under its repository's own terms

Action segmentation labels every frame of an untrimmed video: pour, stir, cut. Its metrics reward getting segments right, not only frames, and its classic failure is over-segmentation: a model with good frame accuracy that splits one action into a dozen pieces. Replace the kitchen with a pitch and this is the possession problem, spell metric included.

MS-TCN answers with refinement in stages. A first dilated TCN predicts frame labels. Further stages take those predictions as input and predict again, each correcting the last, so later stages see label context instead of raw features. A smoothing loss penalises abrupt changes in the output between adjacent frames.

ASFormer keeps the staged design and swaps convolution for attention: an encoder for the first pass and decoders that refine it, with attention windows arranged so the model stays trainable on sequences of thousands of frames.

Where it wins

  • They attack fragmentation directly, in the architecture and in the loss.
  • Their segment-level evaluation, edit score and segmental F1, is the closest cousin of spellF1 in the literature.
  • They are full-sequence models. A long recording is processed whole, not in clips.
  • Input-agnostic. Any per-frame embedding works.

What it costs

  • Supervised, and developed on video benchmarks rather than on football tracking.
  • Smoothing is a prior with a strength. Too strong, and real one-touch possessions vanish.
  • Learned refinement is opaque. When it merges two spells there is no cost to inspect.
  • Scores are not calibrated probabilities.

How it plugs into possession

As a lesson first and a model second. The lesson: per-frame accuracy and segment quality are different objectives, and the gap between them must be closed deliberately.

MS-TCN closes it with learned refinement and a smoothing loss. A Viterbi decode closes it with an explicit switching cost and an exact optimum, and needs no training to do so.

With labels in hand, a multi-stage model is a credible emission network. Even then, a structured decoder on top is the cheapest insurance against fragments.

Our read

The clearest statement of the over-segmentation problem. Its learned fix needs labels. The same prior can be imposed exactly with a decoder.

Sources

  1. MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (CVPR 2019)openaccess.thecvf.com
  2. MS-TCN: official codegithub.com
  3. ASFormer: Transformer for Action Segmentation (arXiv 2110.08568, BMVC 2021)arxiv.org
  4. ASFormer: official codegithub.com