Methodology¶
Data¶
MOT17 training split (Milan et al., 2016): 7 sequences, 5,330 frames, public pedestrian ground
truth. Three are static-camera (02, 04, 09), four moving (05, 10, 11, 13); the split is measured, not
assumed -- the camera-motion estimator reads 0.2-0.4 px/frame of global motion on the first group and
4.8-12.7 px on the second (results/sequences.json). Sequences range from 8 to 45 people per frame,
640x480 at 14 fps to 1080p at 30 fps.
MOT17 ships each sequence three times (DPM / FRCNN / SDP) with identical images and different public detections; only one copy is downloaded and the public detections are not used -- all detectors here are our own.
Ground truth used: pedestrian class, consider flag set (the standard MOT filter). The official test
split's ground truth is private and is not used.
Detection¶
Off-the-shelf COCO-pretrained Ultralytics YOLOv8 and YOLO11 (n/s/m), person class only, run once per (sequence, backbone, input size) down to confidence 0.1 and cached. Each tracker applies its own thresholds to that shared input, so differences between trackers are differences in association alone.
Trackers¶
All implemented in this repository (Trackers). The comparison is a ladder, so that every step isolates one idea:
| Variant | Differs from the previous rung by |
|---|---|
| ByteTrack | baseline (motion only, two-stage IoU) |
| BoT-SORT (CMC) | + camera-motion compensation |
| BoT-SORT (ReID only) | + OSNet appearance in the association cost (no CMC) |
| BoT-SORT (CMC + ReID) | both |
| DeepSORT (ResNet18) | the original baseline: gated cosine on generic ImageNet features, single-stage |
| DeepSORT (OSNet) | + a real ReID embedding |
| DeepSORT (OSNet + cascade) | + the paper's matching cascade |
Metrics¶
- HOTA (Luiten et al., 2021): geometric mean of DetA (detection) and AssA (association), averaged over 19 IoU thresholds. The headline metric because it separates "did you find people" from "did you keep their identities". Implemented from the paper; matches TrackEval to 10 decimals on a realistic scenario.
- IDF1, MOTA, ID switches, MOTP: CLEAR-MOT via
py-motmetrics(IoU distance computed locally). - Detection AP50 / AP: COCO 101-point AP against ground truth; persons below 25% visibility are ignore regions.
Tuning (and its limits)¶
Each tracker family is tuned by Optuna (60 TPE trials, seed 0) on the trailing 25% of MOT17-02 with
yolo11s detections, objective HOTA. BoT-SORT inherits ByteTrack's association thresholds and tunes only its
appearance-fusion parameters, so the CMC/ReID ablation is not confounded by extra tuning. The tuned configs are
then applied unchanged to all 42 (sequence, backbone) cells.
Because tuning used part of MOT17-02, every table that matters also reports a held-out mean over the other six sequences.
Evaluation protocol¶
Scored on the whole sequence (tuning only used a slice of one). IoU matching threshold 0.5 for CLEAR-MOT. No distractor-class filtering (so absolute scores are not comparable to MOTChallenge leaderboard numbers; relative comparisons within this repository are what the benchmark supports).
Edge benchmark¶
Separate processes, 150 frames from six 1080p sequences, 5 warm-up frames, AP measured on the same frames, calibration frames disjoint from evaluation frames (Edge deployment).
Reproducibility¶
Seeds are fixed (Optuna 0, NumPy 0); runs are cache-driven and deterministic given the caches. Detector caches
depend on the GPU backend's numerics, so a CPU re-run can differ in the last decimals of confidence scores.
Every figure and results page is regenerated by script from results/.