14,428frames tracked in one video, 100%
9fish followed at once
5body points per fish

Most automated animal tracking is done in the lab, against clean backgrounds and under even light. As an undergraduate researcher in the Hofmann Lab at UT Austin in 2025, I built a pipeline that works in the wild instead: following individual Plains Longear Sunfish (Lepomis aquilensis) through drone footage of a natural stream, where the males gather in breeding colonies and each defends a nest.

The trained model following sunfish over their nests. Each colour is one individual; the pale rings of gravel are nests. Drone footage piloted by Landon Porter.

The question

How does an animal’s behaviour and use of space vary across different environments? Answering that needs numbers: where each animal is in every frame, how it moves, and whom it meets. Pose tracking turns video into that record, and gives a quantitative basis for describing behaviour.

The tool is SLEAP, a deep-learning framework for multi-animal pose tracking (Pereira et al. 2022). It learns from frames a person has labelled by hand, then predicts the same body points on every other frame.

Why wild footage is hard

A stream is close to the worst case for a tracking model:

  • Light distortion. Ripples and dappled shade move across the whole scene.
  • A patterned bottom. Gravel, cobble and algae look a lot like fish.
  • Occlusion. Fish pass under debris, into shadow and over one another.
  • Many animals. Several species share the water, and male and female sunfish look different.
The raw view from the drone: a colony of nests on the stream bed, before any tracking. The fish are the small dark shapes over the pale gravel.

The pipeline

Each fish is described by a five-point skeleton: head, dorsal, tail base, mid-tail and tail tip. From there the work is a loop.

  1. Prepare the video from the raw drone footage.
  2. Label a small set of frames by hand.
  3. Configure two models: one finds each fish, the other places the body points on it.
  4. Train on a remote GPU server.
  5. Run the model over the whole video and link detections into tracks.
  6. Assess the result, correct its mistakes, and train again.
Predictions on a single male at his nest, viewed in SLEAP. The trailing line is his recent path.

Early models needed several rounds of that loop. After refining the labelling and the footage, a good model could be produced in a single round of training, and it tracked every frame of the video.

The refined model on higher-resolution footage.

What the tracks show

Once every fish has a position in every frame, the video becomes data.

Each fish's path over a recording, drawn over the stream bed. Most stay tightly over one nest.
Territories estimated from those paths: the outline of everywhere a fish went, with contours for where it spent most of its time.
Each fish's distance from the centre of its territory over time, in body lengths. The tall single spike is a border conflict; the regular peaks after it are patrolling.

Passing it on

A pipeline is only useful if other people can run it. I wrote the workflow up as a guide for the lab, SLEAP from Scratch, which assumes no prior knowledge and covers everything from installation to training on the remote server and exporting results.

The tracking was presented as part of the poster Resilient Brains: Behavioral and Physiological Homeostasis in Highly Variable Environments (Phofolos, Wong, Husain and Hofmann).

References

Pereira, Talmo D., et al. 2022. “SLEAP: A Deep Learning System for Multi-Animal Pose Tracking.” Nature Methods 19 (4): 486–495. https://doi.org/10.1038/s41592-022-01426-1.