← Back to all projects
Case study · Sept – Nov 2026 · TU Delft Joint Interdisciplinary Project

Teaching a rail trolley to see

YARC is a start-up building an affordable robot that rides the rails and inspects them. In a five-person team I owned the camera perception subsystem: getting clean footage off a hand-pushed trolley, turning it into a dataset I could trust, training the detectors, and running them live on the track.

3camera views,
one detector each
16recording runs
over 4 tracks
1,529top-down frames
extracted
0.986mAP50 on a
held-out track*
63 msper image on a
laptop CPU

*Top-down model, common classes (sleepers, fasteners, bolts), small validation set. The details are below.

01 · The idea

Detect components first, infer defects second

Defects are rare. A detector trained directly on “defect” would have almost nothing to learn from. Components are different: sleepers, fasteners and bolts show up in every frame and look distinct, so a few hundred labelled boxes are enough to teach a model to find them.

Missing parts can then be inferred by rules: a sleeper side with no fastener, a concrete fastener with no clip, a baseplate with fewer bolts than its twin. An alarm can show which rule fired, which supports the inspector's judgement instead of replacing it.

The three cameras see the track so differently that one model would not serve all of them, so each view has its own detector. All are YOLO26 models from the Ultralytics library, fine-tuned from COCO-pretrained weights. With only dozens of labelled images, transfer learning was the practical choice.

CameraViewLooks forModel
HD webcamTop-down over one railSleepers, fasteners, bolts, clips; missing bolts and fastenersYOLO26n, trained
4K webcamForward and downSleeper condition, switches, displaced clipsYOLO26s, trained
360° cameraAll round, mounted highPeople and vehicles near the trackYOLO26n, COCO-pretrained
Top-down view of a rail with boxes around a wooden sleeper, four fasteners and their bolts
The top-down model on a frame from the held-out track: sleeper, fasteners and their bolts all found.
A simple trolley on a railway track carrying a laptop and cameras
Day one was a dry run with an improvised rig. The 4K camera sat on the laptop screen, whose hinge shook with every sleeper. Nearly all the usable data came from the properly mounted second day.
02 · Recording

Getting honest footage off a moving trolley

All cameras plug into one laptop running ROS 2 Jazzy, and every stream is written into a time-stamped MCAP bag. The principle was to record with exactly the sensors that would later run the detectors, so the models never have to transfer between cameras.

  • My own camera node. The standard ROS USB camera driver recorded green, corrupted frames in MJPEG mode. I replaced it with a node that opens each camera through OpenCV and publishes the camera's own JPEG bytes untouched. It costs almost no CPU, and training data and live input share the same compression.
  • Data volume. A 59-second test with four cameras produced 9.7 GiB, every camera near 30 Hz. One field run was about 12 GB.
  • USB limits. The laptop's USB bus could carry only three cameras at once, so the tools adapt to whichever cameras are connected.
  • Operator markers. A grid viewer shows all streams with buttons (track, sleeper, switch, fastener, bolt). A press stores the capture timestamp, pointing to defect frames later.
03 · Dataset

From 30 Hz video to frames worth labelling

At 30 Hz, neighbouring frames are almost identical, and the trolley often stood still. Labelling them all would waste effort and inflate the scores.

  • Sampling by distance, not time. For the top-down camera, forward motion is a pure shift of the image. I measure that shift with phase correlation on a small greyscale preview and keep a frame once the view has moved 40% of its height. Result: two to three frames per sleeper, none while stopped.
  • The forward camera needed something else. In a forward view the motion is closer to a zoom, so no single shift exists; the same method kept 21 frames across 13 runs. That camera is sampled once per second instead.
  • Rescuing cut-off recordings. A flaky USB hub left some bags without their index. A streaming fallback reader recovered 1,789 frames from one run that would otherwise have been lost.
  • Split by track, not by frame. A random split puts a frame in training and its neighbour, the same sleeper one second later, in test. That measures memory. Whole tracks are held out instead.
TrackRoleWhy
Track 1TrainAll the handheld first-day footage
Track 4TrainMost mounted runs
Track 3ValidationHeld out
Track 2TestHeld out
A labelling tool with boxes drawn around fasteners and bolts on a top-down rail image
Labelling in X-AnyLabeling, following a written guide so that boxes stay consistent between people.
Bar chart of labelled boxes per class: 249 bolts, 93 fasteners, 47 wooden sleepers, 6 clips, 3 concrete sleepers, 1 empty bolt hole
Round one: 249 bolts but a single empty bolt hole. The imbalance shapes everything that follows.
04 · Models

What the detectors learned, and what they could not

Top-down model. Trained on 46 labelled frames (399 boxes). Nano and small variants behaved the same, so I kept the nano: equally good, and kinder to the hardware it had to run on.

ClassInstancesPrecisionRecallmAP50
sleeper_wood121.0000.9560.995
fastener210.9281.0000.993
bolt620.9680.9750.970
All950.9650.9770.986

These numbers come from 11 frames of one held-out track on one day. They say the common components are solid. They say nothing about the rare classes, which do not even occur in the validation frames.

Forward model. YOLO26s at 960 pixels, trained on 55 frames. It reaches mAP50 0.49 over all classes: sleepers are found well, but damage and displaced clips have too few examples to learn, and the switch score is not meaningful because every validation switch also appears in training.

People. The COCO-pretrained nano model needed no training at all. People are among the best-represented classes in COCO.

Training curves of the forward model: losses falling, validation mAP50 levelling off near 0.5
Forward model training curves. Losses fall steadily; validation mAP50 levels off around 0.5, held back by the rare classes and the tiny validation set.
Top-down frame where the left baseplate has fewer detected bolts than the right one
Flagged by the rule: the two plates of this sleeper disagree on their bolt count.
05 · Mining

Using the model to find its own training data

I started with exactly one example of a missing bolt. To find more, I ran the top-down model over all 1,529 frames and let a rule flag suspicious ones for review. Each rule I tried exposed something about the data:

RuleFlaggedWhat happened
Fastener with fewer than 2 bolts0The known defect plate has four holes, so it still counted enough
Fewer than 2, or an odd count550The most common plate holds 3 bolts, so normal plates were flagged
The two plates of a sleeper differ95Works for any plate type: 11 confirmed missing-bolt frames

One example became twelve. The winning rule was the one that made no assumption about the type of plate, and the 11 confirmed frames now act as a regression test for later versions.

06 · Live

Running it on the trolley

A generic ROS 2 detector node runs once per camera, configured by parameters: model, topic, input size, rate, confidence threshold, class filter.

  • Never fall behind. The subscription only stores the newest message, like a mailbox. A timer picks it up, runs the detector and publishes. If inference is slower than the camera, frames are skipped, never queued.
  • Separate processes. Detection and the viewer run apart, so a viewer crash does not stop detection.
  • CPU budget. Exported to OpenVINO, the top-down model takes 63 ms per image on an i5 laptop (about 16 fps); on a GPU it takes 6 ms.

At the two-day field test in October 2026, all three detectors ran together while the trolley was pushed along the track. Sleepers, fasteners and bolts were detected reliably, also at higher pushing speed, on a track whose images had been held out from training. The 360° model picked up people near the trolley, and the forward model boxed the one known displaced clip. It also predicted switches too eagerly, and the 4K stream got jittery above walking pace because the cameras share USB bandwidth.

Laptop on the trolley showing live detections in the top-down and forward camera views
The live grid on the trolley, 29 October 2026.
Forward camera view with sleepers boxed and one fastening labelled clip_displaced 0.85
The forward model flags the known displaced clip, despite having seen only a few examples.
07 · Limits

What it cannot do yet

  • Rare defects. Empty holes, clips, sleeper damage and displaced clips have too few examples. The rules are the main defence, and they inherit every mistake of the component detector. A second, targeted labelling round is the next step.
  • A small test. Validation and test are one track each. The held-out test track still has to be scored.
  • Rules are offline. The missing-bolt rule needs persistence over several frames before it can raise a live alarm.
  • What a camera cannot see. A loose bolt can look identical to a tight one from above.
  • A fixed viewpoint. The models are reliable because the camera mount is fixed. On footage from a different camera the forward model only partly transferred and the top-down model did not transfer at all. A new viewpoint needs its own data.
Forward model on footage from a different camera, finding some sleepers with low confidence
Unseen camera: the forward model partly transfers.
Top-down model on footage from a different camera, boxing the whole scene as one sleeper
The top-down model does not: it only knows a view from 40 cm up.