SeeingTag: From PnP Localization to Homography Mapping

SeeingTag is a top-down localization system for a smart-car course. Fixed ArUco markers define the course coordinates; a marker on the vehicle provides position and heading; the result is sent to Unity over UDP.

The system evolved from detection and communication checks, through multi-marker PnP, to a homography matrix. Field conditions—not an abstract ranking of algorithms—broke the assumptions of the first design.

What I worked on

The main visual-localization path, configuration, debugging tools, and later throughput changes are the parts I continuously modified and verified. Unity and the full vehicle belong to the larger team project. SeeingTag provides low-latency, coordinate-consistent position and yaw data.

First close the loop

The first tasks were to detect DICT_4X4_50 markers reliably and send JSON over UDP. Camera, sender, and listener tests included seq and timestamp, making it possible to distinguish a missing visual result from a network loss. Marker false positives were first filtered by image width, a cheap constraint for a fixed-size course at a fixed camera height.

PnP and its hidden premise

The first complete system used two PnP solves: fixed marker world corners to estimate camera pose, and a vehicle marker pose transformed into world coordinates. It could output 3D position and yaw, but depended on trustworthy camera intrinsics and distortion. Without a formal calibration, field-of-view estimates made the numbers continuous but not necessarily geometrically correct.

The lesson was that a returned number is not proof of a valid model. Later checks included course bounds, fixed-point mappings, and heading consistency.

Three field problems

An RTSP phone stream covered the course but accumulated 5–30 seconds of latency. A USB camera had low latency but saw only one or two fixed markers. A wide-angle view covered more but introduced barrel distortion that weakened PnP’s assumptions. Continuing with PnP required low-latency capture, calibration, wide coverage, and stable marker visibility at once.

The course task really needed 2D ground position and heading, not a complete 3D camera pose. That difference suggested a homography.

Homography for the actual task

The new path builds a planar mapping from image corners of fixed markers to known course coordinates, then maps the vehicle-marker center into the course plane:

fixed marker corners + known course coordinates
                  ↓
          findHomography + RANSAC
                  ↓
vehicle marker pixels → perspectiveTransform → course coordinates

Heading comes from mapping two directional corners to the course plane and computing the resulting vector. The main path no longer depends on camera intrinsics. The cost is explicit: reference points must be approximately coplanar, four fixed markers are needed for initialization, and meaningful height or pitch is no longer available.

How to judge the redesign

Evaluation is split into capture, detection, geometry, and output. Capture checks real frame rate and new frames; detection checks fixed and vehicle marker visibility; geometry checks reprojection, course bounds, and heading; output checks UDP frequency, data age, and Unity response. This makes vague reports such as “the image is laggy” traceable to a link.

Stability is not the smallest filter coefficient

Marker-level and output-level filters separate noise sources, but an overly small output coefficient makes the vehicle heading visibly lag in Unity. The objective changed from the smoothest still image to the lowest control and display latency at acceptable jitter. Vehicle turns, acceleration, and short occlusion are required tests; a stationary car encourages filters that are too slow.

Throughput, loss of lock, and observability

The capture thread now keeps reading while the processing thread takes only the newest frame. Display refresh and localization sending are decoupled, and camera, processing, and sending rates are recorded separately. When the vehicle marker disappears briefly, the system does not extrapolate arbitrary history forever. Blind-area mode is manually armed and limited by time, distance, and heading change; the interface shows OFF / ARMED / DRIVING.

The main lesson is not that homography is universally better than PnP. PnP solves a more complete 3D problem; homography discards unused information to obtain a shorter dependency chain and more direct debugging for this course.

Remaining issues

Severe marker occlusion leaves no independent redundancy in single-frame vision. Mapping accuracy depends on fixed-marker coordinates and a planar ground assumption. Blind-area extrapolation handles only short losses and cannot replace vehicle odometry. A systematic trajectory with ground truth is still needed to quantify position and yaw error.

The next useful step would be short-term fusion with wheel speed or an IMU, plus a test trajectory with ground-truth samples. That would move the project from “looks steadier” to repeatable error comparison.