CongCar: From “It Runs” to Stable High-Speed Driving

CongCar is not an offline demo that displays an inference result once. Its goal is to run image capture, semantic segmentation, detection, planning, and vehicle control continuously on an RK3588.

Once the vehicle moves, average FPS stops explaining the important problems. Old frames may remain in a queue, scene states may oscillate, path following may be duplicated across branches, speed commands may jump, and an abnormal condition may need to stop the car safely. The work therefore shortened and instrumented the path from observing the course to sending a control command.

CongCar finished seventh in the North China division of the 21st National Undergraduate Smart Car Competition. The competition is over, so this is both a development record and a project review that starts from the real course result and asks which design choices affected vehicle behavior.

Scope and contribution

CongCar is a team vehicle system. The repository contains segmentation, detection, scene handling, and control work by several members. This review records the parts I followed and helped improve; it does not claim that every module was implemented alone.

My recent focus was pedestrian-scene state management, the effect of vehicle-center offset on risk decisions, and integration with a shared path follower. Multithreading, NPU inference, branching, and other scenes are summarized from the team implementation and commit history.

Separating real-time tasks

The early pipeline kept inference, control, and display close together. A slow operation could stall the whole vehicle. The later design separated image capture, segmentation, detection, control, and display/HTTP streaming. Shared results expose the newest state, and full queues discard old frames.

For a recorder, processing every frame may matter. For a moving vehicle, slightly incomplete but current information is more useful than complete information that arrives late. Real-time optimization is therefore about the age of the data used by control, not merely smooth display.

Segmentation and detection were assigned to different NPU cores, and multiple segmentation engines were tried. Parallelism is not automatically faster: memory transfer, runtime scheduling, and the consumer’s speed can remain bottlenecks. At one stage, returning segmentation to a single core produced a more reproducible 1.0 m/s run. Whole-vehicle repeatability mattered more than a local inference number.

Measure freshness, not just FPS

System evaluation needs at least the camera’s frame-production rate, the rate at which segmentation and detection update, the age of the result when control reads it, and the frequency of UART commands. The number of threads alone cannot answer whether the control path is genuinely real-time.

A field dashboard from an ordinary line-following scene. It keeps perception, the bird’s-eye path, speed, and model-performance information visible together. Model FPS describes only local computation; it does not replace camera input rate or end-to-end control freshness.

A shared PathFollower

As normal driving, vehicle avoidance, pedestrian handling, and branching accumulated, each scene having its own centerline extraction and steering logic became difficult to maintain. A common PathFollower unified the bird’s-eye path, centerline, look-ahead point, and steering control. Scenes now provide constraints such as masks, branch choice, and stopping conditions.

The benefit is behavioral as well as architectural. A shared control state reduces abrupt steering when a scene changes. Debouncing and shared buffers prevent a one-frame fluctuation from reversing an action. The cost is regression risk: one change can affect normal driving, vehicle scenes, pedestrians, and branches, so coordinate, speed, and steering conventions must be checked in each case.

Branches, revert, and redesign

The first branch implementation deleted one side directly from the segmentation mask. It was later reverted because mutating shared output also changed what display and other scenes saw, while fixed pixel widths were unstable in perspective images. The later design moved branch handling into bird’s-eye coordinates and made the retained side a configuration value.

The important lesson is not that a revert is pure failure. Reverting an abstraction that fails on the real course can return the project to a state that is easier to control and redesign.

What speed exposed

Speed was raised from 0.7 to 1.0 and then 1.5 m/s. The change reduced the time available for correction: the same latency covered a longer physical distance. Small low-speed oscillations became possible line crossings, and a one-frame false detection could cause sharp steering or an unnecessary stop.

The system moved curve speed factors into configuration, added speed decay when markers or paths were temporarily lost, buffered acceleration, widened the bird’s-eye view, and moved model, class, color, and scene mappings into JSON. These choices shortened the experiment loop: change a value, run, observe, and save a reproducible setting.

Increasing PWM because a low-speed run finished is not evidence of stable high-speed driving. Perception latency, steering change, curve speed, and failure behavior must be checked again at each speed.

Pedestrians: detection is not risk

“Stop whenever a person is detected” was too simple. Faraway people, people outside the planned path, and brief false detections could cause unnecessary stops, while a single-frame position could react too slowly to real danger.

The improved scene keeps context: whether the person enters the planned path, target distance and risk area, persistence and disappearance time, the conditions for resuming, and the offset between image center and vehicle center. Camera placement, cropping, and bird’s-eye transformation can make image center differ from the vehicle’s actual path, so the offset must be configurable.

The stop condition is intentionally more sensitive than the resume condition. A temporary missed detection should not immediately clear risk, and an object outside the path should not carry the same weight. Unnecessary stopping costs time; a wrong resume can cause a collision.

Engineering conclusion and limits

The history includes new features, refactors, temporary disabling, reverts, and reimplementation. A useful improvement must return to vehicle behavior: is the result new enough, can the same configuration reproduce it, are scene transitions continuous, do false detections trigger dangerous actions, and is the safety boundary still clear at higher speed?

The current system forms a complete loop, but “can finish” is not the same as “can prove stable.” The next useful work is end-to-end timestamping, replayable failure cases, explicit scene state diagrams, speed-binned course metrics, and a clear boundary between software stop, active braking, and hardware emergency stop.

CongCar repository