2025-12-07

Futuristic Autonomous Tram Technology in Smart Cities

AIComputer VisionRoboticsVisual SLAMORB-SLAM3ByteTrackCubeSLAMDocker

Finally wrapped up my Master's capstone project, and honestly, it's been quite a ride. For the past several months, I've been deep in the world of Visual SLAM from zero, trying to figure out how we can mimic autonomous vehicles actually work in busy urban environments. It turned out to be a lot tougher than I expected.

The core question I wanted to tackle was pretty straightforward. How do we help a vehicle "see" and understand dynamic urban scenes, cars cutting across, cyclists appearing out of nowhere, pedestrians doing unpredictable things, while simultaneously building a map and knowing where it is?

Where the architecture started

Before any code, the whole system lived on grid paper. These two sketches capture how the idea evolved from a rough ORB-SLAM plus detector sketch into a concrete plan for fusing sparse point landmarks with 3D object cubes in one map.

Early handwritten sketch: ORB-SLAM mapping and localisation stages, an object detector producing 2D bounding boxes, and axis-aligned 3D boxes in world coordinates. — First pass — ORB-SLAM for mapping and localisation, an object detector for {b₁, b₂, …, bₙ}, and lifting detections into axis-aligned 3D bounding boxes.

Handwritten architecture diagram: video frames through ORB-SLAM to camera pose T_cw and a point-cloud map, with CubeSLAM adding 3D object landmarks defined by centre, size, and yaw. — Refined sketch — frames Iₜ through ORB-SLAM yield pose T_cw and a point-cloud map; CubeSLAM adds object landmarks as 3D cubes (centre, size, yaw) alongside sparse points.

So what did I actually build?

I put together a full pipeline that combines:

✅ YOLOv8 for detecting objects in real-time
✅ ByteTrack to keep consistent IDs on those objects across frames (no more treating the same car as 10 different cars)
✅ CubeSLAM-style 3D bounding box reconstruction
✅ All of this integrated into ORB-SLAM3 running in monocular mode

The whole thing runs on my M1 MacBook Pro with 16GB RAM inside a Docker container. Yes, I had to containerise everything because the Pangolin visualisation library often fails on macOS.

What's in the video?

There are 2 parts in the video. First one (00:00 - 00:24) shows the system running on KITTI odometry sequence 08 with ByteTrack doing its thing, you can see how objects maintain their IDs as the camera moves through the scene. Second part (00:24 - 00:59) is what I'm most excited about: real-world testing on Adelaide suburban streets. Same pipeline, completely different environment.

Visual SLAM pipeline — KITTI sequence 08 (00:00–00:24) and Adelaide suburban streets (00:24–00:59).

What's actually new here?

Most existing work either focuses purely on feature-based or direct approaches, and typically evaluates on a single dataset. I trained on BDD100K and tested on KITTI to see how well the model generalises across different driving scenarios. That cross-dataset evaluation combined with persistent 3D object tracking is something I haven't seen done quite this way before.

But let's be real about the limitations.

Monocular vision is hard. My APE sits around 61-64 metres mean error, which tells you everything about the inherent scale ambiguity problem. And the detector still makes some FPs, in the Adelaide portion, there's a moment where traffic signs were misclassified as cyclists. These edge cases remind me there's still plenty of work ahead.

What's next?

For future work, I'm looking at incorporating semantic segmentation and possibly RANSAC filtering to clean things up. Maybe even some reinforcement learning down the line. I also want to leverage SOTA object detector model such as YOLO26 (released in September 2025) to increase the robustness of the system.

None of this would have been possible without the guidance of my supervisor, Dr Mehdi Hosseinzadeh. I deeply appreciate him for his indispensable guidance and support throughout this research journey.