A robot vacuum that remembers your floor plan, a phone that measures a room, a warehouse robot that weaves around pallets: they lean on some mix of the same three building blocks. A computer vision camera sees the scene, visual SLAM (vSLAM) works out where the device is and builds a map, and a time-of-flight (ToF) sensor measures how far away everything is. This guide explains each one in detail, how they fit together, and where you’ll find them today.
1. The computer vision camera: turning pixels into understanding
An ordinary camera records a flat, 2D grid of colour values. A computer vision camera pairs that sensor with software, often running on a dedicated AI chip, that interprets the image: detecting objects, reading tags, tracking people or finding the edges of a path. On its own, though, a single 2D camera has a blind spot. It sees what is in front of it but not reliably how far away it is.
That is why vision systems usually come in three flavours:
- Monocular: one camera. Cheap and light, but distance and scale must be inferred from motion or learned cues.
- Stereo: two cameras a fixed distance apart, which estimate depth by comparing the two views, much like human eyes.
- RGB-D: a colour camera paired with a depth sensor, such as ToF, so every pixel comes with a distance value.
All three feed the same downstream tasks, and the most important of those for anything that moves is vSLAM.
2. vSLAM: knowing where you are while drawing the map
SLAM stands for simultaneous localization and mapping, the chicken-and-egg problem of building a map of an unknown place while tracking your own position within it. Visual SLAM solves it mainly with cameras.
Why it’s needed. A robot can estimate how far it has moved from wheel ticks, an IMU or by tracking visual features frame to frame (visual odometry). NVIDIA’s Isaac ROS documentation explains the catch: odometry “inevitably begins to drift” because small errors accumulate. vSLAM fixes this by building a graph of distinctive visual landmarks so the robot can recognise a place it has already visited, “closing the loop” to correct the drift (NVIDIA Isaac ROS).
How a typical feature-based vSLAM pipeline works:
- Feature extraction: find distinctive points in each frame, such as corners and textured spots.
- Tracking: match those features between frames to estimate how the camera moved.
- Mapping: triangulate matched features into 3D map points and save important frames as keyframes.
- Optimisation: refine camera poses and map points together (bundle adjustment) to reduce error.
- Loop closure and relocalisation: recognise previously seen places, snap the map back into alignment, and recover if tracking is lost.
A landmark example: ORB-SLAM3. This open-source library, published in IEEE Transactions on Robotics, runs visual, visual-inertial and multi-map SLAM with monocular, stereo and RGB-D cameras. Its stereo-inertial mode averaged 3.6 cm accuracy on the EuRoC drone dataset and 9 mm during quick hand-held motion on TUM-VI, a setting representative of AR and VR use (ORB-SLAM3, arXiv).
3. Time-of-flight: measuring distance with light
A ToF sensor emits light, usually a near-infrared laser or LED, and measures how long it takes to bounce back. Because light travels at a known speed, the round-trip time gives the distance: half the time multiplied by the speed of light. Unlike a 2D camera, the result is depth, the “Z” dimension, for each pixel (Sony Semiconductor Solutions).
Sony describes two main types:
- Direct ToF (dToF): times each light pulse directly. Sensors often use single-photon avalanche diode (SPAD) pixels, which amplify a single incoming photon into a measurable signal, so they can detect faint returns and reach longer distances. Sony’s IMX560 is an example.
- Indirect ToF (iToF): sends continuously modulated light and measures the phase shift of the returning light, accumulated as charge in each pixel. It can use conventional CMOS sensor structures and excels at high-resolution, near- to mid-range depth. Sony’s IMX556 and IMX570 work this way.
A real iToF example: Azure Kinect. Microsoft’s Azure Kinect DK depth camera uses amplitude-modulated continuous-wave ToF with a 1-megapixel chip and pixels just 3.5 µm across. It produces a depth map in millimetres for every pixel and runs at up to 30 frames per second in most modes (Microsoft Learn).
A real dToF example: Apple’s LiDAR Scanner. When Apple added a LiDAR Scanner to the 2020 iPad Pro, it said the sensor measures distance to objects up to 5 metres away, indoors and out, operating “at the photon level at nano-second speeds” (Apple Newsroom).
ToF’s strengths and quirks
ToF gives instant, dense depth without needing texture in the scene, which is exactly where camera-only vSLAM struggles, such as on blank walls. It also works in the dark because it brings its own light. But it has limits. Microsoft documents that depth pixels can be invalidated when the infrared signal is too weak or saturated, or by multi-path interference, when light bounces off two surfaces such as a room corner before returning (Microsoft Learn). Strong sunlight, very dark or shiny surfaces and long distances can also degrade results, which is one reason Sony positions long-range SPAD dToF for outdoor use and iToF for close, detailed work.
Putting it together: vision camera + ToF + vSLAM
Combined, the three cover each other’s weaknesses. The colour camera supplies rich texture for recognising objects and landmarks. The ToF sensor supplies true metric depth, so the map has correct scale from the first frame. vSLAM fuses both, often with an IMU, into a continuously corrected position and 3D map. ORB-SLAM3’s RGB-D mode and NVIDIA’s cuVSLAM, which offers tutorials for RealSense cameras in RGB-D mode, are examples of this approach (NVIDIA Isaac ROS).
Where you’ll see it:
- Augmented reality: Apple says iPadOS combines LiDAR depth points with camera and motion-sensor data so AR objects appear instantly and sit correctly in the room.
- Logistics and factories: Sony lists ToF uses including measuring pallets and cargo, collision detection for AGVs and AMRs, robotic picking and detecting people entering hazardous areas.
- Agriculture: 3D picking for fruit harvesting and monitoring livestock.
- Privacy-aware monitoring: depth-only sensing can detect falls in care homes without recording colour video, according to Sony.
- Drones and mobile robots: visual-inertial SLAM like ORB-SLAM3 is benchmarked on drone flight data.
Key takeaways
- A computer vision camera tells a machine what it sees. Monocular, stereo and RGB-D setups differ in how they get depth.
- vSLAM tells it where it is, tracking visual features, building a map and using loop closure to cancel drift.
- ToF tells it how far, timing reflected light directly (dToF) or by phase shift (iToF).
- Together they power AR on phones, warehouse robots and home robots, and each covers the others’ blind spots.
Conclusion
Seeing, locating and measuring used to need bulky, expensive hardware. Today a phone-sized module can do all three in real time. As depth sensors shrink and vSLAM software matures, the gap between “a camera” and “a robot that understands its surroundings” keeps getting smaller.