How RGB Tactile Sensors Turn Color Into Depth
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
An RGB visual-tactile sensor does not read depth directly from color. The usual pipeline is:
RGB image -> surface normal -> depth gradient -> depth map
Red, green, and blue illumination are arranged so that each color channel carries information from a different lighting direction. Under a photometric-stereo model, the three channel intensities at a pixel constrain the local surface normal. Differential geometry then converts this normal into local slopes. Finally, Poisson reconstruction or a least-squares integration method turns the slope field into a globally consistent depth map.
In real sensors, the ideal equations are only the clean skeleton. Calibration carries much of the practical burden: it learns how RGB values map to normals or gradients after accounting for LED placement, elastomer scattering, camera response, color crosstalk, and non-uniform illumination.
From Color to Shape
A typical vision-based tactile sensor has a deformable elastomer surface, internal colored illumination, and a camera looking at the surface from inside the sensor. When an object presses on the elastomer, the surface bends. The camera observes the resulting color and shading pattern. The core idea is that local surface orientation changes how strongly each colored light contributes to the observed pixel.
In a simplified RGB photometric-stereo setup, red, green, and blue lights come from known directions:
l_R = (l_Rx, l_Ry, l_Rz)
l_G = (l_Gx, l_Gy, l_Gz)
l_B = (l_Bx, l_By, l_Bz)
Each \(l\) is a 3D direction vector, so the three lighting directions together contain nine scalar values. For one pixel, the camera measures:
I_R, I_G, I_B
Under the simplest reflectance model, these intensities satisfy:
\[ \begin{bmatrix} I_R \
I_G \
I_B \end{bmatrix} = \begin{bmatrix} l_{Rx} & l_{Ry} & l_{Rz} \
l_{Gx} & l_{Gy} & l_{Gz} \
l_{Bx} & l_{By} & l_{Bz} \end{bmatrix} \rho \begin{bmatrix} n_x \
n_y \
n_z \end{bmatrix}. \]
Here \(n=(n_x,n_y,n_z)\) is the unit surface normal, and \(\rho\) is an effective reflectance or gain. Solving this small linear system gives \(g=\rho n\). Normalizing \(g\) gives the normal direction:
\[ n = \frac{g}{|g|}, \qquad \rho = |g|. \]
This is the photometric-stereo part: different lights produce different brightnesses, and the brightness triplet lets us infer which way the local surface is facing.
Are the Lighting Directions Constant?
In the ideal textbook model, \(l_R\), \(l_G\), and \(l_B\) are constant over the whole image. Red might illuminate from the left, green from the upper right, and blue from below. That approximation is useful because it turns each pixel into the same three-equation linear problem.
Physical tactile sensors are messier. The LEDs are close to the elastomer, so the light direction can vary with image position. The elastomer and coating scatter light. Camera vignetting changes brightness across the image. Color channels leak into each other. The surface may show specular highlights or saturation near contact edges. Because of these effects, the practical “lighting direction” is often better treated as an effective calibrated response, not as a perfectly known global vector.
This leads to two common implementations. A simple system estimates fixed lighting directions and uses the linear model directly. A stronger system builds a calibrated mapping:
(R, G, B, x, y) -> normal or gradient
or, if position dependence is weak:
(R, G, B) -> normal or gradient
The second form absorbs the messy optics into an empirical lookup table, interpolation model, polynomial fit, or neural network.
Lambertian Reflection
The clean equation above comes from Lambertian reflection. A Lambertian surface is an ideal matte surface whose brightness depends on the angle between the surface normal and the light direction. If both vectors are unit length:
\[ I = \rho (n \cdot l) = \rho \cos \theta . \]
When the light points directly at the surface, \(\theta\) is small and the pixel is bright. When the light arrives at a shallow angle, the pixel is darker. The dot product turns geometry into intensity.
For visual-tactile sensing, Lambertian reflection is a useful approximation, not a full physical description. Elastomers involve scattering, coatings, camera nonlinearities, and local contact artifacts. That is why calibration is not a minor cleanup step; it is part of the measurement model.
From Normal to Depth Gradient
Once the normal is known, the next question is how it relates to depth. Let the deformed tactile surface be a height field:
\[ z = z(x,y). \]
The corresponding 3D surface can be parameterized as:
\[ S(x,y) = (x, y, z(x,y)). \]
Its tangent vector in the \(x\) direction is:
\[ S_x = \left(1, 0, \frac{\partial z}{\partial x}\right), \]
and its tangent vector in the \(y\) direction is:
\[ S_y = \left(0, 1, \frac{\partial z}{\partial y}\right). \]
The normal is perpendicular to both tangents, so it is given by their cross product:
\[ S_x \times S_y = \left( -\frac{\partial z}{\partial x}, -\frac{\partial z}{\partial y}, 1 \right). \]
After normalization:
\[ n = \frac{ \left( -\frac{\partial z}{\partial x}, -\frac{\partial z}{\partial y}, 1 \right) }{ \sqrt{ \left(\frac{\partial z}{\partial x}\right)^2 + \left(\frac{\partial z}{\partial y}\right)^2 + 1 } }. \]
This is the differential-geometry step. It says that the local surface orientation and the depth gradient determine each other. If \(n=(n_x,n_y,n_z)\), then:
\[ \frac{\partial z}{\partial x} = -\frac{n_x}{n_z}, \qquad \frac{\partial z}{\partial y} = -\frac{n_y}{n_z}. \]
So the RGB image gives normals, and normals give local slopes.
From Gradient to Depth
At this point we have a gradient field:
\[ p(x,y) = \frac{\partial z}{\partial x}, \qquad q(x,y) = \frac{\partial z}{\partial y}. \]
The sensor still needs the depth field \(z(x,y)\). In one dimension, recovering height from slope is just integration. In two dimensions, direct path integration is fragile: going right then down should give the same result as going down then right, but noisy gradients rarely agree perfectly.
If \(p\) and \(q\) truly come from a smooth depth function, they satisfy the integrability condition:
\[ \frac{\partial p}{\partial y} = \frac{\partial q}{\partial x}. \]
Real measurements violate this condition because normals are noisy and the optical model is imperfect. The usual solution is to find the depth map whose gradients best match the estimated field:
\[ \min_z \iint \left[ \left(\frac{\partial z}{\partial x} - p\right)^2 + \left(\frac{\partial z}{\partial y} - q\right)^2 \right] dxdy. \]
The Euler-Lagrange equation of this least-squares problem is the Poisson equation:
\[ \nabla^2 z = \frac{\partial p}{\partial x} + \frac{\partial q}{\partial y} = \operatorname{div}(p,q). \]
In a pixel grid, this becomes a large sparse linear system. A common discrete form is:
\[ z_{i+1,j}+z_{i-1,j}+z_{i,j+1}+z_{i,j-1}-4z_{i,j} = p_{i,j}-p_{i-1,j}+q_{i,j}-q_{i,j-1}. \]
Solving this system produces a depth map that is globally consistent with the local slopes. Since gradients are unchanged by adding a constant, the reconstructed depth has an arbitrary offset. A sensor usually fixes this using a reference plane, an untouched boundary, a zero-mean constraint, or a known non-contact region.
Why Least Squares Matters
Least squares is not just mathematical convenience. It is the mechanism that lets the system survive imperfect RGB measurements. Some pixels may be dark, saturated, shadowed, specular, or close to a contact discontinuity. Their estimated normals are less trustworthy. A global least-squares reconstruction lets the entire image negotiate these local errors.
Many implementations further use weighted least squares:
\[ \min_z \sum_{i,j} w^x_{i,j} \left(z_{i+1,j}-z_{i,j}-p_{i,j}\right)^2 + w^y_{i,j} \left(z_{i,j+1}-z_{i,j}-q_{i,j}\right)^2. \]
Reliable pixels receive larger weights; saturated or ambiguous pixels receive smaller weights. This turns depth recovery from a brittle integration problem into a global estimation problem.
What Calibration Actually Calibrates
The main calibration target is the mapping from observed color to local surface shape:
RGB -> normal
or directly:
RGB -> gradient
A common calibration procedure presses the sensor with an object of known geometry, such as a sphere. The sphere provides ground-truth normals because its surface geometry is known. For many pixels, calibration records pairs like:
observed RGB <-> known normal
From these samples, the system learns a lookup table or regression model. Calibration also usually includes a background image for the undeformed surface:
I_0(x,y)
The runtime image is normalized by subtracting or dividing by this background, reducing static non-uniformity from illumination, coating thickness, and camera vignetting.
A complete calibration may also estimate pixel-to-millimeter scale, lens distortion, color-channel crosstalk, depth scale, and the zero-depth reference. But the central bridge remains:
color response -> normal or gradient -> depth
Takeaway
The compact way to remember RGB tactile depth reconstruction is:
Photometric stereo estimates normals.
Differential geometry converts normals to gradients.
Poisson or least-squares integration reconstructs depth.
Calibration makes the ideal model usable on real hardware.
The RGB channels are therefore not three direct depth readings. They are three lighting-conditioned observations of the same deformed surface. The depth map emerges only after geometry, optimization, and calibration are put together.