Reflection-removing Camera

In recent years, computer scientists have been investigating a range of techniques for removing reflections from digital photographs shot through glass. Some have tried to use variability in focal distance or the polarization of light; others, like those at MIT, have exploited the fact that a pane of glass produces not one but two reflections, slightly offset from each other.

At the Institute of Electrical and Electronics Engineers’ International Conference on Acoustics, Speech, and Signal Processing this week, members of the MIT Media Lab’s Camera Culture Group will present a fundamentally different approach to image separation. Their system fires light into a scene and gauges the differences between the arrival times of light reflected by nearby objects — such as panes of glass — and more distant objects.

In earlier projects, the Camera Culture Group has measured the arrival times of reflected light by using an ultrafast sensor called a streak camera. But the new system uses a cheap, off-the-shelf depth sensor of the type found in video game systems.

At first glance, such commercial devices would appear to be too slow to make the fine discriminations that reflection removal requires. But the MIT researchers get around that limitation with clever signal processing. Consequently, the work could also have implications for noninvasive imaging technologies such as ultrasound and terahertz imaging.

“You physically cannot make a camera that picks out multiple reflections,” says Ayush Bhandari, a PhD student in the MIT Media Lab and first author on the new paper. “That would mean that you take time slices so fast that [the camera] actually starts to operate at the speed of light, which is technically impossible. So what’s the trick? We use the Fourier transform.”

The Fourier transform, which is ubiquitous in signal processing, is a method for decomposing a signal into its constituent frequencies. If fluctuations in the intensity of the light striking a sensor, or in the voltage of an audio signal, can be represented as an erratic up-and-down squiggle, the Fourier transform redescribes them as the sum of multiple, very regular squiggles, or pure frequencies.

Phased out

Each frequency in a Fourier decomposition is characterized by two properties. One is its amplitude, or how high the crests of its waves are. This describes how much it contributes to the composite signal.

The other property is phase, which describes the offset of the wave’s troughs and crests. Two nearby frequencies may be superimposed, for instance, so that their first crests are aligned; alternatively, they might align so that the first crest of one corresponds with a trough of the other. With multiple frequencies, differences in phase alignment can yield very different composite signals.

If two light signals — one reflected from a nearby object such as a window and one from a more distant object — arrive at a light sensor at slightly different times, their Fourier decompositions will have different phases. So measuring phase provides a de facto method for measuring the signals’ time of arrival.

There’s one problem: A conventional light sensor can’t measure phase. It only measures intensity, or the energy of the light particles striking it. And in other settings, such as terahertz imaging, measuring phase as well as intensity can dramatically increase costs.

So Bhandari and his colleagues — his advisor, Ramesh Raskar, the NEC Career Development Associate Professor of Media Arts and Sciences; Aurélien Bourquard, a postdoc in MIT’s Research Laboratory of Electronics; and Shahram Izadi of Microsoft Research — instead made a few targeted measurements that allowed them to reconstruct phase information.

In collaboration with Microsoft Research, the researchers developed a special camera that emits light only of specific frequencies and gauges the intensity of the reflections. That information, coupled with knowledge of the number of different reflectors positioned between the camera and the scene of interest, enables the researchers’ algorithms to deduce the phase of the returning light and separate out signals from different depths.

Page 1 of 2

• Next Page >