Type to search

Tags: ,

Researchers Develop Optical Microphone That Can See Sound, Distinguish Instruments


A research team at Carnegie Mellon University has recently unveiled an incredible optical microphone capable of capturing and isolating sounds through a dual-shutter, optical vibration-sensing system.

This imaging device is being developed by a team of research associates, professors, and students at the School of Computer Science’s Robotics Institute (RI).

“We’ve invented a new way to see sound. It’s a new type of camera system, a new imaging device, that is able to see something invisible to the naked eye,” said Mark Sheinin, a member of the research group and an Illumination and Imaging Laboratory (ILIM) post-doctoral research associate.

This unique system uses two cameras and a laser to capture surface vibrations, re-constructing sound waves, and distinguish multiple instruments from afar.

The idea behind this rests on the very concept of how sound is produced. Sound is simply a series of traveling pressure waves created from vibrating objects or sound sources. By capturing these vibrations from the surface of a sound source, the optical microphone is able to create an image of the specific sound.

But even high-end microphones are unable to distinguish different sound images from one another, let alone isolate them or eliminate them completely. This novel technology employs a unique way of capturing individual sounds through a laser that creates a precise speckle pattern in symphony with audio vibrations.

How the Optical Microphone Works

Using two pairs of cameras and a laser, this dual-shutter imaging device is able to capture and reconstruct sound waves from not just one, but multiple sound sources playing at the same time — even if they were producing two completely different sounds.

The laser captures and creates a precise speckle pattern that distorts as the surface vibrates. The two cameras, which operate on two different shutters (global and rolling), document this pattern, which can then be sent to a software algorithm that analyzes all the footage to reconstruct the audio signal.

The cameras use a 63-fps frame rate, which may seem a little slow. However, the use of both a global shutter and a rolling shutter allows the device to capture two different aspects of the sound vibrations, one that can read an entire image sensor and another that can read thousands of horizontal lines in a single frame. Combined, this lets the device read up to 63,000 Hz of sound.

To truly test the device’s capabilities, the optical microphone was used to capture vibrations coming from multiple guitars playing a duet, two speakers playing a different song, on tuning forks, and wildly enough, a bag of Doritos chips placed in front of a speaker.

The Importance of Sound Imaging Technology

Optical microphones certainly aren’t anything new. In fact, the research project was partly a tribute to the work of researchers at the Manhattan Institute of Technology (MIT), who originated the first ever visual microphone back in 2014.

However, this novel imaging device vastly improves on its predecessors in both technique and algorithm.

“This system pushes the boundary of what can be done with computer vision,” said RI assistant professor Matthew O’Toole of their project. “This is a new mechanism to capture high speed and tiny vibrations, and presents a new area of research.”

The invention will allow sound engineers to fine-tune ensemble mixes, and manufacturers to closely monitor the health of their machines over on the factory floor.

“If your car starts to make a weird sound, you know it is time to have it looked at,” adds Mark Sheinin. “Now imagine a factory floor full of machines. Our system allows you to monitor the health of each one by sensing their vibrations with a single stationary camera.” 

Watch: Dual-Shutter Microphone

Naomi Feller

Originally from the East Coast, Naomi started singing as young as 3 years old. In her early teens Naomi made some embarrassing YouTube videos before settling on a love for Podcast editing. When she's not pouring over endless amounts of audio, she lends her expertise to us here at Shout4Music with her crystal clear and finely tuned microphone reviews.

  • 1