photo: vimeo.com

After growing up to adulthood blinded from birth, a man now has taken a peculiar hobby: photography. Were it not for the efforts of a group of researchers who have devised a system that converts images into sequences of sound, this new found pastime had been impossible. Hobbies or not, the technology is particular impressive and judging from the stream of data reported thus far, it could prove to be a marvelous system for everyday use, helping the blind navigate their surroundings, recognize people and even appreciate visual arts — all through sound.

In all began in 1992 when a Dutch engineer called  Peter Meijer invented vOICe – an algorithm that converted simple grayscale, low-resolution images into sounds that would break into an unique, discernible pattern by the trained ear. As the algorithm scans from left to right, each pixel or group of pixels has a corresponding frequency (higher positions in the image  –> higher acoustic frequencies). A simple image, for instance, only showing  a diagonal line stretching upward from left to right becomes a series of ascending musical notes, while a more complicate image, say a man leaning on a chair, turns into a veritable screeching spectacle.

Amir Amedi and his colleagues at the Hebrew University of Jerusalem took things further and made vOICe portable, while also studying the participants’ brain activity for clues. They recruited people that had been blind all their lives from birth, but after just 70 hours of training and obviously despite any visual cues,   the individuals went from “hearing” simple dots and lines to “seeing” whole images such as faces and street corners composed of 4500 pixels. Mario on Nintendo only has 192 pixels and it still felt freaking realistic sometimes (was that just me as kid or what?).

Seeing with sound

Using head-mounted cameras that communicated with the vOICe technology, the blind participants could then navigate their surroundings and even recognize human silhouettes. To prove they could visually sense accurately, the  participants mimicked the silhouette’s stances.

Things turned really interesting when the researchers analyzed the brain activity data. The traditional sensory-organized brain model says the brain is organized in regions each devoted to certain senses. For instance, the visual cortex is used for sight processing; in the blind, where these areas aren’t used conventionally, these brain regions are re-purposed to boost some other sense, like hearing. Amedi and colleagues found, however,  that the area of the visual cortex responsible for recognizing body shapes in sighted people was signaling powerfully when the blind participants were  interpreting the human silhouettes. Neuroscientist Ella Striem-Amit of Harvard University, who co-authored the paper, thinks it’s time for a new model. “The brain, it turns out, is a task machine, not a sensory machine,” she says. “You get areas that process body shapes with whatever input you give them—the visual cortex doesn’t just process visual information.”

“The idea that the organization of blind people’s brains is a direct analog to the organization of sighted people’s brains is an extreme one—it has an elegance you rarely actually see in practice,” says Ione Fine, a neuroscientist at the University of Washington, Seattle, who was not involved in the study. “If this hypothesis is true, and this is strong evidence that it is, it means we have a deep insight into the brain.” In an alternative task-oriented brain model, parts of the brain responsible for similar tasks—such as speech, reading, and language—would be closely linked together.

The team also devised a vOICe version that can be run as a free iPhone app, called EyeMusic. The researchers demonstrated that using the app, blind participants could recognize drawn faces and distinguish colours. The video below showcases the app. The study was reported in the journal Current Biology.

source: scimag