A new study shows how neural networks can reconstruct sounds from brain signals
Researchers from the University of California, San Francisco have developed a novel method to decode sounds from brain activity using artificial neural networks. The team was able to reconstruct a Pink Floyd song that was played to participants while they were inside a functional magnetic resonance imaging (fMRI) scanner. The study, published in the journal Nature Communications, demonstrates the potential of using machine learning to study how the brain processes complex auditory stimuli.
How the brain encodes sound
The human brain has a remarkable ability to process sounds, such as speech and music, and extract meaningful information from them. However, the neural mechanisms underlying this ability are not fully understood. Previous studies have used fMRI to measure brain activity while participants listened to different sounds, but these methods could only identify which brain regions were involved, not how they encoded the specific features of the sounds.
To overcome this limitation, the researchers used a technique called voxel-wise modeling, which analyzes the activity of individual voxels (small units of brain tissue) in response to different sounds. The researchers trained artificial neural networks to learn the relationship between the sound features and the voxel activity, and then used these networks to predict the sound features from new voxel activity data.
Reconstructing a Pink Floyd song
The researchers tested their method on 10 participants who listened to various natural and synthetic sounds, as well as a Pink Floyd song called “Have a Cigar”. The song was chosen because it has a complex structure and a variety of instruments and vocals. The researchers recorded the participants’ brain activity using fMRI while they listened to the song, and then fed the data to the neural networks. The networks were able to reconstruct the song from the brain activity with high accuracy, preserving the pitch, timbre, and rhythm of the original song.
The researchers also compared their method with two other methods that use different types of neural networks: one that uses convolutional neural networks (CNNs), which are good at processing images, and one that uses recurrent neural networks (RNNs), which are good at processing sequences. They found that their method outperformed both CNNs and RNNs in reconstructing the song, suggesting that voxel-wise modeling is more suitable for decoding complex auditory stimuli.
Implications and future directions
The study shows that voxel-wise modeling can be used to decode sounds from brain activity with high fidelity, and that artificial neural networks can learn how the brain encodes sound features. This could have implications for understanding how the brain processes speech and music, as well as for developing brain-computer interfaces that can translate thoughts into sounds. For example, such interfaces could help people who have lost their ability to speak or hear due to injury or disease.
The researchers plan to extend their method to other types of sounds, such as speech and environmental sounds, and to other modalities, such as vision and touch. They also hope to improve their method by incorporating more advanced neural network architectures and by using higher-resolution brain imaging techniques.
