All resources referenced in this post can be found here.
As the title says I have been experimenting with audio classification with a convolutional neural network. Input to the network is loudness adjusted spectrograms. The spectrograms cover 65-3500Hz for two seconds. The temporal resolution is 40ms and the frequency resolution is ~11Hz, 48x319 values.
The network looks like this:
80 6x250 (240ms x 2692Hz) filters that are supposed to capture overtone features (timbre)
ReLU
3x1 max pooling to keep the network size down
40 8x70 (960ms x 754Hz) filters that are supposed to capture longer temporal features
ReLU
Max pooling to one value
The classes are populated from the layer 2 outputs according to a weight vector for each class and then converted to probabilities using softmax. The network contains ~1.9 million trainable variables and performs ~377 million operations.
The loss function is the mean of the cross entropies between the expected probability distribution and the actual probability distribution for each example.
For more detailed information check out audio_classifier.py.
To test it all out I decided to train the network to recognize Nathan Englander's voice. To do this I used two classes. One class for Nathan's voice, which was trained on 50 minutes of Nathan's voice from three different recordings. Another class for everything else, which was trained on 5 hours of audio not containing Nathan's voice (music, sound from Youtube and movies, book readings, etc).
python audio_classifier.py --train 2 nathan_englander.raw other.raw
The network quickly achieved 100% accuracy on the training data, something like 10-20 hours on my i5-4460.
The following images show the performance on the test data. None of the test data is present in the training data. No person, except Nathan Englander, and no recording exists in both data sets.
python audio_classifier.py 2 nathan_englander_interview.raw
![]() |
| Interview with Nathan Englander |
![]() |
| Nathan Englander reading a book |
![]() |
| Interview with Ariana Grande |
![]() |
| Dialog from the Ex Machina movie |
![]() |
| Interview with Samuel L. Jackson |
![]() |
| Skrillex - Bangarang |
![]() |
| Will Self reading a book |







No comments:
Post a Comment