Sunday, December 18, 2016

Better voice recognition

Edited 20180121

Code, a trained network, and training/testing data can be found here.

I have previously done voice recognition with my audio classifier network (here), and that has worked well, but the network can only recognize voices it has been trained to recognize. The goal here is to be able to label previously unheard voices. To achieve this I have constructed a network that takes two voice samples as inputs and outputs whether they came from the same person or not. By running the network extensively over an audio file and doing some clustering the voices in the file can be labeled (more details in voice_recognition.py).

Input 1.1
2 x loudness adjusted spectrograms (65-3500Hz, 4096 window size, 40ms step, logarithmic scale for amplitudes) created from 2.5 seconds of 44.1kHz, 16-bit mono audio.

Rather than having the amplitudes in the "value" dimension a new third dimension has been added to the spectrograms that holds the amplitudes (the value dimension is unused and will always be 0 or 1). The reason for this is that a single filter is very good at detecting if something is present or not, but it can not detect that A and B should both be present and A should be exactly twice as strong as B, i.e. the filters have poor expressiveness in the value dimension.

Amplitude normalization is done per spectrum rather than over the entire spectrogram. This makes it easier to detect amplitude patterns over frequencies. The downside is that temporal amplitude patterns are lost.

Pitch has been normalized to make timbre detection easier. Pitchless sounds, like the short t sound, are destroyed by this process and are handled separately in layer 1.3.

Layer 1.1
200ms filters that are swept over the spectrograms and pooled to one value (per spectrogram).

The idea is that these filters will capture timbre features that can be used to separate voices. Such a feature would be continuously present in many voices but not in all voices. Typically each filter would be able to split voices into two classes and by using multiple filters a very good separation can be achieved. In a perfect world 10 filters could split voices into 2^10=1024 classes.

Input 1.2
Same as input 1.1 but without all the unusual stuff, i.e. 2-d, amplitude normalization over the entire spectrogram, and no pitch normalization.

Layer 1.2
Same as layer 1.1 but with a focus on pitch features.

Input 1.3
Same as input 1.1 but without pitch normalization and with a 2048 window size and a 20ms step to get more temporal resolution and less frequency resolution.

Layer 1.3
Same as layer 1.1 but for pitchless sounds.

Each layer 1 filter outputs a match value for voice sample 1 and a match value for voice sample 2. The difference between the match values is calculated so that we have a diff for each filter. All the filter diffs are then added up and if the sum is smaller than X the voice samples are from the same person.

I wrote some code to test how well humans perform this task, to get a target to aim for, and the results are:

python3 voice_recognition.py --test-human-performance voice_data/training_1/* voice_data/training_2/*

Known language, 67 voices, ~110 iterations: 97.3%
Unknown language, 13 voices, ~60 iterations: 98.9%

My network produced the following results after training. The training data was 89 well recorded (english) voices, a few minutes of each.

python3 voice_recognition.py --train voice_data/training_1/* voice_data/training_2/* voice_data/training_3/*

Training data: 92%
Test data: 85%

Below is a video of the network labeling never before seen data.

python3 voice_recognition.py test.raw


Labeling the training data looks like this.


I experimented with other inputs like cepstrograms and waveforms, but they worked worse than spectrograms. Applying filters directly on the waveform was found to not work well with gradient descent optimization. The problem is that the frequency the filter is trying to find will exist in many different phases in a batch and the gradients from the different versions will cancel each other out.

I also tried to use spectrograms with phase information as input but this did not improve performance. The phases where normalized in such a way that the component with the highest magnitude in each spectrum had a zero phase. This should make the filters able to detect phase differences between overtones.

No comments:

Post a Comment