Thursday, December 29, 2016

Are you getting paid enough for your good looks and charming personality?


All resources referenced in this post can be found here.

I have obtained salary information for a large number of people and it can be used to answer the important question in the title

Created with married_couple_salary_diff.py

With the information in the diagram above you can check if your relationship is competitive or not. Use the following process.
  1. If money is excluded how does the balance look in your relationship?
  2. What is the probability of finding a couple with an equal or worse imbalance? The direction is important here, whether the man or woman is the "winner".

    If there where only 10 couples with the following imbalances



    ... the probabilities would look like this.

    c1 probability = 1 / 10 = 0.1
    c2 probability = 2 / 10 = 0.2
    c4 probability = 7 / 10 = 0.7
    c6 probability = 5 / 10 = 0.5
  3. Locate the probability found in step 2 on the y-axis and read out the factor on the x-axis
Lets do an example. Alice wants to see if her marriage is competitive. If money is excluded she thinks she is a bit "better" than her husband Bob. Alice estimates that the woman is at least this much better than the man in 20% of all marriages. Looking at the diagram she sees that Bob should be making ~2.15 times as much as she does, which is the case, so no action needed.

How did I get hold of the data? The information originally comes from the Swedish tax authorities but to get hold of it in large quantities you need to buy it in book form, "Taxeringskalendern". To save space in the book people with low income have been filtered out, but the limits are pretty low so almost everyone working should be in there.

To use the information for anything more sophisticated than checking how much money your neighbor makes you need the data in digital form. To do this I shot a movie while flipping through the book and ran Tesseract (OCR software) on every frame (parse_tax_records.py). This approach worked pretty poorly but I managed to collect enough information to do the analysis in this post. I think the poor performance was mainly due to the book's content not being in Tesseract's dictionary and the font being really dense.

As a bonus I give you the salary distribution in the Malmö/Lund area, derived from the same data.

Created with salary_distribution.py

Estimate the likelihood of finding someone equal or better than you in the workforce (everyone working, not just your profession), locate the likelihood on the y-axis and read out the salary you should have on the x-axis.

Sunday, December 18, 2016

Better voice recognition

Edited 20180121

Code, a trained network, and training/testing data can be found here.

I have previously done voice recognition with my audio classifier network (here), and that has worked well, but the network can only recognize voices it has been trained to recognize. The goal here is to be able to label previously unheard voices. To achieve this I have constructed a network that takes two voice samples as inputs and outputs whether they came from the same person or not. By running the network extensively over an audio file and doing some clustering the voices in the file can be labeled (more details in voice_recognition.py).

Input 1.1
2 x loudness adjusted spectrograms (65-3500Hz, 4096 window size, 40ms step, logarithmic scale for amplitudes) created from 2.5 seconds of 44.1kHz, 16-bit mono audio.

Rather than having the amplitudes in the "value" dimension a new third dimension has been added to the spectrograms that holds the amplitudes (the value dimension is unused and will always be 0 or 1). The reason for this is that a single filter is very good at detecting if something is present or not, but it can not detect that A and B should both be present and A should be exactly twice as strong as B, i.e. the filters have poor expressiveness in the value dimension.

Amplitude normalization is done per spectrum rather than over the entire spectrogram. This makes it easier to detect amplitude patterns over frequencies. The downside is that temporal amplitude patterns are lost.

Pitch has been normalized to make timbre detection easier. Pitchless sounds, like the short t sound, are destroyed by this process and are handled separately in layer 1.3.

Layer 1.1
200ms filters that are swept over the spectrograms and pooled to one value (per spectrogram).

The idea is that these filters will capture timbre features that can be used to separate voices. Such a feature would be continuously present in many voices but not in all voices. Typically each filter would be able to split voices into two classes and by using multiple filters a very good separation can be achieved. In a perfect world 10 filters could split voices into 2^10=1024 classes.

Input 1.2
Same as input 1.1 but without all the unusual stuff, i.e. 2-d, amplitude normalization over the entire spectrogram, and no pitch normalization.

Layer 1.2
Same as layer 1.1 but with a focus on pitch features.

Input 1.3
Same as input 1.1 but without pitch normalization and with a 2048 window size and a 20ms step to get more temporal resolution and less frequency resolution.

Layer 1.3
Same as layer 1.1 but for pitchless sounds.

Each layer 1 filter outputs a match value for voice sample 1 and a match value for voice sample 2. The difference between the match values is calculated so that we have a diff for each filter. All the filter diffs are then added up and if the sum is smaller than X the voice samples are from the same person.

I wrote some code to test how well humans perform this task, to get a target to aim for, and the results are:

python3 voice_recognition.py --test-human-performance voice_data/training_1/* voice_data/training_2/*

Known language, 67 voices, ~110 iterations: 97.3%
Unknown language, 13 voices, ~60 iterations: 98.9%

My network produced the following results after training. The training data was 89 well recorded (english) voices, a few minutes of each.

python3 voice_recognition.py --train voice_data/training_1/* voice_data/training_2/* voice_data/training_3/*

Training data: 92%
Test data: 85%

Below is a video of the network labeling never before seen data.

python3 voice_recognition.py test.raw


Labeling the training data looks like this.


I experimented with other inputs like cepstrograms and waveforms, but they worked worse than spectrograms. Applying filters directly on the waveform was found to not work well with gradient descent optimization. The problem is that the frequency the filter is trying to find will exist in many different phases in a batch and the gradients from the different versions will cancel each other out.

I also tried to use spectrograms with phase information as input but this did not improve performance. The phases where normalized in such a way that the component with the highest magnitude in each spectrum had a zero phase. This should make the filters able to detect phase differences between overtones.

Monday, December 5, 2016

Using class distribution in the training data as optimization target

All resources referenced in this post can be found here.

In my audio classification network (post) every single example needs to be labeled. Labeling is very time consuming, and boring, so I would like to avoid it as much as possible. If the class distribution in the training data is known one could possibly use that as an optimization target and thereby label the entire training data at once. You would tell the network to split the training data into, for example, 2 classes with the probabilities 0.3 and 0.7. Hopefully the network splits the data the way you intended to.

The training data could be a news show where you know the anchorman speaks 20% of the time. To train the network to identify the anchorman's voice you would use x number of shows as training data and tell the network to split them into two classes with the probabilities 0.2 and 0.8. The 0.2 class will probably be assigned to the anchorman's voice.

Using the probability that a class wins in a loss function is not possible as it is not smooth. For the probability to change one example needs to change class and at that time the probability will jump to a new value. The result of this is a terraced search space where the gradient is either 0 or infinity at all positions. A gradient descent optimizer will not work in such a search space.

To work around this problem I used a composite loss function that looked at two things.
  1. The mean of the class probabilities for all the examples. In the news show example you would be aiming for 0.2, 0.8. Notice that this does not necessarily mean that class 0 has won 20% of the time and class 1 has won 80% of the time. If every example has generated the probabilities 0.2, 0.8 class 1 has won 100% of the time but the mean of the class probabilities is 0.2, 0.8.
  2. The standard deviation of the probabilities in one class. For class 0 in the new show example you would want 20% of the probabilities to be 1 and the rest 0. For class 1 you would want 80% of the probabilities to be 1 and the rest 0.
When combined the search space will be smooth and the minima will be correctly placed. For more detailed information look at the code in audio_classifier.py.

To test this idea I used my audio classifier network and the same Nathan Englander data I used last time. I just put the training data for both classes in one file and used the new class distribution loss function.

python audio_classifier.py --train --labeled-data-weight 0.0 nathan_data/nathan_englander_and_other.raw 0.135 0.865 nathan_data/nathan_englander_short.raw nathan_data/other_short.raw

The result looks as follows.

Interview with Nathan Englander
Nathan Englander reading a book
Interview with Ariana Grande
Dialog from the Ex Machina movie
Interview with Samuel L. Jackson
Skrillex - Bangarang
Will Self reading a book

The network has split the data the way I wanted it to but the accuracy is not as good as with labeled training data. Not sure why the accuracy dropped but it did not look like more training was going to fix it.

Adding some labeled data could possibly steer the solution in the right direction and increase the accuracy without requiring a lot of manual labor. Below I have labeled 2% of the training data.

python audio_classifier.py --train nathan_data/nathan_englander_and_other.raw 0.135 0.865 nathan_data/nathan_englander_short.raw nathan_data/other_short.raw

Interview with Nathan Englander
Nathan Englander reading a book
Interview with Ariana Grande
Dialog from the Ex Machina movie
Interview with Samuel L. Jackson
Skrillex - Bangarang
Will Self reading a book


Better. But still a bit worse than with fully labeled training data.

It would also be interesting to see how well the 2% set of the training data does on its own, without the class distribution. Maybe the class distribution only makes things worse. So I tried that also.

python audio_classifier.py --train --labeled-data-weight 1.0 nathan_data/nathan_englander_and_other.raw 0.135 0.865 nathan_data/nathan_englander_short.raw nathan_data/other_short.raw

Interview with Nathan Englander
Nathan Englander reading a book
Interview with Ariana Grande
Dialog from the Ex Machine movie
Interview with Samuel L Jackson
Skrillex - Bangarang
Will Self reading a book


It actually works much better than I expected. The mixed version is better, but not by much.