Monday, December 5, 2016

Using class distribution in the training data as optimization target

All resources referenced in this post can be found here.

In my audio classification network (post) every single example needs to be labeled. Labeling is very time consuming, and boring, so I would like to avoid it as much as possible. If the class distribution in the training data is known one could possibly use that as an optimization target and thereby label the entire training data at once. You would tell the network to split the training data into, for example, 2 classes with the probabilities 0.3 and 0.7. Hopefully the network splits the data the way you intended to.

The training data could be a news show where you know the anchorman speaks 20% of the time. To train the network to identify the anchorman's voice you would use x number of shows as training data and tell the network to split them into two classes with the probabilities 0.2 and 0.8. The 0.2 class will probably be assigned to the anchorman's voice.

Using the probability that a class wins in a loss function is not possible as it is not smooth. For the probability to change one example needs to change class and at that time the probability will jump to a new value. The result of this is a terraced search space where the gradient is either 0 or infinity at all positions. A gradient descent optimizer will not work in such a search space.

To work around this problem I used a composite loss function that looked at two things.
  1. The mean of the class probabilities for all the examples. In the news show example you would be aiming for 0.2, 0.8. Notice that this does not necessarily mean that class 0 has won 20% of the time and class 1 has won 80% of the time. If every example has generated the probabilities 0.2, 0.8 class 1 has won 100% of the time but the mean of the class probabilities is 0.2, 0.8.
  2. The standard deviation of the probabilities in one class. For class 0 in the new show example you would want 20% of the probabilities to be 1 and the rest 0. For class 1 you would want 80% of the probabilities to be 1 and the rest 0.
When combined the search space will be smooth and the minima will be correctly placed. For more detailed information look at the code in audio_classifier.py.

To test this idea I used my audio classifier network and the same Nathan Englander data I used last time. I just put the training data for both classes in one file and used the new class distribution loss function.

python audio_classifier.py --train --labeled-data-weight 0.0 nathan_data/nathan_englander_and_other.raw 0.135 0.865 nathan_data/nathan_englander_short.raw nathan_data/other_short.raw

The result looks as follows.

Interview with Nathan Englander
Nathan Englander reading a book
Interview with Ariana Grande
Dialog from the Ex Machina movie
Interview with Samuel L. Jackson
Skrillex - Bangarang
Will Self reading a book

The network has split the data the way I wanted it to but the accuracy is not as good as with labeled training data. Not sure why the accuracy dropped but it did not look like more training was going to fix it.

Adding some labeled data could possibly steer the solution in the right direction and increase the accuracy without requiring a lot of manual labor. Below I have labeled 2% of the training data.

python audio_classifier.py --train nathan_data/nathan_englander_and_other.raw 0.135 0.865 nathan_data/nathan_englander_short.raw nathan_data/other_short.raw

Interview with Nathan Englander
Nathan Englander reading a book
Interview with Ariana Grande
Dialog from the Ex Machina movie
Interview with Samuel L. Jackson
Skrillex - Bangarang
Will Self reading a book


Better. But still a bit worse than with fully labeled training data.

It would also be interesting to see how well the 2% set of the training data does on its own, without the class distribution. Maybe the class distribution only makes things worse. So I tried that also.

python audio_classifier.py --train --labeled-data-weight 1.0 nathan_data/nathan_englander_and_other.raw 0.135 0.865 nathan_data/nathan_englander_short.raw nathan_data/other_short.raw

Interview with Nathan Englander
Nathan Englander reading a book
Interview with Ariana Grande
Dialog from the Ex Machine movie
Interview with Samuel L Jackson
Skrillex - Bangarang
Will Self reading a book


It actually works much better than I expected. The mixed version is better, but not by much.

No comments:

Post a Comment