Tuesday, February 14, 2017

Reinforcement learning

Code and the trained network can be found here.

To gain a deeper understanding of reinforcement learning I decided to write my own implementation.

I made my own environment (or game if you like). There are two circles. Each circle has a force acting on it. By controlling the direction of the force you can control the movement of the circle. One circle moves randomly and the other is controlled by the player. The goal is for the player to keep his circle as close as possible to the randomly moving circle.

I use a fully connected two layer network. The inputs are, target position, target velocity, player position, and player velocity. The first layer contains 160 neurons and is supposed to re-arrange the input to make it available via a present/not present approach rather than a parse strength approach, which is more difficult for neurons. This is done by having different neurons activate at different input strengths. By doing something like "neuron A but not neuron B" the second layer can get a specific input's strength.

The second layer contains 100 neurons and is supposed to identify environment states. Notice that we are not talking about exact states, could be something like, if the target is in this area and I'm in this area.

The output of the second layer is used to populate 32 classes. The classes are then turned into probabilities with softmax. Each class represents a specific force angle.

The loss function is a standard mean cross entropy construction.

The training contains three steps which are repeated till no more progress is made.
  1. Ten sessions/games are executed. At every frame the network is feed the current state and produces a probability for each output class. A random action (force angle) is chosen with the probability distribution returned from the network. So if the network has assign equal probability to all classes (the network is unsure of what to do) the action will be truly random, if the network is 70% sure class 4 should be used then class 4 will be used in 70% of the cases, if the network is 100% sure class 6 should be used then class 6 will definitely be used.
  2. Training data for the network is constructed from the states and actions in the best performing session.
  3. Seven network training steps are performed to increase the probabilities for the actions taken in the most successful session.
State-action pairs that are often in the winning session will be assigned higher probabilities. Higher probabilities means less exploration in those cases, so after a while it becomes likely that all the basic moves are in all sessions. At this point becoming the winning sessions is all about fine tuning the basic moves. This entire thing is repeated endlessly with ever finer fine tuning.

State-action pairs can contradict each other and this is how all the bad state-action pairs in the winning sessions are "removed". The bad state-action pairs in one iteration cancels out the bad state-action pairs in another iteration.

This is what it looks like before training



... and this is after a night of training

No comments:

Post a Comment