By combining Hebbian theory, reinforcement, and noise you can actually create something that possibly could work, but surprisingly I could not find anyone that had attempted it. Probably because every one that has tried have failed, and their work have therefore not reached any wider audience. Being overly curious I decided to try it out myself. Unfortunately I also failed to achieve any significant learning but if you want to know a bit more about how I put it all together then just keep reading. The code is available here in case you want to see exactly how I implemented something.
The neurons in my artificial brain are based on the typical Hebbian theory neurons and stabilized by forcing the sum of the weights to equal one. The learning rate is derived from "reinforcement / average_reinforcement". The idea is that the neurons shall learn when the network is performing better than normally.
I added a non-traditional feature to the neurons where a weight that is lower than x becomes negative when calculating the neuron output. The reason for this is that I think the biological brain has some sort of negative weight construction. Without negative weights all neurons would trigger on an all one input. If this was the case we would see everything when looking at the sun, which does not happen (for me at least).
The noise is not as random as one might think. The neuron controls both the characteristics and timing of the noise. The timing is particularly interesting. A neuron needs to identify when it is under-performing relative to the rest of the network, and at that time re-purpose itself by applying noise to its weights. My neurons use the following five rules when deciding if it is time to apply noise or not.
- If the reinforcement is low the noise probability shall be high and vice versa. If things aren't going well we need to take more risks trying to find a way forward. If things are going well we don't want to risk ruining our well functioning network.
- If the neuron is primarily triggering when the reinforcement is higher than normally then the neuron is over-performing so the noise probability for that neuron shall be lower, and vice versa.
- If the neuron has not applied noise for a long time the probability for it applying noise shall be reduced. We don't want to destroy something that has worked well for a long time.
- If no one is picking up the neuron's output there is no interest for the pattern this neuron is detecting, so the neuron shall be re-purposed.
- If a neuron is constantly or never triggering it is not adding anything to the network and shall therefore be re-purposed.
I've tried to make sure that everything is compatible with the biological brain. The averages are for example approximations in one variable as storing long histories is probably difficult for a biological neuron.
For the simulation I used a slightly modified version of the environment from my previous reinforcement experiment (post). There are two circles, one moves randomly and the other is controlled by the artificial brain. The goal is to keep the two circles as close as possible. The artificial brain had 24+4 neurons in a 2-layer feed-forward configuration.
To have any chance of getting this working it all needs to be carefully tuned. There are 58 tunable variables and they where tuned using stochastic hill climbing.
After training for 100000 iterations (the video is 1000 iterations long) the behavior of the artificial brain looks as follows.
Not particularly impressive. Looks like the secrets of the biological brain are safe, for now.
20210502: I revisited this project and fixed some minor issues, scaled up both the size of the network and the tuning time (to ~6 months), but still no success. The new code can be found here.
No comments:
Post a Comment