To generate text you use a network that can predict the next character in a stream of characters. The network outputs a probability for each possible next character, 128 probabilities for ASCII text. If you input "cat" the network might give a 40% likelihood of the next character being a space, 20% that the next character is "s" (to create "cats"), and 0.001% that next character is "\". Using the probabilities the network produced you append the stream with a random character and then you re-run it all. By repeating this process a random text of any length can be produced that has the characteristics of the text the network was trained on.
A LSTM-network is typically used for the character prediction. The input to the network is just a single character but since the network has an internal state it can remember the previously seen characters. For the "cat" example above the network would be run three times, once for each character, but since the network has an internal state the last iteration would be aware of the entire history. One thing to keep in mind is that during training an unrolled version of the network is used. You unroll maybe 100 iterations into a huge network where the weights are re-used (100 times). The unrolled network accepts 100 inputs and produces 100 outputs, of which only the last one is interesting. A consequence of this is that the network can not learn patterns that do not fit in the unrolled network. If the next character depends heavily on something that happened ~200 characters ago the network would not be able to figure it out because during training it was never able to see that far back in the history.
My network has three layers and each layer has a 512 element state, ~5 million variables. I unroll 175 iterations during training. The loss function is a standard softmax + cross-entropy construction. If you want to have look at the code or try it out yourself the script can be downloaded here.
I trained the network on all the C code in the Linux kernel and after ~24 hours of training it generates text that looks as follows. My script uses the adam optimizer, which gives very quick progress, but it tends to freak out after a while (error in tensorflow's implementation?) so using another optimizer might be a good idea.
static void ct1812_all_channels(struct device *dev, int width, int chan,
struct cec_dev *cd, int gpio,
struct c41d_spdif_data *data, int di_size)
{
struct cs42xxxfb_data *ddata = cs5>data->csi2;
struct fsl_adc_priv *dev = fe->device;
for (i = 0; i < ARRAY_SIZE(cs45l42_padswrt); i++)
snd_pcm_csi_buradc(chip->usb_ports[i].pwm, &asus,
CS4231_ISCSI_PKT_AEBA_PG);
/* Call dis_tdp_queue chained data */
cs42l42 = cs45;
dev_dbg(cs->dev, "cc %u rxt %p\n", cs35l33);
cs428_set_mem_period(cs, csd->msb_cmd, bus, CS35L47_GPT00,
bss[i+mips_csio_blk->type]);
}
It looks like C code but if you look at what the code does it makes no sense at all.
No comments:
Post a Comment