Neural Net

A NeuralNet takes a Vector<double> as input and predicts the appropriate Vector<double>to output. By givingNeuralNeta training data set, consisting of pairs of sample inputs with the correct output, theNeuralNetwill learn to produce more accurate outputs. Not only should theNeuralNet` perform accuately with the training data given, but it should also perform just as accuately with similar inputs that it has not been trained on.

Creating a Blank NeuralNet

See NeuralNetFactory for how to create a "blank" NeuralNet that is ready to train.

Preparing Training Data for a NeuralNet

The NeuralNet will learn how to produce approximately the right output for a given input. Thus, you will need to prepare your data as a list of inputs paired with the corresponding output you want to be produced.

Because a NeuralNet is a mathematical object, each input and output will need to be a Vector<double>. Every input vector must have the same length, and every output vector must have the same length. The output vectors can have different lengths to the input vectors, however. If you want your NeuralNet to input or output a single number, you must use Vector<double>s of length 1.

See creating your first neural network for a detailed example.

Fitting a NeuralNet to Training Data

Once you have prepared your training data into a collection of paired input and output vectors, use NeuralNet.Fit(...) to train the NeuralNet. This will repeatedly adjust the neural network so that its predicted output vectors are increasingly closer to the expected output vectors.

List<(Vector<double>, Vector<double>)> trainingData = ... ; // see above
neuralNet.Fit(trainingData, numEpochs: 50000);

There are three arguments to configure the training:

Number of Epochs

The NeuralNet trains from the entire data set once every epoch. In other words, the amount of epochs is the amount of times the NeuralNet learns from each data point in traningData.

Up to a point, the more times the NeuralNet learns from each data point, the more accurate the NeuralNet will be. However, if the number of epochs is higher than necessary, the NeuralNet will try "too hard" to match the input data exactly, and start to lose its ability to perform well on similar data that it has not seen. This is called over-fitting.

It is important to pick a value of numEpochs that is high enough for the NeuralNet to learn well from each data point, but not too high that the NeuralNet starts to over-fit. Read Measuring Performance while Training for more information.

Batch Size

NeuralNet.Fit(...) performs batch gradient descent: instead of learning from each data point one at a time, the NeuralNet learns from multiple data points at once, the amount of which is the batch size. Reccommended values range from 4 to 256, with 256 being the default value.

Batch Updating in Parallel

If this options is enabled, the NeuralNet learns from each data point in a given batch in parallel. (Technical note: the NeuralNet calculates the cost gradient for each data point in the batch in parallel, then learns from the average of the cost gradients sequentially.) This is done for performance gains, which will be greater the higher the Batch Size.

Measuring Performance

Average Cost

You can measure a NeuralNet's performance using NeuralNet.AverageCost(...).

When fitting to training data, the neural network seeks to decrease the cost quantifying the error in the neural network's predicted outputs versus the expected outputs in the training data. NeuralNet.AverageCost(...) returns the average cost of the neural network's error in the supplied testing data.

NeuralNet neuralNet = ... ;
List<(Vector<double> input, Vector<double> expectedOutput)> testingData = ... ;
double averageCost = neuralNet.AverageCost(testingData);

Warning

To get a meaningful indication of how the neural network performs, supply AverageCost(...) with testing data that is different from the training data This way, the average cost will measure how well the neural network generalises to data it has not trained on

Measuring Cost while Training

You can assess a NeuralNet's performance while training by supplying testing data to NeuralNet.Fit(...). This will assess the NeuralNet's performance by writing the average cost of the NeuralNet on the testing data periodically while training.

NeuralNet neuralNet = ... ;
List<(Vector<double>, Vector<double>)> trainingData = ... ;
List<(Vector<double>, Vector<double>)> testingData = ... ;
neuralNet.Fit(trainingData, testingData);

The amount of times the average cost is written to the console can be configured by adjusting numAssessments. This number is 15 by default.

NeuralNet neuralNet = ... ;
List<(Vector<double>, Vector<double>)> trainingData = ... ;
List<(Vector<double>, Vector<double>)> testingData = ... ;
neuralNet.Fit(trainingData, testingData, numAssessments: 20);

Predicting Output Vectors

Given an input Vector<double>, the NeuralNet predicts the appropriate output Vector<double>. Once the NeuralNet has sufficiently fitted to training data, you can expect the predicted output to be accurate, but not a complete match.

The following example uses a NeuralNet with an input layer of size 10 and an output layer of size 5. The example shows the NeuralNet's predicted output on three random input vectors.

List<NeuralLayerConfig> layerStructure = new()
{
    new InputLayer(size: 10),
    ...
    new OutputLayer(size: 5)
};

NeuralNet neuralNet = NeuralNetFactory.OptimisedForRelu(layerStructure, ...);
neuralNet.Fit(...);

for (int i = 0; i < 3; i++)
{
    Vector<double> input = Vector<double>.Build.Random(length: 10);
    Vector<double> predictedOutput = neuralNet.Predict(input); // is of length 5

    Console.WriteLine($"input: {input}");
    Console.WriteLine($"predicted output: {predictedOutput}");
    Console.WriteLine();
}