Neural Net Factory

The NeuralNetFactory class makes it easy to initialise a NeuralNet for most practical purposes.

Starting From Scratch

Here, we will initialise a NeuralNet that is ready to learn how to square numbers, while showing the other options along the way. Our NeuralNet's input will be a vector containing a single number to square, between -50 and 50. The output will be a vector containing a single number that is, approximately, the input number squared.

Layer Structure

First, decide on the structure you want your layers to have. This is done by creating a list of NeuralLayerConfig. For example, to create our "x squared" neural net described above, we will need the input and output layers to both be of size 1. The rest is up to you, but a good structure to go for is:

• an input layer of size 1
• a hidden layer of size 8, using ReLU activation
• a hidden layer of size 8, using ReLU activation
• an output layer of size 1, using no activation at all (that is, identity activation)

To do this, create the following list:

List<NeuralLayerConfig> layerStructure = new () 
{
    new InputLayer(size: 1),
    new HiddenLayer(size: 8, activation: new ReluActivation()),
    new HiddenLayer(size: 8, activation: new ReluActivation()),
    new OutputLayer(size: 1, activation: new IdentityActivation())
};

Gradient Descender

Next, decide on which method of gradient descent you wish to use. Adam gradient descent with default arguments is reccomended for most training data sets, and is used in this example:

GradientDescender gradientDescender = new AdamGradientDescender();

Cost Function

Next, decide on which cost function you wish to use. This will depend heavily on what kind of data you wish to learn. Here, since our outputs lie in a wide range (between -2500 and 2500), mean squared error is used:

CostFunction cost = new MSECost();

Creating your Neural Network

There are various ways of best initializing a NeuralNet, depending on which activators it will use and what training data it will be learning.

Optimising for ReLU learning

If your layer structure mostly uses ReLU activation (such as in our example), you can create a Neural Net that is optimised for learning with ReLU:

NeuralNet neuralNet = NeuralNetFactory.OptimisedForRelu(layerStructure, gradientDescender, cost);

This is it: our NeuralNet has been created, ready to learn how to square numbers!

Optimising for tanh learning

If, however, your layer structure mostly uses tanh activation, you can create a NeuralNet that is optimised for learning with tanh:

NeuralNet neuralNet = NeuralNetFactory.OptimisedForTanh(layerStructure, gradientDescender, cost);

Optimising for learning a particular data set

A more recent option is to create a NeuralNet that is optimised for learning some particular training data. When optimising, this will take into account the NeuralNet's layer structure and activators. However, it will not take into account the gradient descent method or cost function used.

IEnumerable<(Vector<double> input, Vector<double> desiredOutput)> trainingData = ... ;
NeuralNet neuralNet = NeuralNetFactory.OptimisedForTrainingData(layerStructure, trainingData, gradientDescender, cost);

Since the method used in NeuralNetFactory.OptimisedForTrainingData() only uses the input vectors in the training data, there is an overload that simply takes the training input vectors:

IEnumerable<Vector<double>> trainingInputs = ... ;
NeuralNet neuralNet = NeuralNetFactory.OptimisedForTrainingData(layerStructure, trainingInputs, gradientDescender, cost);

Reading a NeuralNet from a directory

If you have previously written a NeuralNet to a directory using NeuralNet.WriteToDirectory(), then NeuralNetFactory.ReadFromDirectory() will load the NeuralNet back in. NeuralNet.ReadFromDirectory() returns a new NeuralNet with identical:

• Parameter (so, identical weights, biases and layer structure)
• Activators applied to their corresponding layers
• GradientDescender
• CostFunction to the original NeuralNet that was written to the directory:

NeuralNet neuralNet = ... ;
neuralNet.Fit(...);
neuralNet.WriteToDirectory("../../neural-net-state");

NeuralNet read = NeuralNetFactory.ReadFromDirectory("../../neural-net-state");
// `read` now has the same internal state as `neuralNet`.