Creating your first Neural Network

This article will show you how to initialise and train your first neural network.

Here we'll create our first neural network. The neural network we're creating will do as simple a task as possible, while still being (technically) impressive: we'll teach it how to square numbers! Although our goal here is kind of boring, it won't get in the way of the process shown here, which will apply to any other neural network you make.

This example was adapted from a question on stackoverflow that I stumbled across. I recommend reading through the question and top answer. They address the same problem using the same method as this article, with helpful explanatory comments.

First, we need to prepare our training data. Then, we initialise and then train our neural network. Finally, we'll round off by seeing how well our neural network performs.

Preparing our data

A NeuralNet is, fundamentally, a mathematical object. It takes in vectors as inputs and predicts the appropriate vectors to output. By giving a NeuralNet many sample input vectors paired with the expected vectors the NeuralNet should ideally output, the NeuralNet "learns" to produce increasingly accurate output vectors. This process is known as fitting.

Since all we want our NeuralNet to do is square numbers, the data we are working with is very simple. Our inputs are random numbers between -50 and 50, stored in a Vector<double> of length 1. Our outputs are the corresponding inputs squared.

We'll start by writing a function that'll give us such input-output pairs.

List<(Vector<double>, Vector<double>)> InputOutputPairs(int numPairs)
{
    List<(Vector<double> input, Vector<double> expectedOutput)> pairs = new(capacity: numPairs);
    
    for (int i = 0; i < numPairs; i++)
    {
        Vector<double> input = 50 * Vector<double>.Build.Random(length: 1);
        Vector<double> expectedOutput = input.PointwisePower(2);
        pairs.Add((input, expectedOutput));
    }
    return pairs;
}

Initialising our Neural Network

Before we start training, we need to create a "blank" NeuralNet that is ready to learn. We use NeuralNetFactory to create a NeuralNet that is optimised for our project.

Layer Structure

First, we have to decide on our neural network's layer structure. In this project, we take a single number as input and output a single number, so our input layer and output layer will both have a size of 1. To give our neural network some complexity, we'll also have two hidden layers, both of size 8. These hidden layers will use ReLU activation: an activation which works well for a wide range of projects.

Overall, our layer structure is: - an input layer of size 1 - a hidden layer of size 8 using ReLU activation - a hidden layer of size 8 using ReLU activation - an output layer of size 1

We'll package the structure of each layer into an InputLayer, HiddenLayer or OutputLayer record, all three of which inherit from NeuralLayerConfig. The structure of our neural network is represented by a List<NeuralLayerConfig>:

List<NeuralLayerConfig> layerStructure = new()
{
    new InputLayer(size: 1),
    new HiddenLayer(size: 8, activation: new ReluActivation()),
    new HiddenLayer(size: 8, activation: new ReluActivation()),
    new OutputLayer(size: 1)
};

Initialising using NeuralNetFactory

The NeuralNetFactory library class will create a NeuralNet that is optimised to learn for our project. Since we are using ReLU activation, we will make a NeuralNet, using our layer structure, that is optimised to learn with ReLU:

NeuralNet neuralNet = NeuralNetFactory.OptimisedForRelu(layerStructure);

Training our Neural Network

To train our neural network, we supply neuralNet with 8000 training data to fit to.

We will also supply a separate collection of 2000 testing data. The neural network will not train to fit to this data: instead, the data will be used to periodically assess neuralNet's performace as it learns.

List<(Vector<double>, Vector<double>)> trainingData = InputOutputPairs(8000);
List<(Vector<double>, Vector<double>)> testingData = InputOutputPairs(2000);

The neural network will learn from each pair in trainingData once every epoch. For this project, we will set the number of epochs to 15,000.

neuralNet.Fit(trainingData, testingData, numEpochs: 15000);

Fitting the neural network to trainingData took around 11 minutes on my mid-range laptop.

Seeing how well our Neural Network performs

To see how good neuralNet is at squaring numbers, let's see neuralNet's performance on ten random inputs, comparing neuralNet's predicted values with the true values:

foreach ((Vector<double> input, Vector<double> expectedOutput) in InputOutputPairs(10))
{
    Vector<double> predictedOutput = neuralNet.Predict(input);
    Console.WriteLine($"{input[0]} --> \t {predictedOutput[0]} \t (expected: {expectedOutput[0]})");
}

(Feel free to add your own formatting.)

My neuralNet performed as follows. Yours should perform similarly.

Performance of `neuralNet` with 10 randomly chosen inputs, compared with the expected outputs.

It's safe to say that neuralNet has roughly understood how squaring works: not only is neuralNet fairly accurate numerically, but it also seems to do as well with negative inputs as positive inputs. However, it struggles with squaring small numbers: e.g. 9.42 and 9.56 both get "squared" to 39.37 .

Ironing out problems like these will require more customisation and experimentation. To start, explore the different options in NeuralNetFactory and NeuralNet. Or, make yourself more confident with what you've already learned by creating a NeuralNet that learns a different function: perhaps f(x) = x^3 or f(x) = sin(x) rather than f(x) = x^2.