Last time, we played with Karmen Blake's Neural Network Library. We just used it to construct some basic networks. Let's have a look at the code to see how he pulled it together.

Project

We'll start by looking at the or mix task we created last time. The first thing we did was:

    {:ok, network_pid} = Network.start_link([2,1])

Let's look at that function. It takes a list of integers. Each integer defines the number of neurons in incrementally-declared layers of the network. In our case, we have an input layer with two neurons and an output layer with one neuron, with no hidden layers.

defmodule NeuralNetwork.Network do
  @moduledoc """
  Contains layers which makes up a matrix of neurons.
  """
  # ...
  @doc """
  Pass in layer sizes which will generate the layers for the network.
  The first number represents the number of neurons in the input layer.
  The last number represents the number of neurons in the output layer.
  [Optionally] The middle numbers represent the number of neurons for hidden layers.
  """
  def start_link(layer_sizes \\ []) do
    {:ok, pid} = Agent.start_link(fn -> %Network{} end)

    layers = map_layers(
      input_neurons(layer_sizes),
      hidden_neurons(layer_sizes),
      output_neurons(layer_sizes))

    pid |> update(layers)
    pid |> connect_layers
    {:ok, pid}
  end
  # ...
end

This function starts an agent and returns its pid. The agent wraps a %Network{} struct. That struct looks like this:

  defstruct pid: nil, input_layer: nil, hidden_layers: [], output_layer: nil, error: 0

  # It has a pid, an input layer, some hidden layers, an output layer, and an error.

Next, start_link maps the layers, producing the input layer, the hidden layers, and the output layer.

The Input Layer

The input layer consists of a Layer with the number of neurons that were specified as the first element of the list provided to start_link - 2, in our case.

  defp input_neurons(layer_sizes) do
    size = layer_sizes |> List.first
    {:ok, pid} = Layer.start_link(%{neuron_size: size})
    pid
  end

This layer is linked to the caller that linked the first network itself, since it's code that executes in the caller - it's called from start_link. It seems to me that perhaps it would make more sense for it to be linked to the Network process itself, called in the init callback of a GenServer, but that doesn't affect anything in normal operation.

Let's look at what starting a Layer looks like as well:

defmodule NeuralNetwork.Layer do
  @moduledoc """
  List of neurons. The are used to apply behaviors on sets of neurons.
  A network is made up layers (which are made up of neurons).
  """
  # ...
  def start_link(layer_fields \\ %{}) do
    {:ok, pid} = Agent.start_link(fn -> %Layer{} end)
    neurons = create_neurons(Map.get(layer_fields, :neuron_size))
    pid |> update(%{pid: pid, neurons: neurons})

    {:ok, pid}
  end
  # ...
end

Each layer is an Agent wrapping a %Layer{} struct:

  defstruct pid: nil, neurons: []

  # It's just a pid and a list of neurons

An appropriate number of neurons are created, by calling Neuron.start_link:

defmodule NeuralNetwork.Neuron do
  @moduledoc """
  A neuron makes up a network. It's purpose is to sum its inputs
  and compute an output. During training the neurons adjust weights
  of its outgoing connections to other neurons.
  """
  # ...
  @doc """
  Create a neuron agent
  """
  def start_link(neuron_fields \\ %{}) do
    {:ok, pid} = Agent.start_link(fn -> %Neuron{} end)
    update(pid, Map.merge(neuron_fields, %{pid: pid}))

    {:ok, pid}
  end
  # ...
end

Again, these are simple agents wrapping a %Neuron{} struct:

  defstruct pid: nil, input: 0, output: 0, incoming: [], outgoing: [], bias?: false, delta: 0

  # A neuron knows its pid, has an input and output, and tracks incoming and
  # outgoing values. It could be a bias neuron, and it also tracks how close it
  # is to 'right' when it's being trained, in the 'delta' field.

The Hidden Layer

We didn't define hidden layers, so I won't spend any time on this.

The Output Layer

We have a single output for or. Here's the code that generates the output layer:

  defp output_neurons(layer_sizes) do
    size = layer_sizes |> List.last
    {:ok, pid} = Layer.start_link(%{neuron_size: size})
    pid
  end

map_layers

The map_layers function turns these three functions into a map that breaks them apart:

  defp map_layers(input_layer, hidden_layers, output_layer) do
    %{
      input_layer:    input_layer,
      output_layer:   output_layer,
      hidden_layers:  hidden_layers
    }
  end

Updating the agent

Next, the agent is updated with these three layers:

    pid |> update(layers)
  @doc """
  Update the network layers.
  """
  def update(pid, fields) do
    fields = Map.merge(fields, %{pid: pid}) # preserve the pid!!
    Agent.update(pid,  &(Map.merge(&1, fields)))
  end

This merges the fields into the neural network, as well as puts the pid into the Network struct. Now the agent is wrapping a struct that represents the whole network, aware of the other layers that are running concurrently in their own agents.

Connecting the layers

Next, the layers are connected, with:

    pid |> connect_layers
  defp connect_layers(pid) do
    layers = pid |> Network.get |> flatten_layers

    layers
    |> Stream.with_index
    |> Enum.each(fn(tuple) ->
      {layer, index} = tuple
      next_index = index + 1

      if Enum.at(layers, next_index) do
        Layer.connect(layer, Enum.at(layers, next_index))
      end
    end)
  end

For each layer, if there's a "next layer", Layer.connect is called for the layer and the next layer.

defmodule NeuralNetwork.Layer do
  # ...
  @doc """
  Connect every neuron in the input layer to every neuron in the target layer.
  """
  def connect(input_layer_pid, output_layer_pid) do
    input_layer  = get(input_layer_pid)

    unless contains_bias?(input_layer) do
      {:ok, pid} = Neuron.start_link(%{bias?: true})
      input_layer_pid |> add_neurons([pid])
    end

    for source_neuron <- get(input_layer_pid).neurons, target_neuron <- get(output_layer_pid).neurons do
      Neuron.connect(source_neuron, target_neuron)
    end
  end
  # ...
end

This subsequently calls Neuron.connect for each pair of neurons in the target layer and the output layer:

defmodule NeuralNetwork.Neuron do
  # ...
  @doc """
  Connect two neurons
  """
  def connect(source_neuron_pid, target_neuron_pid) do
    {:ok, connection_pid} = Connection.connection_for(source_neuron_pid, target_neuron_pid)

    source_neuron_pid
    |> update(%{outgoing: get(source_neuron_pid).outgoing ++ [connection_pid]})

    target_neuron_pid
    |> update(%{incoming: get(target_neuron_pid).incoming ++ [connection_pid]})
  end
  # ...
end

This creates a connection for each pair and updates the individual neurons to know about the incoming and outgoing connections.

defmodule NeuralNetwork.Connection do
  @moduledoc """
  Neurons communciate via connections.
  Connection weights determine the network output and are updated while training occurs.
  Network capability is represented in the network matrix of weight values.
  """
  # ...
  defstruct pid: nil, source_pid: nil, target_pid: nil, weight: 0.4 # make weight random at some point
  # ...
end

A connection just has a source, a target, and a weight. The weight will be updated as the network trains.

Summary

That's a pretty good place to stop for now. We've walked through the setup of the network. The next step is to look at how training happens. See you soon!

Resources