Introduction to Neural Networks

An Application of Matrix Multiplication

 

Written By:        Yash Gad

 

Grade Level:      9 - 12

 

Subjects:              Applied Mathematics – Matrix Algebra

                                    Biology – Neuroscience  

 

Description:        An introduction to neural networks modeling, focusing primarily on the representation of neural systems using matrix algebra.

 

Objectives:          Students will be able to represent small networks of neurons, and compute the “output” using matrix multiplication.

 

Material:                (Optional) Computers with mathematics software, such as Mathematica or Matlab

 

Background

 

            Research in the area of neuroscience has exploded over the last 15 years, thanks in part to the insight gained from computational neuroscience. Computational neuroscience uses mathematical and computer simulations to represent portions of the brain, with varying degrees of detail. At one end of the spectrum, researchers have utilized detailed biophysical models, whose aim it is to accurately represent all the important properties of a neuron. The other end of the spectrum involves neural networks.

 

The basic principle behind neural networks, when applied to neuroscience research, is to create a mathematical model of the connections in a neural system. These connections are the excitatory and inhibitory synapses between neurons. At an abstract level, this is often enough to capture the overall computation performed by the system.

 

Since their introduction in the early part of the century, neural networks have been used in a variety of area. Areas of artificial intelligence have borrowed heavily from the neural circuitry of the brain, and several new models of learning have emerged as a result. Because of their ability to be taught, neural networks have been integrated into software for pattern and speech recognition, and data analysis.

 

For further references regarding Neural Networks:

 

http://www.genesis-sim.org/GENESIS/cnsweb/cns1.html#IntroCNS

 

Lecture Outlines

 

Part I – Refresher on Matrix Algebra

 

For material on Matrix Algebra:

http://www.sosmath.com/matrix/matrix0/matrix0.html

http://www.sosmath.com/matrix/matrix1/matrix1.html

http://www.sosmath.com/matrix/matrix2/matrix2.html

 

Main objectives:

 

Examples

 

Addition:

 

Subtraction

 

Multiplication

Part II – Introduction to Neural Connections

 

Within any neural system, there are 2 basic synapses that can be made from one neuron to another – inhibitory or excitatory. At a cellular level, these connections cause the membrane voltage of the postsynaptic cell to decrease or increase, respectively. Since action potentials, the “bits” of data generate by neurons, are all-or-nothing responses, information is coded by the pattern of these bits and their rate (which is action potentials per second). For this introductory material, we will assume that all outputs and inputs relate to some rate of activity.

 

For further resources:

http://www.mind.ilstu.edu/curriculum2/neuro/neuron_1.html

 

In this and future lessons, excitatory connections will be represented as:

 

While inhibitory connections will be represented as:

 

To see how these types of connections relate to one another, let us look at two scenarios (where the graphs below the neurons will represent activity).

Scenario 1

An excitatory connection from an input neuron onto an output neuron.

To analyze this setup, we would say for the first case that “If there is no input from the excitatory neuron, then there is no activity in the output neuron”. In the second case, we would say that “If there is an input from the excitatory neuron, then there is a proportional amount of activity in the output neuron”.

 

Scenario 2

An excitatory connection from an input neuron onto an output neuron

In this network, the combination of both an active excitatory and active inhibitory input will cancel out, yielding no net activity in the output neuron.

Part III – Matrix Representations of Neural Connections

 

So far, we have illustrated the rules of matrix algebra, and basic neural connections. These two ideas can be combined to yield a powerful method by which to represent large networks. Let us rework scenario 2 using matrices.

When we say excitatory, what we are really saying is that it has a positive influence on the postsynaptic cell. Conversely, when we say inhibitory, it has a negative influence. We have labeled their synapses accordingly with a +1 and -1. We have also given the activities in these input cells a value of 1. If we arbitrarily call the excitatory neuron #1, the inhibitory neuron #2, and the output neuron #3, we can translate all of this information into a set of vectors (i.e. 1 row matrices):

 

Input vector = (1 1), for which the first element corresponds to the activity of neuron 1, and the second element corresponds to the activity of neuron 2.

 

Weight vector = (1 -1), for which the first element corresponds to the connection weight of neuron 1, and the second element corresponds to the connection weight of neuron 2.

 

The outputs of neurons 1 and 2, which are the influences each exerts on neuron 3, are simply the product of their activities and the connection weights.

 

This can be generalized to a matrix multiplication problem, so that

 

Neuron 3’s activity = Input * Weight = 0, because

 

By varying the connection weight values, and the input values, we can see that neuron 3 (which is just taking the sum of all these inputs) can take on a wide range of values. To reiterate, let us try a few examples. You can see that after the structure of the network is clear, the representation can just remain a series of matrices.

 

Example 1

Neuron 1 Activity=1, Weight=2

Neuron 2 Activity=1, Weight=-1

 

Output = ?

 

So the output (the activity of Neuron 3) is 1.

 

Example 2

Neuron 1 Activity=1, Weight=1

Neuron 2 Activity=1, Weight=-2

 

As a general rule of thumb, neurons can NOT have negative firing rates (you can’t have a negative number of action potentials per second). So we will institute a rule that any activity lower than 0 is set equal to 0. This is a threshold, below which there is no activity. Future lessons will exploit our ability to choose what this is.

 

Example 3

Neuron 1 Activity=3, Weight=0.5

Neuron 2 Activity=2, Weight=-0.25

 

So the output (the activity of Neuron 3) is 1.

 

Example 4

Neuron 1 Activity=3, Weight=2

Neuron 2 Activity=0, Weight=-1

 

So the output (the activity of Neuron 3) is 6.

 

Part IV - Follow Up

 

To follow up on these principles, extrapolate these ideas to larger and larger networks. This can be done with or without diagrams (you can imagine that VERY large networks would be difficult to draw)

 

Q1. What would the Input vector look like if we had 4 input cells all connected to a fifth neuron, with the first having an activity of 3, the second an activity of 2, the third an activity of 3, and the fourth and activity of 1?

 

A1. Input = (3 2 3 1)

 

Q2. What would the Weight vector look like for these 4 input neurons, with the first 2 having an excitatory weight of 1, and the second 2 having an inhibitory weight of -1?

 

A2. Weight = (1 1 -1 -1)

 

Q3. What would the output (the activity of Neuron 5) be?

 

A3. Output = Input * Output