Download Perceptron and Backpropagation - Program #3 | CPSC 420 and more Study notes Computer Science in PDF only on Docsity!
CPSC 420-500: Program 3, Perceptron and
Backpropagation
Yoonsuck Choe
Department of Computer Science
Texas A&M University
October 31, 2008
1 Overview
You will implement perceptron learning from scratch (see section 3 for details), and train it on AND, OR, and XOR functions. Then, you will take an existing backprop- agation code (see section 4), train it and test it under different conditions and report your findings. The same three boolean functions will be used for the backpropagation learning. For further details on perceptrons and backpropagation, see the lecture slides and also Hertz et al. (1991). Specific submission instruction will be given in section 5 and section 6.
2 Language and OS
You may use either C/C++, Java, Matlab (or Octave), or Lisp. The resulting code should be able to compile and run on the departmental unix host (unix.cs.tamu.edu). You may use a different language with a permission from the instructor, in which case you will be asked to do a demo in front of the TA.
To compile your programs other than Lisp (which you already know), see the following instructions.
- C program file: perceptron.c to compile: cc -o perceptron perceptron.c -lm to execute: ./perceptron
- C++ program file: perceptron.C to compile: c++ -o perceptron perceptron.C -lm to execute: ./perceptron
- Java program file: perceptron.java to compile: javac perceptron.java to execute: java perceptron
The full paths for the compilers are (on compute.cs.tamu.edu):
- /usr/bin/cc
- /opt/csw/gcc3/bin/g++
- /usr/jdk/latest/bin/java
- /usr/jdk/latest/bin/javac
3 Perceptron
Perceptron activation is defined as:
Output = step
i=
W [i] ∗ IN P [i]
where step(X) = 1 if X ≥ 0 and step(X) = 0 if X < 0 (see figure 1).
W[0] W[1] W[2]
−1 X Y
f(X,Y)
INP[0] INP[1] INP[2]
Figure 1: Perceptron. The input IN P [0] is the bias unit, fixed to − 1 , and has an associated weight W [0], which is the threshold. The two inputs X and Y are given and the output f (X, Y ) will be calculating (or attempting to calculate) a boolean function, one of OR, AND, or XOR.
Implement a perceptron with two input units (three, including the bias unit that has a fixed input value -1) and one output unit. Your program should take three inputs from the command line.
- maximum number of epochs to run (integer),
- learning rate parameter α value (double),
- function selection string (“and”, “or”, and “xor”).
$ ./bp conf/xor.conf | grep ERR
and to view the actual output values for the inputs:
$ ./bp conf/xor.conf | grep OUT
5 Assignment
This section will detail what you actually have to do and have to submit. All projects should be turned in using the csnet turnin.
5.1 Perceptron
Implement perceptron learning algorithm as detailed in section 3, and with the program conduct the following experiments, and submit the required material, along with the code and the README file as usual.
Experiments:
- Test AND, OR, and XOR for learning rates α = 0.001, and 0.0001. For each Boolean function, run the experiment with different initial random weights (use the random number generator function to do this) two times. Discard all runs that ended in 1 epoch (you will see several of these: why would they occur?). The total number of trial will thus be 2 learning rates × 2 random initial weights × 3 functions to learn = 12. Set the max epoch to 10000 for all trials.
- For each trial, report the following in the README file:
(a) initial weights, and show the plot of the decision boundary (the straight line defined by the weights). (b) final weights, and show the plot of the decision boundary. (c) number of epochs taken to complete training if successful (let us call this n), and (d) for each epoch, the sum of squared error for all four input patterns.
- Answer these questions in the README file:
(a) Do you think perceptron will be able to learn the boolean function f (X, Y ) = ¬(X ∨ Y )? What do you think is the role of the sign of the weights in this case? Think about the geometric interpretation in that case.
Table 1: Boolean Function f (X, Y ) = ¬(X ∨ Y ). X Y ¬(X ∨ Y ) 0 0 1 0 1 0 1 0 0 1 1 0
5.2 Backpropagation
With the provided code, conduct experiments on AND, OR, and XOR. Note that in the bp.cc code, learning rate α is a named eta, just in case you want to take a look inside the code.
Experiments:
- Test AND, OR, and XOR for learning rates α = 0.01 and 0.001, and plot the sum of squared error for each trial (a total of 6 trials). A total of 3 set of plots (for AND, OR, and XOR), with each set containing 2 curves (for two α’s) is re- quired. Save the plots in image files and name them and1.jpg, and2.jpg, or1.jpg, or2.jpg, xor1.jpg, xor1.jpg .... Use grep to ex- tract the error values (see below), and load that file (dump.txt) in a spread- sheet.
$ ./bp conf/xor.conf | grep ERR > dump.txt
- With learning rate α = 0. 01 , test AND, OR, and XOR, with 1, 2, and 4 hid- den units, Plot the sum of squared error for each trial. A total of 3 set of plots (for AND, OR, and XOR), with each set containing 3 curves (for three different number of hidden units) is required. Save the plots in image files and name them bp-and1.jpg, bp-and2.jpg, bp-and3.jpg, ....
- For all trials above, count the number of epochs until the end is reached, and measure the time taken using the timex unix command (add user time and sys time):
timex ./bp conf/xor.conf
Report the number of epochs and time spent for each trial in the README file. How many number of hidden units was the best in your opinion and why?
6 Submission Details
The due date is 12/4/2008, 11:59pm. Grading criteria will be similar to the previous programming assignments. You must submit the following:
