Neural weak classifiers for language learning

We describe a neural implementation of the combination of weak classifiers (CWC) algorithm [Ji and Ma, 1997] which is able to learn the one-step-look-ahead task where the input is natural language sentences. The one-step-look-ahead task is more usually implemented with Simple Recurrent Networks [Elman, 1990] whose architecture typically consists of comparitively few neurons but learning requires many thousands of presentations of the training data. Our implementation includes a four layered architecture which consists of (1) an input layer having one neuron for each category, (2) an internally recurrent layer which captures the dynamics of the temporal input, (3) a large hidden layer of weakly classifying perceptrons and (4) a winner-take-all output layer having the same number of neurons as the input.