Multithreading for backpropagation algorithm - c

Question

To speed up some neural network learning, I tried to do some multi-threading, since for a particular layer, the calculations for each neuron are independent from one another.
The original function I used is some basic backpropagation algorithm, for the evaluation of the deltas in the net :
δ = derivative * Σ (weight * previous δ)
void backpropagation (Autoencoder* AE)
{
int i, j, k;
for(i = AE->numLayer-2; i >= 0; i--)
{
for(j = 0; j < AE->layer[i].size; j++)
{
register double sum = 0.0;
for(k = 0; k < AE->layer[i+1].size; k++)
{
sum += AE->layer[i+1].neuron[k].weight[j] * AE->layer[i+1].neuron[k].delta;
}
AE->layer[i].neuron[j].delta = AE->layer[i].neuron[j].derivative * sum;
}
}
}
Autoencoder being the structure containing the neural net. It worked fine enough, if a bit slow, and it seems like a good idea to try this function first.
The modified functions are the following :
void backpropagationmultithread (Autoencoder* AE, unsigned int ncore, pthread_t* pth)
{
int i, j;
unsigned int neuronpercore, extra;
sem_t semaphore;
argThread* args[ncore];
for(i = AE->numLayer-2; i >= 0; i--)
{
neuronpercore = AE->layer[i].size / ncore;
extra = neuronpercore + (AE->layer[i].size % ncore);
sem_init(&semaphore, 0, -ncore);
for(j = 0; j < ncore; j++)
{
args[j] = malloc(sizeof(argThread));
args[j]->layer = i;
args[j]->AE = AE;
args[j]->sem = &semaphore;
args[j]->startat = neuronpercore * j;
args[j]->nneurons = (j!=ncore-1)?neuronpercore:extra;
pthread_create(&pth[j], NULL, backpropagationthread, (void*)args[j]);
}
sem_wait(&semaphore);
for(j = 0; j < ncore; j++)
{
pthread_cancel(pth[j]);
}
}
}
And the function for the new threads :
void* backpropagationthread (void* arg)
{
argThread* args = (argThread*) arg;
unsigned int j,k,layer = args->layer, start = args->startat, end = args->startat + args->nneurons;
Autoencoder* AE = args->AE;
for(j = start; j < end; j++)
{
register double sum = 0.0;
for(k = 0; k < AE->layer[layer+1].size; k++)
{
sum += AE->layer[layer+1].neuron[k].weight[j] * AE->layer[layer+1].neuron[k].delta;
}
AE->layer[layer].neuron[j].delta = AE->layer[layer].neuron[j].derivative * sum;
}
sem_post(args->sem);
free(arg);
return NULL;
}
argThread is just a little structure that contains all the arguments to be passed to the thread, ncore the number of CPU cores. The idea was to split up each layer into a roughly equal number of neurons to be treated individually by each thread (the last one with all the extra neurons if they are not multiples).
The new function does work to some degree, and much faster, but after a certain threshold does not converge anymore, where the old function did, and I cannot find why its behaviour would change. Am I missing some neurons or weights?

Answer 1

I implemented a multi-threaded backpropagation algorithm for Encog some time ago. I wrote this in Java, but when I implemented it in C, I made use of OpenMP, rather than pthreads. You can see my C implementation here.
https://github.com/encog/encog-c
I also wrote an article about my approach for performing backpropagation in multi-threaded.
You can see my article here.
http://www.heatonresearch.com/encog/mprop/compare.html
There are a few other questions about this on Stack Overflow, as well. Most seem to reference my algorithm.
Multithreaded backpropagation
How can I apply multithreading to the backpropagation neural network training?