OpenMP Shared Array in C: Reduction vs Atomic Update Problem - arrays

Question

I am having trouble determining what is going wrong with this OpenMP task example. For context, y is a large shared array and rowIndex is not unique for each task. There may be multiple tasks trying to increment the value y[rowIndex].
My question is, does y needed to be guarded by a reduction clause, or is an atomic update enough? I am currently experiencing a crash with a much larger program and am wondering if I am botching something fundamental with this.
Of the examples I have seen, most array reductions are for very small arrays due to array copying for each thread, while most atomic updates are not used on array elements. There doesn't seem to be much content for updating a shared array one element at a time with potential for a race condition (also the context of task-based parallelism is rare).
#pragma omp parallel shared(y) // ??? reduction(+:y) ???
#pragma omp single
for(i = 0; i < n; i++)
{ 
sum = DoSmallWork_SingleThread(); 
rowIndex = getRowIndex_SingleThread();
#pragma omp task firstprivate(sum, rowIndex) 
{   
sum += DoLotsOfWork_TaskThread();
    // ??? #pragma omp atomic update ???
y[ rowIndex ] += sum; 
}
}

Answer 1

You have basically 3 solutions to avoid these types of race conditions, which you all mention. They all work distinctly:
atomic access, i.e. letting threads/tasks access the same array at the same moment but ensure a proper ordering of operations, this is done using a shared clause for the array with an atomic clause on the operation:
#pragma omp parallel
#pragma omp single
for(i = 0; i < n; i++)
{
sum = DoSmallWork_SingleThread();
rowIndex = getRowIndex_SingleThread();
#pragma omp task firstprivate(sum, rowIndex) shared(y)
{
increment = sum + DoLotsOfWork_TaskThread();
#pragma omp atomic
y[rowIndex] += increment;
}
}
privatisation, i.e. every task/thread has its own copy of the array and they’re then summed together later, which is what a reduction clause does:
#pragma omp parallel
#pragma omp single
#pragma omp taskgroup task_reduction (+:y[0:n-1])
for(int i = 0; i < n; i++)
{
int sum = DoSmallWork_SingleThread();
int rowIndex = getRowIndex_SingleThread();
#pragma omp task firstprivate(sum, rowIndex) in_reduction(+:y[0:n-1])
{
y[rowIndex] += sum + DoLotsOfWork_TaskThread();
}
}
exclusive access to the array, or section of the array, which is what task-dependencies are used for (you could implement using mutexes for a thread-based parallelism model for example):
#pragma omp parallel
#pragma omp single
for(i = 0; i < n; i++)
{
sum = DoSmallWork_SingleThread();
rowIndex = getRowIndex_SingleThread();
#pragma omp task firstprivate(sum, rowIndex) depend(inout:y[rowIndex])
{
y[rowIndex] += sum + DoLotsOfWork_TaskThread();
}
}
When should you use each of them?
An atomic access is a slower type of memory access that provides consistency guarantees, and can be especially slow in case of conflicts, that is when two (or more) threads try to modify the same value simultaneously.
It is preferable to use atomics only when updates to y are few and far between, and the probability of having conflicts is low.
Privatisation avoids that conflict issue by making copies of the arrays and joining them (in your case, adding them) together.
This incurs a memory overhead, and possibly impacts the cache, proportionally to the size of y.
Finally, providing task dependencies avoids the problem altogether by using scheduling, that is by only running simultaneously tasks that modify separate parts of the array. In general this is preferable when y is large and the operations modifying y are frequent in the task.
Your parallelism is however limited by the number of dependencies that you define, so in the example above, by the number of rows in y. If for example you only have 8 rows but 32 cores, this may not be the best approach because you will only use 25% of your computing power.
NB: This means that in the privatisation (aka reduction) case and especially the dependency case, you benefit from grouping together sections of array y, typically by having a task operate on a number of contiguous rows. You can then reduce (for reductions, respectively increase for dependencies) the size of the array chunk provided in the task’s clause.