When I try to do the math expression from the following code the matrix values are not consistent, how can I fix that?
#pragma omp parallel num_threads (NUM_THREADS)
{
#pragma omp for
for(int i = 1; i < qtdPassos; i++)
{
#pragma omp critical
matriz[i][0] = matriz[i-1][0]; /
for (int j = 1; j < qtdElementos-1; j++)
{
matriz[i][j] = (matriz[i-1][j-1] + (2 * matriz[i-1][j]) + matriz[i-1][j+1]) / 4; // Xi(t+1) = [Xi-1 ^ (t) + 2 * Xi ^ (t)+ Xi+1 ^ (t)] / 4
}
matriz[i][qtdElementos-1] = matriz[i-1][qtdElementos-1];
}
}
The problem comes from a race condition which is due to a loop carried dependency. The encompassing loop cannot be parallelised (nor the inner loop) since loop iterations matriz read/write the current and previous row. The same applies for the column.
Note that OpenMP does not check if the loop can be parallelized (in fact, it theoretically cannot in general). It is your responsibility to check that. Additionally, note that using a critical section for the whole iteration serializes the execution defeating the purpose of a parallel loop (in fact, it will be slower due to the overhead of the critical section). Note also that #pragma omp critical only applies on the next statement. Protecting the line matriz[i][0] = matriz[i-1][0]; is not enough to avoid the race condition.
I do not think this current code can be (efficiently) parallelised. That being said, if your goal is to implement a 1D/2D stencil, then you can use a double buffering technique (ie. write in a 2D array that is different from the input array). A similar logic can be applied for 1D stencil repeated multiple times (which is apparently what you want to do). Note that the results will be different in that case. For the 1D stencil case, this double buffering strategy can fix the dependency issue and enable you to parallelize the inner-loop. For the 2D stencil case, the two nested loops can be parallelized.
Related
I have a triple nested loop that I would like to parallelize, however, I am getting a data race issue. I am pretty sure that I need to use a reduction somehow, but I don't quite know how.
This is the loop in question:
#pragma omp parallel for simd collapse(3)
for (uint64 u = 0; u < nu; ++u) {
for (uint64 e = 0; e < ne; ++e) {
for (uint64 v = 0; v < nv; ++v) {
uAT[u][e] += _uT[u][e][v] * wA[e][v];
}
}
}
Could someone explain to me, why this causes a data race? I would really like to understand this, so that I don't run into these issues in the future. Also, can this loop be parallelized at all? If so, how?
EDIT: How do I know that there is a data race?
What this loop should accomplish (and it does in serial) is to compute the element average of a function in a Discontinuous Galerkin frame work. When I run the code a bunch of times, sometimes I get different results, eventhough it should always produce the same results. The resulting wrong values are always smaller than what the should be, which is why I assume that some values are not being added. Maybe this picture explains it better: The average in the third cell is obviously wrong (too small).
Original answer concerning multi-threading, not SIMD
By using the collapse(3) clause, the whole iteration space of nu * ne * nv iterations is distributed among threads. This means that for any combination of u and e, the v loop could be distributed among multiple threads as well. These can then access the same element uAT[u][e] in parallel, which is the data race.
As long as nu * ne is much bigger than the number of CPU cores that you work on, the easiest solution is to instead use collapse(2). As there can be inefficient implementations of collapse (See here), you might even want to leave it away completely depending on nu being big enough.
If you really need the parallelism from all three loops to efficiently use your hardware, you can use either reduction(+: uAT[0:nu][0:ne]) (add it into your existing pragma) or put a #pragma omp atomic update in front of uAT[u][e] += ...;. Which of these options is faster should be benchmarked. The reduction clause will use a lot more memory due to every thread getting its own private copy of the whole memory addressed through uAT. On the other hand the atomic update could in the worst case sequentialize part of your parallel work and give worse performance than using collapse(2).
Edit 1: SIMD
I just saw that you are also concerned with SIMD instead of just multi-threading. My original answer is about the latter. For SIMD I would first take a look at your compilers output without any OpenMP directives (e.g. on Compiler Explorer). If your compiler (with optimization and information on the target processor architecture) does already use SIMD instructions, you might not need to use OpenMP in the first place.
If you still want to use OpenMP for vectorizing your loops, leaving away the collapse clause and putting the pragma in front of the second loop would be my first ansatz. Leaving it in front of the first loop with collapse(2) might also work. Even a reduction should work but seems unnecessary complex in this context, as I would expect there to be enough parallelism in nu or nu * ne to fill your SIMD lanes. I have never used array reduction like described below in the SIMD context, so I'm not quite sure what it would do (i.e. allocating an array for each SIMD lane doesn't seem realistic), or if it even is part of the OpenMP standard (depends on the version of the standard, see here).
The way your code is written right now, the description of the data race for multi-threading technically still applies, I think. I'm not sure though if your code causes (efficient) vectorization at all, so the compiled binary might not have the data race (it might not be vectorized at all).
Edit 2: Threading + SIMD
I benchmarked a few versions of this loop nest for nu = 4, nv = 128 and ne between 1024 and 524288 (2^19). My benchmarking was done using google-benchmark (i.e. C++ instead of C, shouldn't matter here, I would think) with gcc 11.3 (always using -march=native -mtune=native, i.e. for portable performance this might not be helpful). I made sure to initialize all data in parallel (first touch policy) to avoid bad NUMA effects. I slightly modified the problem/code to use contiguous memory and do the multi-dimensional indexing manually. As OP's code didn't show the data type, I used float.
The three best versions I will share here are all three performing relatively similar. So depending on the hardware architecture there might be differences in which one is the best. For me the version inspired by Peter Cordes comments below this answer performed best:
#pragma omp parallel for
for (uint64_t e = 0UL; e < ne; ++e) {
// In the future we will get `#pragma omp unroll partial(4)`.
// The unrolling might actually not be necessary (or even a pessimization).
// So maybe leave it to the compiler.
#pragma GCC unroll 4
for (uint64_t u = 0UL; u < nu; ++u) {
float temp = 0.0f;
#pragma omp simd reduction(+ : temp)
for (uint64_t v = 0UL; v < nv; ++v) {
temp += uT[(u * ne + e) * nv + v] * wA[e * nv + v];
}
uAT[u * ne + e] += temp;
}
}
One could also have a float temp[nu]; and put the unrolled u loop inside the v loop to get even nearer to Peter Cordes' description, but then on would have to use an array reduction as described above. These array reductions consistently caused stack overflows for me, so I settled on this version which depends on nv being small enough that wA can still be cached between u iterations.
This second version just differs in the u loop staying on the outside:
#pragma omp parallel
for (uint64_t u = 0UL; u < nu; ++u) {
#pragma omp for
for (uint64_t e = 0UL; e < ne; ++e) {
float temp = 0.0f;
#pragma omp simd reduction(+ : temp)
for (uint64_t v = 0UL; v < nv; ++v) {
temp += uT[(u * ne + e) * nv + v] * wA[e * nv + v];
}
uAT[u * ne + e] += temp;
}
}
The third version uses collapse(2):
#pragma omp parallel for collapse(2)
for (uint64_t u = 0UL; u < nu; ++u) {
for (uint64_t e = 0UL; e < ne; ++e) {
float temp = 0.0f;
#pragma omp simd reduction(+ : temp)
for (uint64_t v = 0UL; v < nv; ++v) {
temp += uT[(u * ne + e) * nv + v] * wA[e * nv + v];
}
uAT[u * ne + e] += temp;
}
}
TL;DR:
I think the most important points can be seen in all three versions:
#pragma omp parallel for simd is nice for big simple loops. If you have a loop nest, you probably should split the pragma up.
Use simd on a loop which accesses contiguous elements in contiguous iterations.
By respecting the first two points, you don't need the possibly expensive array reduction.
By using a temporary (i.e. register) for the reduction instead of writing back to memory, you make using OpenMP easier and probably have better performance in serial code as well. Due to floating point non-associativity, this optimization can't be done by most compilers without allowing it via e.g. -ffast-math (gcc).
Most compilers can not vectorize the reduction on their own for the same reason.
Details like Using collapse or the the order of the e and the u loop are smaller optimizations which are not as important as long as you provide enough parallelism. I.e. don't parallelize over just u if nu is small (as the question author wrote below this answer).
I want to effectively parallelize the following sum in C:
#pragma omp parallel for num_threads(nth)
for(int i = 0; i < l; ++i) pout[pg[i]] += px[i];
where px is a pointer to a double array x of size l containing some data, pg is a pointer to an integer array g of size l that assigns each data point in x to one of ng groups which occur in a random order, and pout is a pointer to a double array out of size ng which is initialized with zeros and contains the result of summing x over the grouping defined by g.
The code above works, but the performance is not optimal so I wonder if there is somewthing I can do in OpenMP (such as a reduction() clause) to improve the execution. The dimensions l and ng of the arrays, and the number of threads nth are available to me and fixed beforehand. I cannot directly access the arrays, only the pointers are passed to a function which does the parallel sum.
Your code has a data race (at line pout[pg[i]] += ...), you should fix it first, then worry about its performance.
if ng is not too big and you use OpenMP 4.5+, the most efficient solution is using reduction: #pragma omp parallel for num_threads(nth) reduction(+:pout[:ng])
if ng is too big, most probably the best idea is to use a serial version of the program on PCs. Note that your code will be correct by adding #pragma omp atomic before pout[pg[i]] += .., but its performance is questionable.
From your description it sounds like you have a many-to-few mapping. That is a big problem for parallelism because you likely have write conflicts in the target array. Attempts to control with critical sections or locks will probably only slow down the code.
Unless it is prohibitive in memory, I would give each thread a private copy of pout and sum into that, then add those copies together. Now the reading of the source array can be nicely divided up between the threads. If the pout array is not too large, your speedup should be decent.
Here is the crucial bit of code:
#pragma omp parallel shared(sum,threadsum)
{
int thread = omp_get_thread_num(),
myfirst = thread*ngroups;
#pragma omp for
for ( int i=0; i<biglen; i++ )
threadsum[ myfirst+indexes[i] ] += 1;
#pragma omp for
for ( int igrp=0; igrp<ngroups; igrp++ )
for ( int t=0; t<nthreads; t++ )
sum[igrp] += threadsum[ t*ngroups+igrp ];
}
Now for the tricky bit. I'm using an index array of size 100M, but the number of groups is crucial. With 5000 groups I get good speedup, but with only 50, even though I've eliminated things like false sharing, I get pathetic or no speedup. This is not clear to me yet.
Final word: I also coded #Laci's solution of just using a reduction. Testing on 1M groups output: For 2-8 threads the reduction solution is actually faster, but for higher thread counts I win by almost a factor of 2 because the reduction solution repeatedly adds the whole array while I sum it just once, and then in parallel. For smaller numbers of groups the reduction is probably preferred overall.
So I was learning about the basics OpenMP in C and work-sharing constructs, particularly for loop. One of the most famous examples used in all the tutorials is of matrix multiplication but all of them just parallelize the outer loop or the two outer loops. I was wondering why we do not parallelize and collapse all the 3 loops (using atomic) as I have done here:
for(int i=0;i<100;i++){
//Initialize the arrays
for(int j=0;j<100;j++){
A[i][j] = i;
B[i][j] = j;
C[i][j] = 0;
}
}
//Starting the matrix multiplication
#pragma omp parallel num_threads(4)
{
#pragma omp for collapse(3)
for(int i=0;i<100;i++){
for(int j=0;j<100;j++){
for(int k=0;k<100;k++){
#pragma omp atomic
C[i][j] = C[i][j]+ (A[i][k]*B[k][j]);
}
}
}
}
Can you tell me what I am missing here or why is this not an inferior/superior solution?
Atomic operations are very costly on most architectures compared to non-atomic ones (see here to understand why or here for a more detailed analysis). This is especially true when many threads make concurrent accesses to the same shared memory area. To put it simply, one cause is that threads performing atomic operations cannot fully run in parallel without waiting others on most hardware due to implicit synchronizations and communications coming from the cache coherence protocol. Another source of slowdowns is the high-latency of atomic operations (again due to the cache hierarchy).
If you want to write code that scale well, you need to minimize synchronizations and communications (including atomic operations).
As a result, using collapse(2) is much better than a collapse(3). But this is not the only issue is your code. Indeed, to be efficient you must perform memory accesses continuously and keep data in caches as much as possible.
For example, swapping the loop iterating over i and the one iterating over k (that does not work with collapse(2)) is several times faster on most systems due to more contiguous memory accesses (about 8 times on my PC):
for(int i=0;i<100;i++){
//Initialize the arrays
for(int j=0;j<100;j++){
A[i][j] = i;
B[i][j] = j;
C[i][j] = 0;
}
}
//Starting the matrix multiplication
#pragma omp parallel num_threads(4)
{
#pragma omp for
for(int i=0;i<100;i++){
for(int k=0;k<100;k++){
for(int j=0;j<100;j++){
C[i][j] = C[i][j] + (A[i][k]*B[k][j]);
}
}
}
}
Writing fast matrix-multiplication code is not easy. Consider using BLAS libraries such as OpenBLAS, ATLAS, Eigen or Intel MKL rather than writing your own code if your goal is to use this in production code. Indeed, such libraries are very optimized and often scale well on many cores.
If your goal is to understand how to write efficient matrix-multiplication codes, a good starting point may be to read this tutorial.
Collapsing loops requires that you know what you are doing as it may result in very cache-unfriendly splits of the iteration space or introduce data dependencies depending on how the product of the loop counts relates to the number of threads.
Imagine the following constructed example, which is not that uncommon actually (the loop counts are small just to illustrate the point):
for (int i = 0; i < 7; i++)
for (int j = 0; j < 3; j++)
a[i] += b[i][j];
If you parallelise the outer loop, three threads get two iterations and one thread gets just one, but all of them do all the iterations of the inner loop:
---0-- ---1-- ---2-- -3- (thread number)
000111 222333 444555 666 (values of i)
012012 012012 012012 012 (values of j)
Each a[i] gets processed by one thread only. Smart compilers may implement the inner loop using register optimisation, accumulating the values in a register first and only assigning to a[i] at the very end, and it will run very fast.
If you collapse the two loops, you end up in a very different situation. Since there is a total of 7x3 = 21 iterations now, the default split will be (depending on the compiler and the OpenMP runtime, but most of them do this) five iterations per thread and one gets six iterations:
--0-- --1-- --2-- ---3-- (thread number)
00011 12223 33444 555666 (values of i)
01201 20120 12012 012012 (values of j)
As you can see, now a[1] is processed by both thread 0 and thread 1. Similarly, a[3] is processed by both thread 1 and thread 2. And there you have it - you just introduced a data dependency that wasn't there in the previous case, so now you have to use atomic in order to prevent data races. That price that you pay for synchronisation is way higher than doing one iteration more or less! In your case, if you only collapse the two outer loops, you won't need to use atomic at all (although, in your particular case, 4 divides 100 and even when collapsing all the loops together you don't need the atomic construct, but you need it in the general case).
Another issue is that after collapsing the loops, there is a single loop index and both i and j indices have to be reconstructed from this new index using division and modulo operations. For simple loop bodies like yours, the overhead of reconstructing the indices may be simply too high.
There are very few good reasons not to use a library for matrix-matrix multiplication, so as suggested already, please call BLAS instead of writing this yourself. Nonetheless, the questions you ask are not specific to matrix-matrix multiplication, so they deserve to be answered anyways.
There are a few things that can be improved here:
Use contiguous memory.
If K is the innermost loop, you are doing dot-products, which are harder to vectorize. The loop order IKJ will vectorize better, for example.
If you want to parallelize a dot product with OpenMP, use a reduction instead of many atomics.
I have illustrated each of these techniques independently below.
Contiguous memory
int n = 100;
double * C = malloc(n*n*sizeof(double));
for(int i=0;i<n;i++){
for(int j=0;j<n;j++){
C[i*n+j] = 0.0;
}
}
IKJ loop ordering
for(int i=0;i<100;i++){
for(int k=0;k<100;k++){
for(int j=0;j<100;j++){
C[i][j] = C[i][j]+ (A[i][k]*B[k][j]);
}
}
}
Parallel dot-product
double x = 0;
#pragma omp parallel for reduction(+:x)
for(int k=0;k<100;k++){
x += (A[i][k]*B[k][j]);
}
C[i][j] += x;
External resources
How to Write Fast Numerical Code:
A Small Introduction covers these topics in far more detail.
BLISlab is an excellent tutorial specific to matrix-matrix multiplication that will teach you how the experts write a BLAS library call.
I am having trouble determining what is going wrong with this OpenMP task example. For context, y is a large shared array and rowIndex is not unique for each task. There may be multiple tasks trying to increment the value y[rowIndex].
My question is, does y needed to be guarded by a reduction clause, or is an atomic update enough? I am currently experiencing a crash with a much larger program and am wondering if I am botching something fundamental with this.
Of the examples I have seen, most array reductions are for very small arrays due to array copying for each thread, while most atomic updates are not used on array elements. There doesn't seem to be much content for updating a shared array one element at a time with potential for a race condition (also the context of task-based parallelism is rare).
#pragma omp parallel shared(y) // ??? reduction(+:y) ???
#pragma omp single
for(i = 0; i < n; i++)
{
sum = DoSmallWork_SingleThread();
rowIndex = getRowIndex_SingleThread();
#pragma omp task firstprivate(sum, rowIndex)
{
sum += DoLotsOfWork_TaskThread();
// ??? #pragma omp atomic update ???
y[ rowIndex ] += sum;
}
}
You have basically 3 solutions to avoid these types of race conditions, which you all mention. They all work distinctly:
atomic access, i.e. letting threads/tasks access the same array at the same moment but ensure a proper ordering of operations, this is done using a shared clause for the array with an atomic clause on the operation:
#pragma omp parallel
#pragma omp single
for(i = 0; i < n; i++)
{
sum = DoSmallWork_SingleThread();
rowIndex = getRowIndex_SingleThread();
#pragma omp task firstprivate(sum, rowIndex) shared(y)
{
increment = sum + DoLotsOfWork_TaskThread();
#pragma omp atomic
y[rowIndex] += increment;
}
}
privatisation, i.e. every task/thread has its own copy of the array and they’re then summed together later, which is what a reduction clause does:
#pragma omp parallel
#pragma omp single
#pragma omp taskgroup task_reduction (+:y[0:n-1])
for(int i = 0; i < n; i++)
{
int sum = DoSmallWork_SingleThread();
int rowIndex = getRowIndex_SingleThread();
#pragma omp task firstprivate(sum, rowIndex) in_reduction(+:y[0:n-1])
{
y[rowIndex] += sum + DoLotsOfWork_TaskThread();
}
}
exclusive access to the array, or section of the array, which is what task-dependencies are used for (you could implement using mutexes for a thread-based parallelism model for example):
#pragma omp parallel
#pragma omp single
for(i = 0; i < n; i++)
{
sum = DoSmallWork_SingleThread();
rowIndex = getRowIndex_SingleThread();
#pragma omp task firstprivate(sum, rowIndex) depend(inout:y[rowIndex])
{
y[rowIndex] += sum + DoLotsOfWork_TaskThread();
}
}
When should you use each of them?
An atomic access is a slower type of memory access that provides consistency guarantees, and can be especially slow in case of conflicts, that is when two (or more) threads try to modify the same value simultaneously.
It is preferable to use atomics only when updates to y are few and far between, and the probability of having conflicts is low.
Privatisation avoids that conflict issue by making copies of the arrays and joining them (in your case, adding them) together.
This incurs a memory overhead, and possibly impacts the cache, proportionally to the size of y.
Finally, providing task dependencies avoids the problem altogether by using scheduling, that is by only running simultaneously tasks that modify separate parts of the array. In general this is preferable when y is large and the operations modifying y are frequent in the task.
Your parallelism is however limited by the number of dependencies that you define, so in the example above, by the number of rows in y. If for example you only have 8 rows but 32 cores, this may not be the best approach because you will only use 25% of your computing power.
NB: This means that in the privatisation (aka reduction) case and especially the dependency case, you benefit from grouping together sections of array y, typically by having a task operate on a number of contiguous rows. You can then reduce (for reductions, respectively increase for dependencies) the size of the array chunk provided in the task’s clause.
I'm trying to parallelize a code. My code looks like this -
#pragma omp parallel private(i,j,k)
#pragma omp parallel for shared(A)
for(k=0;k<100;<k++)
for(i=1;i<1024;<i++)
for(j=0;j<1024;<j++)
A[i][j+1]=<< some expression involving elements of A[i-1][j-1] >>
On executing this code I'm getting a different result from serial execution of the loops.
I'm unable to understand what I'm doing wrong.
I've also tried the collapse()
#pragma omp parallel private(i,j,k)
#pragma omp parallel for collapse(3) shared(A)
for(k=0;k<100;<k++)
for(i=1;i<1024;<i++)
for(j=0;j<1024;<j++)
A[i][j+1]=<< some expression involving elements of A[][] >>
Another thing I tried was having a #pragma omp parallel for before each loop instead of collapse().
The issue, as I think, is the data dependency. Any idea how to parallelize in case of data dependency?
If this is really your use case, just parallelize for the outer loop, k, this should largely suffice for the modest parallelism that you have on common architectures.
If you want more, you'd have to re-write your loops such that you have an inner part that doesn't have the dependency. In your example case this is relatively easy, you'd have to process by "diagonals" (outer loop, sequential) and then inside the diagonals you'd be independent.
for (size_t d=0; d<nDiag(100); ++d) {
size_t nPoints = somefunction(d);
#pragma omp parallel
for (size_t p=0; p<nPoints; ++p) {
size_t x = coX(p, d);
size_t y = coY(p, d);
... your real code ...
}
}
Part of this could be done automatically, but I don't think that such tools are already readily implemented in everydays OMP. This is an active line of research.
Also note the following
int is rarely a good idea for indices, in particular if you access matrices. If you have to compute the absolute position of an entry yourself (and you see that here you might be) this overflows easily. int usually is 32 bit wide and of these 32 you are even wasting one for the sign. In C, object sizes are computed with size_t, most of the times 64 bit wide and in any case the correct type chosen by your platform designer.
use local variables for loop indices and other temporaries, as you can see writing OMP pragmas becomes much easier, then. Locality is one key to parallelism. Help yourself and the compiler by expressing this correctly.
You're only parallelizing the outer 'k' for loop. Every parallel thread is executing the 'i' and 'j' loops, and they're all writing into the same 'A' result. Since they're all reading and writing the same slots in A, the final result will be non-deterministic.
It's not clear from your problem that any parallelism is possible, since each step seems to depend on every previous step.