Speeding up reading and writing to a 2D Array - c

Question

I have a grid where each element holds 9 values. Every time-step grabs some values from neighbouring elements, does some trivial calculations, and then writes back new values to the same addresses.
On my machine this program runs in about 3 minutes. However, an alternative program that simply reads values from neighbouring elements, and then writes back the values (i.e. the below program just without intermediate calculations) runs in only 50 seconds.
Assuming the intermediate calculations don't take very long to compute (I may be wrong about that), how can I speed up the former program to achieve a similar performance to the latter? The issue is most likely to do with caching, but all changes I've attempted have either had no affect on the performance, or made the performance worse.
What I've attempted so far
Swapping the grid array from a structure of arrays (grid[9][256*256]) to a array of structures (grid[256*256][9]) seemed to have negligible impact on performance.
Hoisting the a[9] array to be outside the loop also didn't seem to affect the performance.
I've run the code through a profiler which tells me that cpu performance is poor whenever I access elements from the grid.
Simplified program:
#include <stdio.h>
#include <stdlib.h>
int main() {
double **grid = (double**)malloc(9*sizeof(double*));
for(int i = 0; i < 9; i++)
grid[i] = (double*)malloc(256*256*sizeof(double));
// double **grid = (double**)malloc(256*256*sizeof(double*));
// for(int i = 0; i < 256*256; i++)
// grid[i] = (double*)malloc(9*sizeof(double));
double res = 0.0;
for (int tt = 0; tt < 80000; tt++) {
for (int ii = 0; ii < 256; ii++) {
for (int jj = 0; jj < 256; jj++) {
int up = (ii + 1) % 256;
int rt = (jj + 1) % 256;
int dn = (ii == 0) ? 255 : (ii - 1);
int lf = (jj == 0) ? 255 : (jj - 1);
double sum = grid[0][ii*256 + jj] + grid[1][ii*256 + lf]
+ grid[2][dn*256 + jj] + grid[3][ii*256 + rt]
+ grid[4][up*256 + jj] + grid[5][dn*256 + lf]
+ grid[6][dn*256 + rt] + grid[7][up*256 + rt]
+ grid[8][up*256 + lf];
double odd = ( grid[1][ii*256 + jj] + grid[3][up*256 + lf]
+ grid[5][dn*256 + rt] + grid[7][up*256 + rt]
) / sum;
double even = ( grid[0][ii*256 + jj] + grid[2][up*256 + lf]
+ grid[4][dn*256 + rt] + grid[6][dn*256 + lf]
+ grid[8][ii*256 + lf]
) / sum;
double hypot = odd*odd + even*even;
double a[9];
a[1] = ( odd ) * hypot;
a[2] = ( even ) * hypot;
a[3] = ( - odd ) * hypot;
a[4] = ( - even ) * hypot;
a[5] = ( odd + even ) * hypot;
a[6] = ( - odd + even ) * hypot;
a[7] = ( - odd - even ) * hypot;
a[8] = ( odd - even ) * hypot;
sum = 0.0;
sum += ( grid[0][ii*256 + jj] = hypot * grid[0][ii*256 + jj] );
sum += ( grid[1][ii*256 + lf] = a[3] * grid[3][ii*256 + rt] );
sum += ( grid[2][dn*256 + jj] = a[4] * grid[4][up*256 + jj] );
sum += ( grid[3][ii*256 + rt] = a[1] * grid[1][ii*256 + lf] );
sum += ( grid[4][up*256 + jj] = a[2] * grid[2][dn*256 + jj] );
sum += ( grid[5][dn*256 + lf] = a[7] * grid[7][up*256 + rt] );
sum += ( grid[6][dn*256 + rt] = a[8] * grid[8][up*256 + lf] );
sum += ( grid[7][up*256 + rt] = a[5] * grid[5][dn*256 + lf] );
sum += ( grid[8][up*256 + lf] = a[6] * grid[6][dn*256 + rt] );
res += sum;
}
}
}
printf("%f", res);
return 0;
}
Vim command to swap the index ordering of all 2D arrays in the program:
:%s/\[\([^\]]\+\)\]\[\([^\]]\+\)]/\[\2\]\[\1\]/g