Multiplying a sparse matrix by an array without copying - arrays

Question

I have a 1d array (vector of size M), a pretty large one, and I definitely don't want to be copying it in memory. I also have a sparse matrix of window N (arbitrary size, basically all elements except the diagonal & N pseudo diagonals are zero).
I want to multiply this sparse matrix by the vector, without having to copy the vector in memory. What's the easiest and most efficient way of doing this? There has to be a neat solution, but I don't know the proper literature and I'm not educated enough to figure this out.
There is a solution for N=1 (where matrix is: a on the diagonal, and b on two closest pseudo diagonals). The solution looks smth like this (e.g., in python):
tmp2 = array[0]
i = 1
while (i < len(array) - 1):
tmp1 = b * (array[i - 1] + array[i + 1]) + a * array[i]
array[i - 1] = tmp2
i += 1
tmp2 = b * (array[i - 1] + array[i + 1]) + a * array[i]
array[i - 1] = tmp1
i += 1
But I can't manage to generalize this for an arbitrary N.
Notes: I absolutely don't want to be copying the size M vector in memory. However, using a temporary array of size 2N+1 is ok, since M >> N. I'm looking for an actual algorithm description, not a smart custom library that does the job.
Thanks in advance!

Answer 1

Consider the matrix
[
1 2 3 0 0 0
2 1 2 4 0 0
3 2 1 2 5 0
0 7 2 1 2 6
0 0 8 2 1 2
0 0 0 9 2 1
]
and the vector v [1,2,3,4,5,6]
For each row, below the involves coeff of v:
[1,2,3]
[1,2,3,4]
[1,2,3,4,5]
[ 2,3,4,5,6]
[ 3,4,5,6]
[ 4,5,6]
As you have noticed, you just need to keep track of a window of v.
That window is originally [1,2,3,4,5] (for i = 0, 1, 2)
Then you shift that window to the right every i (and eventually truncate it for the last rows not to be out of bounds of v...)
Now notice that when you shift to the right, you only need to know the next value from v, and as long as you have not dirtied that value (by writing to v) your new window is valid.
For row i, window is [i-n;i+n] and the coeff which will be modified is v[i]. For the next window, you need to know v[i+n+1] which has not been dirtied. So all good.
So algo be like
window = circularbuffer(2n+1) //you push to the right, and if length > 2n+1, you remove first elem
for i = 0; i<v.size()
v[i] = prod(row_i, window) // only for the row_i coeffs...
if i >= n && < M-3
window.push(v[i+n+1])
else if i>= M-3
window.shift() // just remove the first value
const N = 2
const M_SIZE = 10
function toString(M){
return M.map(x=>x.join(' ')).join('\n')
}
const { M, v } = (_ => {
const M = Array(M_SIZE).fill(0).map(x=>Array(M_SIZE).fill(0))
let z = 1
for(let i = 0; i<M_SIZE; ++i){
for(let j = -N; j<=N; ++j){
if(i+j >= 0 && i+j <M_SIZE){
M[i][i+j] = (z++ % (N*2))+1
}
}
}
const v = Array(M.length).fill(0).map((x,i)=>i)
return { M, v}
})()
function classic(M, v){
return M.map(r => r.reduce((acc, x, j) => acc + v[j]*x, 0))
}
function inplace(M, v){
// captn inefficiency
const circBuf = (init => {
let buf = init
return {
push (x) {
buf.push(x)
buf.shift()
},
shift() {
buf.shift()
},
at (i) { return buf[i] },
toString() {
return buf.join(' ')
}
}
})(v.slice(0, 2 * N + 1))
const sparseProd = (row, buf) => {
let s = 0
row.forEach((x, j) => s += x * buf.at(j))
return s
}
const sparseRows = M.map(r => r.filter(x => x !== 0))
sparseRows.forEach((row, i) => {
v[i] = sparseProd(row, circBuf)
if (i >= sparseRows.length - 3 ) {
circBuf.shift()
} else {
if (i >= N) {
circBuf.push(v[i + N + 1])
}
}
})
}
console.log('classic prod', classic(M, v))
inplace(M, v)
console.log('inplace prod', v)

Answer 2

So I ended up doing something like this. It seems like a generalization of what was done for the N=1 case.
In general, my weights are basically the non-zero components of the central row in my sparse matrix. I.e. if the matrix looks like this (as was noted in the comments, it's usually symmetric, but not necessarily):
| a b c 0 0 ... 0 0 0 0 0 |
| b a b c 0 ... 0 0 0 0 0 |
| c b a b c ... 0 0 0 0 0 |
| 0 c b a b ... 0 0 0 0 0 |
| 0 0 c b a ... 0 0 0 0 0 |
| ... ... ... |
| 0 0 0 0 0 ... a b c 0 0 |
| 0 0 0 0 0 ... b a b c 0 |
| 0 0 0 0 0 ... c b a b c |
| 0 0 0 0 0 ... 0 c b a b |
| 0 0 0 0 0 ... 0 0 c b a |
then the weights vector is simply [c, b, a, b, c] (i.e., N = 2).
So for the general case where N = ntimes I ended doing something like this:
def sparse_multiply(array, weights):
ntimes = (len(weights) - 1) / 2
# reduced dot product
def product(a_, i_, w_):
dot = 0.0
for k, j in enumerate(range(i_ - ntimes, i_ + ntimes + 1)):
if (j >= 0 and j < len(a_)):
dot += a_[j] * w_[k]
return dot
tmp = np.zeros(ntimes + 1)
for i in range(ntimes):
tmp[i] = array[i]
i = ntimes
while (i <= len(array)):
for t in range(-1, ntimes):
tmp[t] = product(array, i, w)
array[i - ntimes] = tmp[t + 1]
i += 1
return array
The only sacrifice you make, is the temporary array of size O(N), which is fine, because as I said, N << M.
Yes, yes, I know some of the operations (like the reduced dot product) could have been done with some python magic. But my point was to transfer this into old school C/Fortran, so that wouldn't help much.
Applications
Actually the application I was interested was to apply a gaussian filter: a_i = 0.5 * a_i + 0.25 * (a_{i-1} + a_{i+1}) to an array N times without having to do N passes and without having to copy the whole array.
So what you can do, is you can raise the sparse matrix of 0.5 on diagonals, and 0.25 on pseudo diagonals to the Nth power, and you will end up with the weights vector and a matrix that looks like the one I showed earlier (but with N nonzero pseudo diagonals). Then you can apply these weights to the array using the method above, so that you don't modify a_i before having to use it for other components, but at the same time get away without copying the whole array.
PS. Unfortunately I did not quite follow #grodzi's answer. A bit more explanation would certainly help.