Context
I am implementing 2D Discrete Fourier Transform algorithm using Python with Numpy.
According to image processing theory in order to center image's transform, before performing the transform, each intensity f(x, y) of the image needs to be multiplied by (-1)^(x + y) where x and y are intensity's indices in 2D array representing the image.
What was tried
The obvious approach is to iterate over each intensity and its indices using two for loops.
Question
Is there a more elegant/faster solution using Python/Numpy matrix operations or should I stick with two for loops?
The idiomatic way would be:
y,x = np.ogrid[:m,:n]
prefactor = (-1)**(y+x)
Here m,n are, of course, the dimensions of your operand array.
Related
I implemented a finite differences algorithm to solve a PDE.
The grid is a structured 2D domain of size [Nx, Nz], solved Nt times.
I pre-allocate the object containing all solutions:
sol = zeros(Nx, Nz, Nt, 'single') ;
This becomes very easily too large and I get a 'out of memory' error.
Unfortunately sparse doesn't work for N-dimensional arrays.
For the sake of the question it's not important to know the values, it goes without saying that the RAM usage grows exponentially with decreasing the grid spacing and increasing the simulation time.
I am aware that I do not need to store each time instant for the purpose of the advancement of the solution. It would be sufficient to just store the previous two time steps. However, for post-processing reasons I need to access the solution at all time-steps (or at least at a submultiple of the total number).It might help to specify that, even after the solution, the grid remains predominantly populated by zeros.
Am I fighting a lost battle or is there a more efficient way to proceed (other type of objects, vectorization...)?
Thank you.
You could store the array in sparse, linear form; that is, a column vector with length equal to the product of dimensions:
sol = sparse([], [], [], Nx*Nz*Nt, 1); % sparse column vector containing zeros
Then, instead of indexing normally,
sol(x, z, t),
you need to translate the indices x, z, t into the corresponding linear index:
For scalar indices you use
sol(x + Nx*(z-1) + Nx*Nz*(t-1))
You can define a helper function for convenience:
ind = #(sol, x, y, t) sol(x + Nx*(z-1) + Nx*Nz*(t-1))
so the indexing becomes more readable:
ind(sol, x, z, t)
For general (array) indices you need to reshape the indices along different dimensions so that implicit expansion produces the appropriate linear index:
sol(reshape(x,[],1,1) + Nx*(reshape(z,1,[],1)-1) + Nx*Nz*(reshape(t,1,1,[])-1))
which of course could also be encapsulated into a function.
Check that the conversion to linear indexing works (general case, using non-sparse array to compare with normal indexing):
Nx = 15; Nz = 18; Nt = 11;
sol = randi(9, Nx, Nz, Nt);
x = [5 6; 7 8]; z = 7; t = [4 9 1];
isequal(sol(x, z, t), ...
sol(reshape(x,[],1,1) + Nx*(reshape(z,1,[],1)-1) + Nx*Nz*(reshape(t,1,1,[])-1)))
gives
ans =
logical
1
You can create a a cell array of sparse matrices to store the results. However computations can be performed on full matrices if working with a full matrix is faster than sparse matrix and convert the full matrix to sparse matrix and place it in the cell.
Most implementations of binomial coefficient computation using dynamic programming makes use of 2-dimensional arrays, as in these examples:
http://www.csl.mtu.edu/cs4321/www/Lectures/Lecture%2015%20-%20Dynamic%20Programming%20Binomial%20Coefficients.htm
http://www.geeksforgeeks.org/dynamic-programming-set-9-binomial-coefficient/
My question is, why not just compute it using a single dimensional array like this:
def C(n, r):
memo = list()
if (r > int(n/2)):
r = n - r
memo.append(1.0)
for i in range(1,r+1):
now = ((n-i+1)*memo[i-1])/i
memo.append(now)
return memo[r]
Basically using the recursive formula:
C(n,r) = ((n-r+1)/r) * C(n,r-1)
This has a O(r) complexity, while the 2 dimensional logic has a O(nr) complexity.
Am I missing something here?
If you want all of the values, then the 2D logic is certainly more efficient. The 2D logic may be more efficient for some parameters on some hardware that, e.g., lacks hardware multiply and divide. You have to be careful about integer overflow when multiplying before dividing, whereas the integer addition in the 2D recurrence is always fine. Other than that, no, the 1D recurrence is better.
Multiplication of sparse tensors with themselves or with dense tensors does not seem to work in TensorFlow. The following example
from __future__ import print_function
import tensorflow as tf
x = tf.constant([[1.0,2.0],
[3.0,4.0]])
y = tf.SparseTensor(indices=[[0,0],[1,1]], values=[1.0,1.0], shape=[2,2])
z = tf.matmul(x,y)
sess = tf.Session()
sess.run(tf.initialize_all_variables())
print(sess.run([x, y, z]))
fails with the error message
TypeError: Input 'b' of 'MatMul' Op has type string that does not match type
float32 of argument 'a'
Both tensors have values of type float32 as seen by evaluating them without the multiplication op. Multiplication of y with itself returns a similar error message. Multipication of x with itself works fine.
General-purpose multiplication for tf.SparseTensor is not currently implemented in TensorFlow. However, there are three partial solutions, and the right one to choose will depend on the characteristics of your data:
If you have a tf.SparseTensor and a tf.Tensor, you can use tf.sparse_tensor_dense_matmul() to multiply them. This is more efficient than the next approach if one of the tensors is too large to fit in memory when densified: the documentation has more guidance about how to decide between these two methods. Note that it accepts a tf.SparseTensor as the first argument, so to solve your exact problem you will need to use the adjoint_a and adjoint_b arguments, and transpose the result.
If you have two sparse tensors and need to multiply them, the simplest (if not the most performant) way is to convert them to dense and use tf.matmul:
a = tf.SparseTensor(...)
b = tf.SparseTensor(...)
c = tf.matmul(tf.sparse_tensor_to_dense(a, 0.0),
tf.sparse_tensor_to_dense(b, 0.0),
a_is_sparse=True, b_is_sparse=True)
Note that the optional a_is_sparse and b_is_sparse arguments mean that "a (or b) has a dense representation but a large number of its entries are zero", which triggers the use of a different multiplication algorithm.
For the special case of sparse vector by (potentially large and sharded) dense matrix multiplication, and the values in the vector are 0 or 1, the tf.nn.embedding_lookup operator may be more appropriate. This tutorial discusses when you might use embeddings and how to invoke the operator in more detail.
For the special case of sparse matrix by (potentially large and sharded) dense matrix, tf.nn.embedding_lookup_sparse() may be appropriate. This function accepts one or two tf.SparseTensor objects, with sp_ids representing the non-zero values, and the optional sp_weights representing their values (which otherwise default to one).
Recently, tf.sparse_tensor_dense_matmul(...) was added that allows multiplying a sparse matrix by a dense matrix.
https://www.tensorflow.org/versions/r0.9/api_docs/python/sparse_ops.html#sparse_tensor_dense_matmul
https://github.com/tensorflow/tensorflow/issues/1241
In TF2.4.1 you can use the methods in tensorflow.python.ops.linalg.sparse.sparse_csr_matrix_ops to multiply to arbitrary SparseTensor (I think up to 3 dimensions).
Something like the following should be used (in general you turn the sparse tensors into a CSR representation)
import tensorflow as tf
from tensorflow.python.ops.linalg.sparse import sparse_csr_matrix_ops
def tf_multiply(a: tf.SparseTensor, b: tf.SparseTensor):
a_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
a.indices, a.values, a.dense_shape
)
b_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
b.indices, b.values, b.dense_shape
)
c_sm = sparse_csr_matrix_ops.sparse_matrix_sparse_mat_mul(
a=a_sm, b=b_sm, type=tf.float32
)
c = sparse_csr_matrix_ops.csr_sparse_matrix_to_sparse_tensor(
c_sm, tf.float32
)
return tf.SparseTensor(
c.indices, c.values, dense_shape=c.dense_shape
)
For a while I was prefering scipy multiplication (via a py_function) because this multiplication in TF (2.3 and 2.4) was not performing as well as scipy. I tried again recently and, either I changed something on my code, or there was some fix in 2.4.1 that makes the TF sparse multiplication faster that using scipy, in both CPU and GPU.
It seems that
tf.sparse_matmul(
a,
b,
transpose_a=None,
transpose_b=None,
a_is_sparse=None,
b_is_sparse=None,
name=None
)
is not for multiplication of two SparseTensors.
a and b are Tensors not SparseTensors. And I have tried that, it is not working with SparseTensors.
tf.sparse_matmul is for multiplying two dense tensor not sparse type of data structure. That function is just an optimized version of tensor multiplication if the given matrix (or both two matrixes) have many zero value. Again it does not accept sparse tensor data type. It accepts dense tensor data type. It might fasten your calculations if values are mostly zero.
as far as I know there is no implementation of two sparse type tensor muliplication. but just one sparse one dense which is tf.sparse_tensor_dense_matmul(x, y) !
To make the answer more complete:
tf.sparse_matmul(
a,
b,
transpose_a=None,
transpose_b=None,
a_is_sparse=None,
b_is_sparse=None,
name=None
)
exists as well:
https://www.tensorflow.org/api_docs/python/tf/sparse_matmul
In Matlab I have an array v of length m, a matrix of order n and a function F that takes as an input a single matrix and outputs a number. Starting from v I would like to apply the function to the whole array of matrices whose i-th element consists of a matrix M_i whose entries are obtained by multiplicating all the entries of M by v_i. The output would be itself an array of length n.
As far as I can see there are two ways of achieving this:
Looping on all i=1:n, computing F on all the M_is and store all the corresponding values in an array
Defining a 3-array structure that contains all the matrices M_i and correspondingly extending the function F as to act on 3-arrays instead of matrices. However this entails overloading some matrix operators and functions (transpose, exponential, logarithm, square root, inverse etc...) as to formally handle a 3-array.
I have done the simpler option 1. It takes a long time to execute. Number 2 promises to be faster- However, I am not sure if this is the case, and I am not familiar with overloading operators on Matlab. In particular: how to extend a matrix operator to a 3-array in such a way that it performs the related function on all of its entries.
A for loop is probably no slower than vectorising this, especially for larger problems where memory starts to limit speed. Nevertheless, here are two ways of doing it:
M=rand(3,3,5) % I'm using a 3x3x5 matrix
v=1:5
F=#sum % This is the function
M2=bsxfun(#times,M,permute(v.',[2 3 1])) % Multiply the M(:,:,i) matrix by v(i)
R=arrayfun(#(t) F(M2(:,:,t)),(1:size(M,3)).','UniformOutput',false) % applies the function F to the resulting matrices
cell2mat(R) % Convert from cell array to matrix, since my F function returns row vectors
R2=zeros(size(M,3),size(M,1)); % Initialise R2
for t=1:size(M,3)
R2(t,:)=F(M(:,:,t)*v(t)); % Apply F to M(:,:,i)*v(i)
end
R2
You should do some testing to see which will be more efficient for your actual problem. The vectorised version should be faster for small problems, but use more memory, whereas the for loop will be slower for small problems but use less memory, and so could be faster on larger problems.
I have a MATLAB code which I have to convert to C language. According to the MATLAB code,
n1 = 11; x1 = randn(2,n1) + repmat([-1 1]’,1,n1);
w = [0 0]’;
here acccording to my calculation, the output of
w’*x1
will be a 1x3 matrix, that is a row vector as far as I know.
Then what will be the output of the following code,
z = exp(repmat(b,1,n1)+w’*x1);
where repmat() also creates a 1xn1 matrix (I'm not sure about this, figured it out from manual). My understanding is that addition of two 1x3 matrices will not give a scalar.
How is an exponential is taken here? Can exponential be applied to a matrix?
Like many MATLAB functions, the exp function operates element-wise when applied to arrays. For further details, please refer to the documentation.
Yes, you can apply exponentation to a matrix. From the Wikipedia article: Matrix exponential
Let X be an n×n real or complex matrix. The exponential of X, denoted by eX or exp(X), is the n×n matrix given by the power series
e^X = Sum(k=0, infinity) 1/k! * X^k
As #John Bartholomew pointed though, this is not what the exp() of Matlab does.