Related
Context
I am implementing 2D Discrete Fourier Transform algorithm using Python with Numpy.
According to image processing theory in order to center image's transform, before performing the transform, each intensity f(x, y) of the image needs to be multiplied by (-1)^(x + y) where x and y are intensity's indices in 2D array representing the image.
What was tried
The obvious approach is to iterate over each intensity and its indices using two for loops.
Question
Is there a more elegant/faster solution using Python/Numpy matrix operations or should I stick with two for loops?
The idiomatic way would be:
y,x = np.ogrid[:m,:n]
prefactor = (-1)**(y+x)
Here m,n are, of course, the dimensions of your operand array.
Multiplication of sparse tensors with themselves or with dense tensors does not seem to work in TensorFlow. The following example
from __future__ import print_function
import tensorflow as tf
x = tf.constant([[1.0,2.0],
[3.0,4.0]])
y = tf.SparseTensor(indices=[[0,0],[1,1]], values=[1.0,1.0], shape=[2,2])
z = tf.matmul(x,y)
sess = tf.Session()
sess.run(tf.initialize_all_variables())
print(sess.run([x, y, z]))
fails with the error message
TypeError: Input 'b' of 'MatMul' Op has type string that does not match type
float32 of argument 'a'
Both tensors have values of type float32 as seen by evaluating them without the multiplication op. Multiplication of y with itself returns a similar error message. Multipication of x with itself works fine.
General-purpose multiplication for tf.SparseTensor is not currently implemented in TensorFlow. However, there are three partial solutions, and the right one to choose will depend on the characteristics of your data:
If you have a tf.SparseTensor and a tf.Tensor, you can use tf.sparse_tensor_dense_matmul() to multiply them. This is more efficient than the next approach if one of the tensors is too large to fit in memory when densified: the documentation has more guidance about how to decide between these two methods. Note that it accepts a tf.SparseTensor as the first argument, so to solve your exact problem you will need to use the adjoint_a and adjoint_b arguments, and transpose the result.
If you have two sparse tensors and need to multiply them, the simplest (if not the most performant) way is to convert them to dense and use tf.matmul:
a = tf.SparseTensor(...)
b = tf.SparseTensor(...)
c = tf.matmul(tf.sparse_tensor_to_dense(a, 0.0),
tf.sparse_tensor_to_dense(b, 0.0),
a_is_sparse=True, b_is_sparse=True)
Note that the optional a_is_sparse and b_is_sparse arguments mean that "a (or b) has a dense representation but a large number of its entries are zero", which triggers the use of a different multiplication algorithm.
For the special case of sparse vector by (potentially large and sharded) dense matrix multiplication, and the values in the vector are 0 or 1, the tf.nn.embedding_lookup operator may be more appropriate. This tutorial discusses when you might use embeddings and how to invoke the operator in more detail.
For the special case of sparse matrix by (potentially large and sharded) dense matrix, tf.nn.embedding_lookup_sparse() may be appropriate. This function accepts one or two tf.SparseTensor objects, with sp_ids representing the non-zero values, and the optional sp_weights representing their values (which otherwise default to one).
Recently, tf.sparse_tensor_dense_matmul(...) was added that allows multiplying a sparse matrix by a dense matrix.
https://www.tensorflow.org/versions/r0.9/api_docs/python/sparse_ops.html#sparse_tensor_dense_matmul
https://github.com/tensorflow/tensorflow/issues/1241
In TF2.4.1 you can use the methods in tensorflow.python.ops.linalg.sparse.sparse_csr_matrix_ops to multiply to arbitrary SparseTensor (I think up to 3 dimensions).
Something like the following should be used (in general you turn the sparse tensors into a CSR representation)
import tensorflow as tf
from tensorflow.python.ops.linalg.sparse import sparse_csr_matrix_ops
def tf_multiply(a: tf.SparseTensor, b: tf.SparseTensor):
a_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
a.indices, a.values, a.dense_shape
)
b_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
b.indices, b.values, b.dense_shape
)
c_sm = sparse_csr_matrix_ops.sparse_matrix_sparse_mat_mul(
a=a_sm, b=b_sm, type=tf.float32
)
c = sparse_csr_matrix_ops.csr_sparse_matrix_to_sparse_tensor(
c_sm, tf.float32
)
return tf.SparseTensor(
c.indices, c.values, dense_shape=c.dense_shape
)
For a while I was prefering scipy multiplication (via a py_function) because this multiplication in TF (2.3 and 2.4) was not performing as well as scipy. I tried again recently and, either I changed something on my code, or there was some fix in 2.4.1 that makes the TF sparse multiplication faster that using scipy, in both CPU and GPU.
It seems that
tf.sparse_matmul(
a,
b,
transpose_a=None,
transpose_b=None,
a_is_sparse=None,
b_is_sparse=None,
name=None
)
is not for multiplication of two SparseTensors.
a and b are Tensors not SparseTensors. And I have tried that, it is not working with SparseTensors.
tf.sparse_matmul is for multiplying two dense tensor not sparse type of data structure. That function is just an optimized version of tensor multiplication if the given matrix (or both two matrixes) have many zero value. Again it does not accept sparse tensor data type. It accepts dense tensor data type. It might fasten your calculations if values are mostly zero.
as far as I know there is no implementation of two sparse type tensor muliplication. but just one sparse one dense which is tf.sparse_tensor_dense_matmul(x, y) !
To make the answer more complete:
tf.sparse_matmul(
a,
b,
transpose_a=None,
transpose_b=None,
a_is_sparse=None,
b_is_sparse=None,
name=None
)
exists as well:
https://www.tensorflow.org/api_docs/python/tf/sparse_matmul
I have a 4D matrix of size, let's say, 100x200x3x10 where 100x200 is the size of one image, 3 is the number of channels (RGB images) and 10 is the number of images.
I want to compute the inner product of each RGB vector in each image with itself. The resulting image should be of size 100x200x10. How can I efficiently compute these products, possibly without the use of loops?
Thanks.
If you call your matrix M, this should work:
squeeze(dot(M,M,3))
The squeeze is because matlab gives a 100x200x1x10 matrix as the result, and squeeze knocks out the redundant dimension.
I am writing a function to take the Mahalanobis distance between two vectors. I understand that this is achieved using the equation a'*C^-1*b, where a and b are vectors and C is the covariance matrix. My question is, is there an efficient way to find the inverse of the matrix without using Gauss-Jordan elimination, or is there no way around this? I'm looking for a way to do this myself, not with any predefined functions.
I know that C is a Hermitian, positive definite matrix, so is there any way that I can algorithmically take advantage of this fact? Or is there some clever way compute the Mahalanobis distance without calculating the inverse of the covariance at all? Any help would be appreciated.
***Edit: The Mahalanobis distance equation above is incorrect. It should be
x'*C^-1*x where x = (b-a), and b and a are the two vectors whose distance we are trying to find (thanks LRPurser). The solution posited in the selected answer is therefore as follows:
d=x'*b, where b = C^-1*x
C*b = x, so solve for b using LU factorization or LDL' factorization.
You can (and should!) use LU decomposition instead of explicitly inverting the matrix: solving C x = b using a decomposition has better numeric properties than computing C^-1 and multiplying the vector b.
Since your matrix is symmetric, an LU decomposition is effectively equivalent to an LDL* decomposition, which is what you should actually use in your case. Since your matrix is also positive-definite, you should be able to perform this decomposition without pivoting.
Edit: note that, for this application, you don't need to solve the full C x = b problem.
Instead, given C = L D L* and difference vector v = a-b, solve L* y = v for y (which is half as much work as the full LU solver).
Then, y^t D^-1 y = v^t C^-1 v can be computed in linear time.
First Mahalanobis Distance (MD) is the normed distance with respect to uncertainty in the measurement of two vectors. When C=Indentity matrix, MD reduces to the Euclidean distance and thus the product reduces to the vector norm. Also MD is always positive definite or greater than zero for all non-zero vectors. By your formulation with the appropriate choice of the vectors a and b, a*C^-1*b can be less than zero. Hopefully the difference of vectors you are looking for is x=(a-b) which makes the calculation x^t*C^-1*x where x^t is the transpose of the vector x. Also note that MD=sqrt(x^t*C^-1*x)
Since your matrix is symmetric and positive definite then you can utilize the Cholesky decomposition (MatLab-chol) which uses half of the operations as LU and is numerically more stable. chol(C)=L where C=L*L^t where L is a lower triangular matrix and L^t is the transpose of L which make it upper triangular. Your algorithm should go something like this
(Matlab)
x=a-b;
L=chol(C);
z=L\x;
MD=z'*z;
MD=sqrt(MD);
# Mahalanobis Distance Matrix
import numpy as np
from scipy.spatial import distance
from scipy.spatial.distance import mahalanobis
from scipy.spatial.distance import pdist
from scipy.spatial.distance import squareform
# Example
Data = np.array([[1,2],[3,2],[4,3]])
Cov = np.cov(np.transpose([[1,2],[3,2],[4,3]]))
invCov = np.linalg.inv(Cov)
Y = pdist(Data, 'mahalanobis', invCov)
MD = squareform(Y)
I have a MATLAB code which I have to convert to C language. According to the MATLAB code,
n1 = 11; x1 = randn(2,n1) + repmat([-1 1]’,1,n1);
w = [0 0]’;
here acccording to my calculation, the output of
w’*x1
will be a 1x3 matrix, that is a row vector as far as I know.
Then what will be the output of the following code,
z = exp(repmat(b,1,n1)+w’*x1);
where repmat() also creates a 1xn1 matrix (I'm not sure about this, figured it out from manual). My understanding is that addition of two 1x3 matrices will not give a scalar.
How is an exponential is taken here? Can exponential be applied to a matrix?
Like many MATLAB functions, the exp function operates element-wise when applied to arrays. For further details, please refer to the documentation.
Yes, you can apply exponentation to a matrix. From the Wikipedia article: Matrix exponential
Let X be an n×n real or complex matrix. The exponential of X, denoted by eX or exp(X), is the n×n matrix given by the power series
e^X = Sum(k=0, infinity) 1/k! * X^k
As #John Bartholomew pointed though, this is not what the exp() of Matlab does.