I have a problem with the runtime of my code. The only module that is really slow is my for-loop over every matrix element in a (144, 208)-array.
I have to check every element if the condition is fulfilled and if so, i have to perform several actions like shifting another (144, 208)-array and add it to an existing one.
Is this not changeable or is my implementation way too beginner-like?
Here is my code:
# With this codeblock i am loading a specific image into python and
binarize it
g = Initialization()
b_init = g.initialize_grid(".\\geometries\\1.png")
# this function will modify the matrix m_sp, which i load in as csv.file
def expand_blockavg(x, h, w):
m, n = x.shape
return np.broadcast_to((x/float(h*w))[:, None, :, None], (m, h, n, w)).reshape(m*h, -1)
m_adapt = expand_blockavg(m_sp, 16, 16) / 256
# This is my actual calculation block
for index, x in np.ndenumerate(b_init):
if x == 1:
a = np.asarray(index)
y = np.subtract(a, index_default)
m_shift = shift(m_adapt, (y[0], y[1]), cval=0)
b = np.add(m_shift, b)
SO, the last block (calculation) is what takes so long. I know that the loop has to check 30k elements. But i thought that with numpy it will be faster.
Can some1 tell me if there's potential for optimization or do i have to live with the fact that it'll take so long.
thanks
Iteration in python is very slow compared to vectorized numpy operations.
An immediate optimization is to only iterate over the indices where the matrix is 1, rather than checking each index. Do this with:
indices = np.argwhere(b_init == 1)
for a in indices:
y = np.array(a) - index_default
m_shift = shift(m_adapt, y[:2], cval=0)
b += m_shift
Not knowing the details of shift it’s hard to say if you can vectorize that also. I replaced function calls with equivalent operations which should be faster; np.add etc. are mostly useful when the operation is being selected programmatically.
Related
I need to multiply parts of a column vector with a fixed row vector. I solved this problem using a for-loop. However, I am wondering if the performance can be improved as I have to perform this kind of computation around 50 million times. Here's my code so far:
multMat = 1:5;
mat = randi(5,10,1);
windowSize = 5;
vout = nan(10,1);
for r = windowSize : 10
vout(r) = multMat * mat( (r - windowSize + 1) : r);
end
I was thinking about uisng arrayfun. However, first I don't know how to adress the cell range (i.e. the previous five cells including the current cell), and second, I am not sure if arrayfun will be any faster than using the loop?
This sliding vector multiplication you're describing is an example of what is known as convolution. The following produces the same result as the loop in your example:
vout = [nan(windowSize-1,1);
conv(mat,flip(multMat),'valid')];
If your output doesn't really need the leading NaN values which aren't overwritten in your loop then the conv expression is sufficient without concatenating the NaN elements to it.
For sufficiently large vectors this is of course not guaranteed to be as fast as you'd like it to be, but MATLAB's built-in convolution implementation is likely to be pretty close to an optimal tool for the job.
I am getting an error using repmat. My Matlab version is 2017a. "Requested 3711450x2726 (75.4GB) array exceeds maximum array size..." First, some context.
I have an adjacency matrix of social network data call it D. D is 2725x2725 with 1s denoting a link between agents i and j and 0s otherwise. I have been provided a function and sub-functions for a network formation model. There are K regressors (x variables). The model requires forming a dyad-specific regressor matrix W that is W = 0.5N(N-1) x K. In my data, this is 3711450 x K. For a start, I select only one x variable so K=1.
In the main function, there are two steps. The first step calculates the joint MLE from a logit. I have a problem in the second step computation of the variance covariance matrix with array size. Inside this step, there is a calculation that creates a 3711450 x n (2725) matrix using repmat.
INFO = ((repmat((exp_Xbeta ./ (1+exp_Xbeta).^2),1,K) .* X)'*X);
exp_Xbeta is 3711450 x K and X is a sparse 3711450 x 2725 matrix with Bytes = 178171416 of class double. The error occurs at INFO.
I've tried converting X to a tall matrix but thus far no joy. I've tried adding sparse to the INFO line but again no joy. Anyone have any ideas short of going to a cluster or getting more ram? Could I somehow convert X from a sparse matrix to a full matrix inside a datastore and then call the datastore using tall? I have not been able to figure out how to do that if it is possible.
Once INFO is constructed as an array it will be used later in one of the sub-functions. So, it needs to be callable. In case you're curious, INFO is the second derivative matrix.
I have found that producing the INFO matrix all at once was too much for my memory constraints. I split up the steps, but still, repmat and subsequent steps were a problem. Now, I've turned to building up the INFO matrix one step at a time, while never holding more than exp_Xbeta, X, and two vectors in memory. Replacing the construction of INFO with
for i = 1:d
s1_i = step1(:,1).*X(:,i);
s1_i = s1_i';
for j = 1:d;
INFO(i,j) = s1_i*X(:,j);
end
clear s1_i;
end
has dropped the memory requirement, though its slow, and things seem to be working. For anyone interested, below is a little example illustrating the point.
clear all
N = 20
n = 0.5*N*(N-1)
exp_Xbeta = rand(n,1);
X = rand(n,N);
step1 = (exp_Xbeta ./ (1+exp_Xbeta).^2);
[c,d] = size(X);
INFO = zeros(d,d);
for i = 1:d
s1_i = step1(:,1).*X(:,i)
s1_i = s1_i'
for j = 1:d
INFO(i,j) = s1_i*X(:,j)
end
clear s1_i
end
K = 1
INFO2 = ((repmat((exp_Xbeta ./ (1+exp_Xbeta).^2),1,K) .* X)'*X);
% Methods produce equivalent matrices
INFO
INFO2
I am a complete beginner at Octave!
I am struggling to figure out the minimum value from the array mse_array. I keep getting an error saying: "subscript indices must be either positive integers less than 2^31 or logicals"
Could someone help me? Here's my code:
function randomLines(data1, data2)
mse_array = []
A = rand(2,10)*100
for i = columns(A)
max = A (1, i)
min = A (2, i)
m = (max-min)/0.16
p = [m min];
array_function_values = polyval(p,data2);
current_mse = mseFunction(data1,array_function_values);
mse_array(end + 1) = current_mse
endfor
min_value = min(mse_array)
endfunction
Q: How to find a minimum from an array in Octave?
octave has built-in self-describing documentation to use once in doubts
A: Always re-read both help min and doc min
octave-3.2.4.exe:1> help min
`min' is a function from the file C:\Octave\3.2.4_gcc-4.4.0\libexec\octave\3.2.4\oct\i686-pc-mingw32\max.oct
-- Loadable Function: min (X)
-- Loadable Function: min (X, Y)
-- Loadable Function: min (X, Y, DIM)
-- Loadable Function: [W, IW] = min (X)
For a vector argument, return the minimum value. For a matrix
argument, return the minimum value from each column, as a row
vector, or over the dimension DIM if defined. For two matrices
(or a matrix and scalar), return the pair-wise minimum. Thus,
min (min (X))
returns the smallest element of X, and
min (2:5, pi)
=> 2.0000 3.0000 3.1416 3.1416
compares each element of the range `2:5' with `pi', and returns a
row vector of the minimum values.
For complex arguments, the magnitude of the elements are used for
comparison.
If called with one input and two output arguments, `min' also
returns the first index of the minimum value(s). Thus,
[x, ix] = min ([1, 3, 0, 2, 0])
=> x = 0
ix = 3
See also: max, cummin, cummax
Re-defining (thus rendering un-usable) elements as reserved words just increases troubles. Try to use your own naming convention, that will not harm the rest of the system -- i.e. anArrayMinimumVALUE or aSmallestVALUE, rather than "killing" the Octave function min() by creating a variable named by accident as "also" min
There are several things wrong with your code. Here are a few to get you started.
The error is probably occurring at this line:
min_value = min(mse_array)
You're using mse_array as an index into min, but the values in mse_array are likely to valid array indices as the error states. (I'm guessing at this purely based on the name of the variable, mse_array, which appear to be from some form of mean squared error calculation.) It's important to check the line number and everything else in error messages.
Another issue in that your for loop only evaluates once. The columns function will return the scalar number of columns of A. Thus your your loop declaration is equivalent to:
for i = 10
In other words, the code will only look at the last column of A. You probably want to use:
for i = 1:columns(A)
Lastly, it's a bad idea to overwrite the built-in min and max functions in Octave with your own variable names. If you overwrite the built-in min function, you won't be able to use it anymore directly until you call clear min (alternatively, you can call builtin, but you should avoid that). Instead, just choose better variable names.
This is another step of my battle with multi-dimensional arrays in R, previous question is here :)
I have a big R array with the following dimensions:
> data = array(..., dim = c(x, y, N, value))
I'd like to perform a sort of bootstrap comparing the mean (see here for a discussion about it) obtained with:
> vmean = apply(data, c(1,2,3), mean)
With the mean obtained sampling the N values randomly with replacement, to explain better if data[1,1,,1] is equals to [v1 v2 v3 ... vN] I'd like to replace it with something like [v_k1 v_k2 v_k3 ... v_kN] with k values sampled with sample(N, N, replace = T).
Of course I want to AVOID a for loop. I've read this but I don't know how to perform an efficient indexing of this array avoiding a loop through x and y.
Any ideas?
UPDATE: the important thing here is that I want a different sample for each sample in the fourth (value) dimension, otherwise it would be simple to do something like:
> dataSample = data[,,sample(N, N, replace = T), ]
Also there's the compiler package which speeds up for loops by using a Just In Time compiler.
Adding thes lines at the top of your code enables the compiler for all code.
require("compiler")
compilePKGS(enable=T)
enableJIT(3)
setCompilerOptions(suppressAll=T)
I have the above loop running on the above variables:
A is a 2d array of size mxn.
mask is a 1d logical array of size 1xn
results is a 1d array of size 1xn
B is a vector of the form mx1
C is a mxm matrix, m is the same as the above.
Edit: expanded foo(x) into the function.
here is the code:
temp = (B.'*C*B);
for k = 1:n
x = A(:,k);
if(mask(k) == 1)
result(k) = (B.'*C*x)^2 / (temp*(x.'*C*x)); %returns scalar
end
end
take note, I am already successfully using the above code as a parfor loop instead of for. I was hoping you would be able to suggest some way to use meshgrid or the sort to yield better performance improvement. I don't think I have RAM problems so a solution can also be expensive memory wise.
Many thanks.
try this:
result=(B.'*C*A).^2./diag(temp*(A.'*C*A))'.*mask;
This vectorization via matrix multiplication will also make sure that result is a 1xn vector. In the code you provided there can be a case where the last elements in mask are zeros, in this case your code will truncate result to a smaller length, whereas, in the answer it'll keep these elements zero.
If your foo admits matrix input, you could do:
result = zeros(1,n); % preallocate result with zeros
mask = logical(mask); % make mask logical type
result(mask) = foo(A(mask),:); % compute foo for all selected columns