This is another step of my battle with multi-dimensional arrays in R, previous question is here :)
I have a big R array with the following dimensions:
> data = array(..., dim = c(x, y, N, value))
I'd like to perform a sort of bootstrap comparing the mean (see here for a discussion about it) obtained with:
> vmean = apply(data, c(1,2,3), mean)
With the mean obtained sampling the N values randomly with replacement, to explain better if data[1,1,,1] is equals to [v1 v2 v3 ... vN] I'd like to replace it with something like [v_k1 v_k2 v_k3 ... v_kN] with k values sampled with sample(N, N, replace = T).
Of course I want to AVOID a for loop. I've read this but I don't know how to perform an efficient indexing of this array avoiding a loop through x and y.
Any ideas?
UPDATE: the important thing here is that I want a different sample for each sample in the fourth (value) dimension, otherwise it would be simple to do something like:
> dataSample = data[,,sample(N, N, replace = T), ]
Also there's the compiler package which speeds up for loops by using a Just In Time compiler.
Adding thes lines at the top of your code enables the compiler for all code.
require("compiler")
compilePKGS(enable=T)
enableJIT(3)
setCompilerOptions(suppressAll=T)
Related
I have a problem with the runtime of my code. The only module that is really slow is my for-loop over every matrix element in a (144, 208)-array.
I have to check every element if the condition is fulfilled and if so, i have to perform several actions like shifting another (144, 208)-array and add it to an existing one.
Is this not changeable or is my implementation way too beginner-like?
Here is my code:
# With this codeblock i am loading a specific image into python and
binarize it
g = Initialization()
b_init = g.initialize_grid(".\\geometries\\1.png")
# this function will modify the matrix m_sp, which i load in as csv.file
def expand_blockavg(x, h, w):
m, n = x.shape
return np.broadcast_to((x/float(h*w))[:, None, :, None], (m, h, n, w)).reshape(m*h, -1)
m_adapt = expand_blockavg(m_sp, 16, 16) / 256
# This is my actual calculation block
for index, x in np.ndenumerate(b_init):
if x == 1:
a = np.asarray(index)
y = np.subtract(a, index_default)
m_shift = shift(m_adapt, (y[0], y[1]), cval=0)
b = np.add(m_shift, b)
SO, the last block (calculation) is what takes so long. I know that the loop has to check 30k elements. But i thought that with numpy it will be faster.
Can some1 tell me if there's potential for optimization or do i have to live with the fact that it'll take so long.
thanks
Iteration in python is very slow compared to vectorized numpy operations.
An immediate optimization is to only iterate over the indices where the matrix is 1, rather than checking each index. Do this with:
indices = np.argwhere(b_init == 1)
for a in indices:
y = np.array(a) - index_default
m_shift = shift(m_adapt, y[:2], cval=0)
b += m_shift
Not knowing the details of shift it’s hard to say if you can vectorize that also. I replaced function calls with equivalent operations which should be faster; np.add etc. are mostly useful when the operation is being selected programmatically.
I need to multiply parts of a column vector with a fixed row vector. I solved this problem using a for-loop. However, I am wondering if the performance can be improved as I have to perform this kind of computation around 50 million times. Here's my code so far:
multMat = 1:5;
mat = randi(5,10,1);
windowSize = 5;
vout = nan(10,1);
for r = windowSize : 10
vout(r) = multMat * mat( (r - windowSize + 1) : r);
end
I was thinking about uisng arrayfun. However, first I don't know how to adress the cell range (i.e. the previous five cells including the current cell), and second, I am not sure if arrayfun will be any faster than using the loop?
This sliding vector multiplication you're describing is an example of what is known as convolution. The following produces the same result as the loop in your example:
vout = [nan(windowSize-1,1);
conv(mat,flip(multMat),'valid')];
If your output doesn't really need the leading NaN values which aren't overwritten in your loop then the conv expression is sufficient without concatenating the NaN elements to it.
For sufficiently large vectors this is of course not guaranteed to be as fast as you'd like it to be, but MATLAB's built-in convolution implementation is likely to be pretty close to an optimal tool for the job.
Working in MATLAB R2017a. I'm trying to optimise a piece of code I'm working on. It uses arrays to store field values on a grid.
In order to create a specific function in a field array I originally used the straight forward method of two for loops iterating over all the array elements. But i know for loops are slow so since then I came back and tried my best to remove them. However I could only manage to remove one of the loops; leaving me with this:
for n = 1:1:K
%%% define initial pertubation
t=n*dt;
% create array for source Ez field.
xtemps = (1:Ng)*dX;
for k = 1:Ng
ztemp = k*dX;
Ez0(k,:) = THzamp * (1/(1+exp(-(t-stepuppos)))) * exp(-((xtemps-...
THzstartx).^2)./(bx^2)) .* (t-((ztemp-THzstartz)/vg))*exp(-((t-((ztemp-...
THzstartz)/vg))^2)/(bt^2));
end
The important bit here is the last 5 lines, but I figured the stuff before might be important for context. I've removed the for loop looping over the x coordinates. I want to vectorize the z/k for loop but I can't figure out how to distinguish between the dimensions with the array oporators.
Edit: THzamp, stepuppos, bx, bt, THzstartz, THzstartx are all just scalars, they control the function (Ez0) I'm trying to create. dX and t are also just scalars. Ez0 is a square array of size Ng.
What I want to achieve is to remove the for loop that loops over k, so that that the values of ztemp are defined in a vector (like xtemps already is), rather than individually in the loop. However, I don't know how I'd write the definition of Ez0 in that case.
First time posting here, if I'm doing it wrong let me know. If you need more info just ask.
It isn't clear if n is used in the other headers and as stated in the comments your sizes aren't properly defined so you'll have to ensure the sizes are correct.
However, you can give this vectorize code a try.
n = 1:K
%%% define initial pertubation
t=n*dt;
% create array for source Ez field.
xtemps = (1:Ng)*dX;
for k = 1:Ng
ztemp = k*dX;
Ez0(k,:) = THzamp .* (1./(1+exp(-(t-stepuppos)))) .* exp(-((xtemps-...
THzstartx).^2)./(bx^2)) .* (t-((ztemp-THzstartz)/vg)).*exp(-((t-((ztemp-...
THzstartz)/vg)).^2)/(bt.^2));
end
So now t has the size K you'll need to ensure stepupposand (ztemp-THzstartz)/vg) have the same size K. Also you can take a look at vectors vs array operators here.
I have the above loop running on the above variables:
A is a 2d array of size mxn.
mask is a 1d logical array of size 1xn
results is a 1d array of size 1xn
B is a vector of the form mx1
C is a mxm matrix, m is the same as the above.
Edit: expanded foo(x) into the function.
here is the code:
temp = (B.'*C*B);
for k = 1:n
x = A(:,k);
if(mask(k) == 1)
result(k) = (B.'*C*x)^2 / (temp*(x.'*C*x)); %returns scalar
end
end
take note, I am already successfully using the above code as a parfor loop instead of for. I was hoping you would be able to suggest some way to use meshgrid or the sort to yield better performance improvement. I don't think I have RAM problems so a solution can also be expensive memory wise.
Many thanks.
try this:
result=(B.'*C*A).^2./diag(temp*(A.'*C*A))'.*mask;
This vectorization via matrix multiplication will also make sure that result is a 1xn vector. In the code you provided there can be a case where the last elements in mask are zeros, in this case your code will truncate result to a smaller length, whereas, in the answer it'll keep these elements zero.
If your foo admits matrix input, you could do:
result = zeros(1,n); % preallocate result with zeros
mask = logical(mask); % make mask logical type
result(mask) = foo(A(mask),:); % compute foo for all selected columns
Is there any way to "vector" assign an array of struct.
Currently I can
edges(1000000) = struct('weight',1.0); //This really does not assign the value, I checked on 2009A.
for i=1:1000000; edges(i).weight=1.0; end;
But that is slow, I want to do something more like
edges(:).weight=[rand(1000000,1)]; //with or without the square brackets.
Any ideas/suggestions to vectorize this assignment, so that it will be faster.
Thanks in advance.
This is much faster than deal or a loop (at least on my system):
N=10000;
edge(N) = struct('weight',1.0); % initialize the array
values = rand(1,N); % set the values as a vector
W = mat2cell(values, 1,ones(1,N)); % convert values to a cell
[edge(:).weight] = W{:};
Using curly braces on the right gives a comma separated value list of all the values in W (i.e. N outputs) and using square braces on the right assigns those N outputs to the N values in edge(:).weight.
You can try using the Matlab function deal, but I found it requires to tweak the input a little (using this question: In Matlab, for a multiple input function, how to use a single input as multiple inputs?), maybe there is something simpler.
n=100000;
edges(n)=struct('weight',1.0);
m=mat2cell(rand(n,1),ones(n,1),1);
[edges(:).weight]=deal(m{:});
Also I found that this is not nearly as fast as the for loop on my computer (~0.35s for deal versus ~0.05s for the loop) presumably because of the call to mat2cell. The difference in speed is reduced if you use this more than once but it stays in favor of the for loop.
You could simply write:
edges = struct('weight', num2cell(rand(1000000,1)));
Is there something requiring you to particularly use a struct in this way?
Consider replacing your array of structs with simply a separate array for each member of the struct.
weights = rand(1, 1000);
If you have a struct member which is an array, you can make an extra dimension:
matrices = rand(3, 3, 1000);
If you just want to keep things neat, you could put these arrays into a struct:
edges.weights = weights;
edges.matrices = matrices;
But if you need to keep an array of structs, I think you can do
[edges.weight] = rand(1, 1000);
The reason that the structs in your example don't get initialized properly is that the syntax you're using only addresses the very last element in the struct array. For a nonexistent array, the rest of them get implicitly filled in with structs that have the default value [] in all their fields.
To make this behavior clear, try doing a short array with clear edges; edges(1:3) = struct('weight',1.0) and looking at each of edges(1), edges(2), and edges(3). The edges(3) element has 1.0 in its weight like you want; the others have [].
The syntax for efficiently initializing an array of structs is one of these.
% Using repmat and full assignment
edges = repmat(struct('weight', 1.0), [1 1000]);
% Using indexing
% NOTE: Only correct if variable is uninitialized!!!
edges(1:1000) = struct('weight', 1.0); % QUESTIONABLE
Note the 1:1000 instead of just 1000 when indexing in to the uninitialized edges array.
There's a problem with the edges(1:1000) form: if edges is already initialized, this syntax will just update the values of selected elements. If edges has more than 1000 elements, the others will be left unchanged, and your code will be buggy. Or if edges is a different type, you could get an error or weird behavior depending on its existing datatype. To be safe, you need to do clear edges before initializing using the indexing syntax. So it's better to just do full assignment with the repmat form.
BUT: Regardless of how you initialize it, an array-of-structs like this is always going to be inherently slow to work with for larger data sets. You can't do real "vectorized" operations on it because your primitive arrays are all broken up in to separate mxArrays inside each struct element. That includes the field assignment in your question – it is not possible to vectorize that. Instead, you should switch a struct-of-arrays like Brian L's answer suggests.
You can use a reverse struct and then do all operations without any errors
like this
x.E(1)=1;
x.E(2)=3;
x.E(2)=8;
x.E(3)=5;
and then the operation like the following
x.E
ans =
3 8 5
or like this
x.E(1:2)=2
x =
E: [2 2 5]
or maybe this
x.E(1:3)=[2,3,4]*5
x =
E: [10 15 20]
It is really faster than for_loop and you do not need other big functions to slow your program.