Here is a snippet of code I have to convert a numpy array to c_float ctype array so I can pass it to some functions in C language:
arr = my_numpy_array
arr = arr/255.
arr = arr.flatten()
new_arr = (c_float*len(arr))()
new_arr[:] = arr
but since the last line is actually a for loop and we all know how notorious python is when it comes to for loops for a medium size image array it takes about 0.2 seconds!! so this one line is right now the bottle neck of my whole pipeline. I want to know if there is any faster way of doing it.
Update
Please note "to pass to a function in C" in the question. To be more specific I want to put a numpy array in IMAGE data structure and pass it to rgbgr_image function. You can find both here
The OP's answer makes 4 copies of the my_numpu_array, at least 3 of which should be unnecessary. Here's a version that avoids them:
# random array for demonstration
my_numpy_array = np.random.randint(0, 255, (10, 10))
# copy my_numpy_array to a float32 array
arr = my_numpy_array.astype(np.float32)
# divide in place
arr /= 255
# reshape should return a view, not a copy, unlike flatten
ctypes_arr = np.ctypeslib.as_ctypes(arr.reshape(-1))
In some circumstances, reshape will return a copy, but since arr is guaranteed to own it's own data, it should return a view here.
So I managed to do it in this weird way using numpy:
arr = my_numpu_array
arr = arr/255.
arr = arr.flatten()
arr_float32 = np.copy(arr).astype(np.float32)
new_arr = np.ctypeslib.as_ctypes(arr_float32)
In my case it works 10 times faster.
[Edit]: I don't know why it doesn't work without np.copy or with reshape(-1). So it would be awesome if anyone can explain.
I have an issue with a code performing some array operations. It is too slow, because I use loops and input data are quite big. It was the easiest way for me, but now I am looking for something faster than for loops. I was trying to optimize or rewrite code, but unsuccessful. I really aprecciate Your help.
In my code I have three arrays x1, y1 (coordinates of points in grid), g1 (values in the points) and for example their size is 300 x 300. I treat each matrix as composition of 9 and I make calculation for points in the middle one. For example I start with g1(101,101), but I am using data from g1(1:201,1:201)=g2. I need to calculate distance from each point of g1(1:201,1:201) to g1(101,101) (ll matrix), then I calculate nn as it is in the code, next I find value for g1(101,101) from nn and put it in N array. Then I go to g1(101,102) and so on until g1(200,200), where in this last case g2=g1(99:300,99:300).
As i said, this code is not very efficient, even I have to use larger arrays than I gave in the example, it takes too much time. I hope I explain enough clearly what I expect from the code. I was thinking of using arrayfun, but I have never worked with this function, so I don't know how should use it, however it seems to me it won't handle. Maybe there are other solutions, however I couldn't find anything apropriate.
tic
x1=randn(300,300);
y1=randn(300,300);
g1=randn(300,300);
m=size(g1,1);
n=size(g1,2);
w=1/3*m;
k=1/3*n;
N=zeros(w,k);
for i=w+1:2*w
for j=k+1:2*k
x=x1(i,j);
y=y1(i,j);
x2=y1(i-k:i+k,j-w:j+w);
y2=y1(i-k:i+k,j-w:j+w);
g2=g1(i-k:i+k,j-w:j+w);
ll=1./sqrt((x2-x).^2+(y2-y).^2);
ll(isinf(ll))=0;
nn=ifft2(fft2(g2).*fft2(ll));
N(i-w,j-k)=nn(w+1,k+1);
end
end
czas=toc;
For what it's worth, arrayfun() is just a wrapper for a for loop, so it wouldn't lead to any performance improvements. Also, you probably have a typo in the definition of x2, I'll assume that it depends on x1. Otherwise it would be a superfluous variable. Also, your i<->w/k, j<->k/w pairing seems inconsistent, you should check that as well. Also also, just timing with tic/toc is rarely accurate. When profiling your code, put it in a function and run the timing multiple times, and exclude the variable generation from the timing. Even better: use the built-in profiler.
Disclaimer: this solution will likely not help for your actual problem due to its huge memory need. For your input of 300x300 matrices this works with arrays of size 300x300x100x100, which is usually a no-go. Still, it's here for reference with a smaller input size. I wanted to add a solution based on nlfilter(), but your problem seems to be too convoluted to be able to use that.
As always with vectorization, you can do it faster if you can spare the memory for it. You are trying to work with matrices of size [2*k+1,2*w+1] for each [i,j] index. This calls for 4d arrays, of shape [2*k+1,2*w+1,w,k]. For each element [i,j] you have a matrix with indices [:,:,i,j] to treat together with the corresponding elements of x1 and y1. It also helps that fft2 accepts multidimensional arrays.
Here's what I mean:
tic
x1 = randn(30,30); %// smaller input for tractability
y1 = randn(30,30);
g1 = randn(30,30);
m = size(g1,1);
n = size(g1,2);
w = 1/3*m;
k = 1/3*n;
%// these will be indexed on the fly:
%//x = x1(w+1:2*w,k+1:2*k); %// size [w,k]
%//y = x1(w+1:2*w,k+1:2*k); %// size [w,k]
x2 = zeros(2*k+1,2*w+1,w,k); %// size [2*k+1,2*w+1,w,k]
y2 = zeros(2*k+1,2*w+1,w,k); %// size [2*k+1,2*w+1,w,k]
g2 = zeros(2*k+1,2*w+1,w,k); %// size [2*k+1,2*w+1,w,k]
%// manual definition for now, maybe could be done smarter:
for ii=w+1:2*w %// don't use i and j as variables
for jj=k+1:2*k %// don't use i and j as variables
x2(:,:,ii-w,jj-k) = x1(ii-k:ii+k,jj-w:jj+w); %// check w vs k here
y2(:,:,ii-w,jj-k) = y1(ii-k:ii+k,jj-w:jj+w); %// check w vs k here
g2(:,:,ii-w,jj-k) = g1(ii-k:ii+k,jj-w:jj+w); %// check w vs k here
end
end
%// use bsxfun to operate on [2*k+1,2*w+1,w,k] vs [w,k]-sized arrays
%// need to introduce leading singletons with permute() in the latter
%// in order to have shape [1,1,w,k] compatible with the first array
ll = 1./sqrt(bsxfun(#minus,x2,permute(x1(w+1:2*w,k+1:2*k),[3,4,1,2])).^2 ...
+ bsxfun(#minus,y2,permute(y1(w+1:2*w,k+1:2*k),[3,4,1,2])).^2);
ll(isinf(ll)) = 0;
%// compute fft2, operating on [2*k+1,2*w+1,w,k]
%// will return fft2 for each index in the [w,k] subspace
nn = ifft2(fft2(g2).*fft2(ll));
%// we need nn(w+1,k+1,:,:) which is exactly of size [w,k] as needed
N = reshape(nn(w+1,k+1,:,:),[w,k]); %// quicker than squeeze()
N = real(N); %// this solution leaves an imaginary part of around 1e-12
czas=toc;
Is there a way to grow a 3D array in the third dimension using the end index in a loop in Matlab?
In 2D it can be done like
a = [];
for x = y
a(end + 1, :) = f(x);
end
But in 3D the same thing will not work as a(1,1,end) will try to index a(1,1,1) the first iteration (not a(1,1,0) as one might expect). So I can't do
im = [];
for x = y
im(:, :, end + 1) = g(x);
end
It seems the end of a in third dimension is handled a bit differently than in the first two:
>> a = [];
>> a(end,end,end) = 1
Attempted to access a(0,0,1); index must be a positive integer or logical.
Am I missing something about how end indexing works here?
What you're asking...
If you know the size of g(x), initialize im to an empty 3d-array:
im = zeros(n, m, 0); %instead of im = [];
I think your code should work now.
A better way...
Another note, resizing arrays each iteration is expensive! This doesn't really matter if the array is small, but for huge matrices, there can be a big performance hit.
I'd initialize to:
im = zeros(n, m, length(y));
And then index appropriately. For example:
i = 1;
for x = y
im(:, :, i) = g(x);
i = i + 1;
end
This way you're not assigning new memory and copying over the whole matrix im each time it gets resized!
I tried to check if a 3D array is not all zeros using the next code:
notAll_n0_GreaterThan_ni=1;
while notAll_n0_GreaterThan_ni
notAll_n0_GreaterThan_ni=0;
mask=(n0<ni);
numDimensions=ndims(mask);
for dim_ind=1:numDimensions
if any(mask,dim_ind)
notAll_n0_GreaterThan_ni=1;
break;
end
end
if notAll_n0_GreaterThan_ni
n0(mask)=n0(mask)+1;
end
end
It seems I have error in the code because at the end I get for example: n_0(11,3,69)=21 while ni(11,3,69)=21.1556.
I can't find the error. I'll appreciate if someone shows me where I'm wrong and also if there is a simpler way to check existence of nonzero elements in a 3D array.
Let x denote an n-dimensional array. To check if it contains at least one non-zero element, just use
any(x(:))
For example:
>> x = zeros(2,3,4);
>> any(x(:))
ans =
0
>> x(1,2,2) = 5;
>> any(x(:))
ans =
1
Other, more exotic possibilities include:
sum(abs(x(:)))>0
and
nnz(x)>0
This is what you looking for
B = any(your_Array_name_here(:) ==0); no need for loops
the (:) turns the elements of your_Array into a single column vector, so you can use this type of statement on an array of any size
I 've tested this and it works
A = rand(3,7,5) * 5;
B = any(A(:) ==0);
I have a cell, Data, it contains three double arrays,
Data =
[74003x253 double] [8061x253 double] [7241x253 double]
I'm using a loop to read these arrays and perform some functions,
for ii = 1 : 3
D = Data {ii} ;
m = mean (D') ;
// rest of the code
end
Which gets a warning for mean and says:
consider using different DIMENSION input argument for MEAN
However when I change it to,
for ii = 1 : 3
D = Data {ii}' ;
m = mean (D) ;
// rest of the code
end
I get Out of memory error.
Comparing two codes, can someone explain what happens?
It seems that I get the error only with a Complex conjugate transpose (my data is real valued).
To take the mean for the n:th dimension it is possible use mean(D,n) as already stated. Regarding the memory consumption, I did some tests monitoring with the windows resource manager. The output was kind of expected.
When doing the operation D=Data{ii} only minimum memory is consumed since here matlab does no more than copying a pointer. However, when doing a transpose, matlab needs to allocate more memory to store the matrix D, which means that the memory consumption increases.
However, this solely does not cause a memory overflow, since the transpose is done in both cases.
Case 1
Separately inD = Data{ii}';
Case 2
in
D = Data {ii}; m = mean(D');
The difference is that in case 2 matlab only creates a temporary copy of Data{ii}' which is not stored in the workspace. The memory allocated is the same in both cases, but in case 1 Data{ii}' is stored in D. When the memory later increases this can cause a memory overflow.
The memory consumption of D is not that bad (< 200 Mb), but the guess is that the memory got high already and that this was enough to give memory overflow.
The warning message means that instead of,
m = mean (D') ;
you should do:
m = mean (D,2);
This will take the mean along the second dimension, leaving you with a column vector the length of size(D,1).
I don't know why you only get the out of memory error when you do D = Data {ii}'. Perhaps it's becauase when you have it in side of mean (m = mean (D') ; the JIT manages to optimize somehow and save you wasted memory.
Here are some ways of doing this:
for i = 1 : length(Data)
% as chappjc recommends this is an excellent solution
m = mean(Data{i}, 2);
end
Or if you want the transpose and you know the data is real (not complex)
for i = 1 : length(Data)
m = mean(Data{i}.');
end
Note, the dot before the transpose.
Or, skip the loop all together
m = cellfun(#(d) mean(d, 2), Data, 'uniformoutput', false);
When you do:
D = Data{i}'
Matlab will create a new copy of your data. This will allocate 74003x253 doubles, which is about 150MB. As patrick pointed out, given that you might have other data you can easily exceed the allowed memory allocation usage (especially on a 32-bit machine).
If you are running with memory problems, the computations are not sensitive, you may consider using single precision instead of double, i.e.:
data{i} = single(data{i});
Ideally, you want to do the single precision at point of allocation to avoid unnecessary new allocation and copies.
Good luck.