Say, Y is a 7-dimensional array, and I need an efficient way to maximize it along the last 3 dimensions, that will work on GPU.
As a result I need a 4-dimensional array with maximal values of Y and three 4-dimensional arrays with the indices of these values in the last three dimensions.
I can do
[Y7, X7] = max(Y , [], 7);
[Y6, X6] = max(Y7, [], 6);
[Y5, X5] = max(Y6, [], 5);
Then I have already found the values (Y5) and the indices along the 5th dimension (X5). But I still need indices along the 6th and 7th dimensions.
Here's a way to do it. Let N denote the number of dimensions along which to maximize.
Reshape Y to collapse the last N dimensions into one.
Maximize along the collapsed dimensions. This gives argmax as a linear index over those dimensions.
Unroll the linear index into N subindices, one for each dimension.
The following code works for any number of dimensions (not necessarily 7 and 3 as in your example). To achieve that, it handles the size of Y generically and uses a comma-separated list obtained from a cell array to get N outputs from sub2ind.
Y = rand(2,3,2,3,2,3,2); % example 7-dimensional array
N = 3; % last dimensions along which to maximize
D = ndims(Y);
sz = size(Y);
[~, ind] = max(reshape(Y, [sz(1:D-N) prod(sz(D-N+1:end))]), [], D-N+1);
sub = cell(1,N);
[sub{:}] = ind2sub(sz(D-N+1:D), ind);
As a check, after running the above code, observe for example Y(2,3,1,2,:) (shown as a row vector for convenience):
>> reshape(Y(2,3,1,2,:), 1, [])
ans =
0.5621 0.4352 0.3672 0.9011 0.0332 0.5044 0.3416 0.6996 0.0610 0.2638 0.5586 0.3766
The maximum is seen to be 0.9011, which occurs at the 4th position (where "position" is defined along the N=3 collapsed dimensions). In fact,
>> ind(2,3,1,2)
ans =
4
>> Y(2,3,1,2,ind(2,3,1,2))
ans =
0.9011
or, in terms of the N=3 subindices,
>> Y(2,3,1,2,sub{1}(2,3,1,2),sub{2}(2,3,1,2),sub{3}(2,3,1,2))
ans =
0.9011
Related
I have two 2D arrays, one M is 2000x3 and the other N is 20x3 (sets of x,y,z coords). I would like to subtract N from M to produce a 3D array 2000x20x3. Currently I get a ValueError: operands could not be broadcast together with shapes (2000,3) (20,3)
A more simple example as a working exercise
M = np.array([[1,1,1],[2,1,1],[3,1,1],[4,1,1],[1,2,1],[2,2,1],[3,2,1],[4,2,1]])
N = np.array([[0,0,0],[1,0,0]])
M.shape = (8,3)
N.shape = (2,3)
I wish to do A=M-N to produce an 8x2x3 array, where for each value 1->M, there are N sets of differences in the x,y,z coordinates.
In other words:
A = array([[[1,1,1],[0,1,1]],[[2,1,1],[1,1,1]],[[3,1,1],[2,1,1]],[[4,1,1],[3,1,1]],[[1,2,1],[0,2,1]]...])
Is this possible, and if so how? Preferably without the use of any for loops
Use broadcasting:
A = M[:,None]-N
A.shape
# (8, 2, 3)
Consider n row vectors in Matlab, each of size 1xU. For example,
U=20;
n=3;
sU=[U U U];
vectors = arrayfun(#(x) {1:x}, sU);
where vector{1} is the first row vector, vector{2} is the second row vector,..., vector{n} is the last row vector.
We create the matrix Tcoord of size U^n x n reporting all the possible n-tuples from the n row vectors. For each row i of Tcoord, Tcoord(i,1) is an element of the first row vector, Tcoord(i,2) is an element of the second row vector, ... , Tcoord(i,n) is an element of the last row vector.
Tcoord_temp = cell(1,n);
[Tcoord_temp{:}] = ndgrid(vectors{:});
Tcoord_temp = cat(n+1, Tcoord_temp{:});
Tcoord = reshape(Tcoord_temp,[],n);
Suppose now that I augment each of the n row vectors of 3 elements. For example,
vectors_augmented{1}=[vectors{1} 8 9 10];
vectors_augmented{2}=[vectors{2} 11 12 13];
vectors_augmented{3}=[vectors{3} 14 15 16];
I then create a matrix similar to Tcoord but now using vectors_augmented.
Tcoord_temp = cell(1,n);
[Tcoord_temp{:}] = ndgrid(vectors_augmented{:});
Tcoord_temp = cat(n+1, Tcoord_temp{:});
Tcoord_augmented = reshape(Tcoord_temp,[],n); %(U+3)^nxn
I would like your help to re-order the rows of the matrix Tcoord_augmented in a matrix Tcoord_augmented_reshape such that
Tcoord_augmented_reshape(1:U^n,:) is equal to Tcoord.
The remaining rows of Tcoord_augmented_reshape contains the other left rows of Tcoord_augmented.
The simplest approach is to build an auxiliary zero-one matrix the same size as Tcoord_augmented and sort rows based on that:
aug_size = [3 3 3]; % augment size of each vector. Not necessarily equal
vectors_aux = arrayfun(#(a) {[false(1,U) true(1, a)]}, aug_size);
T_aux = cell(1,n);
[T_aux{:}] = ndgrid(vectors_aux{:});
T_aux = cat(n+1, T_aux{:});
T_aux = reshape(T_aux,[],n);
[~, ind] = sortrows(T_aux, n:-1:1); % indices of stably sorting the rows.
% Most significant column is rightmost, as per your code
Tcoord_augmented_reorder = Tcoord_augmented(ind, :);
As a simplified example, suppose I have a dataset composed of 40 sorted values. The values of this example are all integers, though this is not necessarily the case for the actual dataset.
import numpy as np
data = np.linspace(1,40,40)
I am trying to find the maximum value inside the dataset for certain window sizes. The formula to compute the window sizes yields a pattern that is best executed with arrays (in my opinion). For simplicity sake, let's say the indices denoting the window sizes are a list [1,2,3,4,5]; this corresponds to window sizes of [2,4,8,16,32] (the pattern is 2**index).
## this code looks long because I've provided docstrings
## just in case the explanation was unclear
def shapeshifter(num_col, my_array=data):
"""
This function reshapes an array to have 'num_col' columns, where
'num_col' corresponds to index.
"""
return my_array.reshape(-1, num_col)
def looper(num_col, my_array=data):
"""
This function calls 'shapeshifter' and returns a list of the
MAXimum values of each row in 'my_array' for 'num_col' columns.
The length of each row (or the number of columns per row if you
prefer) denotes the size of each window.
EX:
num_col = 2
==> window_size = 2
==> check max( data[1], data[2] ),
max( data[3], data[4] ),
max( data[5], data[6] ),
.
.
.
max( data[39], data[40] )
for k rows, where k = len(my_array)//num_col
"""
my_array = shapeshifter(num_col=num_col, my_array=data)
rows = [my_array[index] for index in range(len(my_array))]
res = []
for index in range(len(rows)):
res.append( max(rows[index]) )
return res
So far, the code is fine. I checked it with the following:
check1 = looper(2)
check2 = looper(4)
print(check1)
>> [2.0, 4.0, ..., 38.0, 40.0]
print(len(check1))
>> 20
print(check2)
>> [4.0, 8.0, ..., 36.0, 40.0]
print(len(check2))
>> 10
So far so good. Now here is my problem.
def metalooper(col_ls, my_array=data):
"""
This function calls 'looper' - which calls
'shapeshifter' - for every 'col' in 'col_ls'.
EX:
j_list = [1,2,3,4,5]
==> col_ls = [2,4,8,16,32]
==> looper(2), looper(4),
looper(8), ..., looper(32)
==> shapeshifter(2), shapeshifter(4),
shapeshifter(8), ..., shapeshifter(32)
such that looper(2^j) ==> shapeshifter(2^j)
for j in j_list
"""
res = []
for col in col_ls:
res.append(looper(num_col=col))
return res
j_list = [2,4,8,16,32]
check3 = metalooper(j_list)
Running the code above provides this error:
ValueError: total size of new array must be unchanged
With 40 data points, the array can be reshaped into 2 columns of 20 rows, or 4 columns of 10 rows, or 8 columns of 5 rows, BUT at 16 columns, the array cannot be reshaped without clipping data since 40/16 ≠ integer. I believe this is the problem with my code, but I do not know how to fix it.
I am hoping there is a way to cutoff the last values in each row that do not fit in each window. If this is not possible, I am hoping I can append zeroes to fill the entries that maintain the size of the original array, so that I can remove the zeroes after. Or maybe even some complicated if - try - break block. What are some ways around this problem?
I think this will give you what you want in one step:
def windowFunc(a, window, f = np.max):
return np.array([f(i) for i in np.split(a, range(window, a.size, window))])
with default f, that will give you a array of maximums for your windows.
Generally, using np.split and range, this will let you split into a (possibly ragged) list of arrays:
def shapeshifter(num_col, my_array=data):
return np.split(my_array, range(num_col, my_array.size, num_col))
You need a list of arrays because a 2D array can't be ragged (every row needs the same number of columns)
If you really want to pad with zeros, you can use np.lib.pad:
def shapeshifter(num_col, my_array=data):
return np.lib.pad(my_array, (0, num_col - my.array.size % num_col), 'constant', constant_values = 0).reshape(-1, num_col)
Warning:
It is also technically possible to use, for example, a.resize(32,2) which will create an ndArray padded with zeros (as you requested). But there are some big caveats:
You would need to calculate the second axis because -1 tricks don't work with resize.
If the original array a is referenced by anything else, a.resize will fail with the following error:
ValueError: cannot resize an array that references or is referenced
by another array in this way. Use the resize function
The resize function (i.e. np.resize(a)) is not equivalent to a.resize, as instead of padding with zeros it will loop back to the beginning.
Since you seem to want to reference a by a number of windows, a.resize isn't very useful. But it's a rabbit hole that's easy to fall into.
EDIT:
Looping through a list is slow. If your input is long and windows are small, the windowFunc above will bog down in the for loops. This should be more efficient:
def windowFunc2(a, window, f = np.max):
tail = - (a.size % window)
if tail == 0:
return f(a.reshape(-1, window), axis = -1)
else:
body = a[:tail].reshape(-1, window)
return np.r_[f(body, axis = -1), f(a[tail:])]
Here's a generalized way to reshape with truncation:
def reshape_and_truncate(arr, shape):
desired_size_factor = np.prod([n for n in shape if n != -1])
if -1 in shape: # implicit array size
desired_size = arr.size // desired_size_factor * desired_size_factor
else:
desired_size = desired_size_factor
return arr.flat[:desired_size].reshape(shape)
Which your shapeshifter could use in place of reshape
How can I reshape a 2d array to a 3d array with the last column being used as pages?
All data found in array2d should be in pages
example:
array2d=[7,.5,12; ...
1,1,1; ...
1,1,1; ...
4,2,4; ...
2,2,2; ...
2,2,2; ...
3,3,3; ...
3,3,3; ...
3,3,3];
The first page in the array would be
7,.5,12;
1,1,1;
1,1,1;
The second page in the array would be
4,2,4;
2,2,2;
2,2,2;
The third page in the array would be
3,3,3;
3,3,3;
3,3,3;
This is a 9x3 array how can I get it to be a 9x3x? (not sure what this number should be so I placed a question mark as a place holder) multidimensional array?
What I'm trying to get is to have
All the ones would be on one dimension/page all the two's would be another dimension/page etc... –
I tried reshape(array2d,[9,3,1]) and it's still a 9x3
Use permute with reshape -
N = 3; %// Cut after every N rows to form a "new page"
array3d = permute(reshape(array2d,N,size(array2d,1)/N,[]),[1 3 2]) %// output
Assuming that each slice of your matrix is the same in dimensions, we can do this very easily. Let's call the number of rows and columns that each slice would have to be M and N respectively. In your example, this would be M = 3 and N = 3. As such, assuming array2d is of the above form, we can do the following:
M = 3;
N = 3; %// This is also simply the total number of columns we have,
%// so you can do size(array2d, 2);
outMatrix = []; %// Make this empty. We will populate as we go.
%// Figure out how many slices we need
numRows = size(array2d,1) / M;
for k = 1 : numRows
%// Extract the k'th slice
%// Reshape so that it has the proper dimensions
%// of one slice
sliceK = reshape(array2d(array2d == k), M, N);
%// Concatenate in the third dimension
outMatrix = cat(3,outMatrix,sliceK);
end
With your example, we thus get:
>> outMatrix
outMatrix(:,:,1) =
1 1 1
1 1 1
1 1 1
outMatrix(:,:,2) =
2 2 2
2 2 2
2 2 2
outMatrix(:,:,3) =
3 3 3
3 3 3
3 3 3
This method should generalize for any number of rows and columns for each slice, provided that each slice shares the same dimensions.
Your array is already of size 1 in the 3rd dimension (in other words, it is already 9x3x1, to prove this try entering array2d(1,1,1)). If you want to concatenate 2d matrices along the 3rd dimension you can use cat.
For example:
a = [1,2;3,4];
b = [5,6;7,8];
c = cat(3,a,b);
c will be a 2x2x2 matrix.
This piece of code is specific for this example, I hope you will be able to understand how to go for other data samples.
out2 = [];
col = size(array2d,2);
for i = 1:3
temp2 = reshape(array2d(array2d == i),[],col);
out2 = cat(3,out2,temp2);
end
I am trying to do some numpy matrix math because I need to replicate the repmat function from MATLAB. I know there are a thousand examples online, but I cannot seem to get any of them working.
The following is the code I am trying to run:
def getDMap(image, mapSize):
newSize = (float(mapSize[0]) / float(image.shape[1]), float(mapSize[1]) / float(image.shape[0]))
sm = cv.resize(image, (0,0), fx=newSize[0], fy=newSize[1])
for j in range(0, sm.shape[1]):
for i in range(0, sm.shape[0]):
dmap = sm[:,:,:]-np.array([np.tile(sm[j,i,:], (len(sm[0]), len(sm[1]))) for k in xrange(len(sm[2]))])
return dmap
The function getDMap(image, mapSize) expects an OpenCV2 HSV image as its image argument, which is a numpy array with 3 dimensions: [:,:,:]. It also expects a tuple with 2 elements as its imSize argument, of course making sure the function passing the arguments takes into account that in numpy arrays the rows and colums are swapped (not: x, y, but: y, x).
newSize then contains a tuple containing fracions that are used to resize the input image to a specific scale, and sm becomes a resized version of the input image. This all works fine.
This is my goal:
The following line:
np.array([np.tile(sm[i,j,:], (len(sm[0]), len(sm[1]))) for k in xrange(len(sm[2]))]),
should function equivalent to the MATLAB expression:
repmat(sm(j,i,:),[size(sm,1) size(sm,2)]),
This is my problem:
Testing this, an OpenCV2 image with dimensions 800x479x3 is passed as the image argument, and (64, 48) (a tuple) is passed as the imSize argument.
However when testing this, I get the following ValueError:
dmap = sm[:,:,:]-np.array([np.tile(sm[i,j,:], (len(sm[0]),
len(sm[1]))) for k in xrange(len(sm[2]))])
ValueError: operands could not be broadcast together with
shapes (48,64,3) (64,64,192)
So it seems that the array dimensions do not match and numpy has a problem with that. But my question is what? And how do I get this working?
These 2 calculations match:
octave:26> sm=reshape(1:12,2,2,3)
octave:27> x=repmat(sm(1,2,:),[size(sm,1) size(sm,2)])
octave:28> x(:,:,2)
7 7
7 7
In [45]: sm=np.arange(1,13).reshape(2,2,3,order='F')
In [46]: x=np.tile(sm[0,1,:],[sm.shape[0],sm.shape[1],1])
In [47]: x[:,:,1]
Out[47]:
array([[7, 7],
[7, 7]])
This runs:
sm[:,:,:]-np.array([np.tile(sm[0,1,:], (2,2,1)) for k in xrange(3)])
But it produces a (3,2,2,3) array, with replication on the 1st dimension. I don't think you want that k loop.
What's the intent with?
for i in ...:
for j in ...:
data = ...
You'll only get results from the last iteration. Did you want data += ...? If so, this might work (for a (N,M,K) shaped sm)
np.sum(np.array([sm-np.tile(sm[i,j,:], (N,M,1)) for i in xrange(N) for j in xrange(M)]),axis=0)
z = np.array([np.tile(sm[i,j,:], (N,M,1)) for i in xrange(N) for j in xrange(M)]),axis=0)
np.sum(sm - z, axis=0) # let numpy broadcast sm
Actually I don't even need the tile. Let broadcasting do the work:
np.sum(np.array([sm-sm[i,j,:] for i in xrange(N) for j in xrange(M)]),axis=0)
I can get rid of the loops with repeat.
sm1 = sm.reshape(N*M,L) # combine 1st 2 dim to simplify repeat
z1 = np.repeat(sm1, N*M, axis=0).reshape(N*M,N*M,L)
x1 = np.sum(sm1 - z1, axis=0).reshape(N,M,L)
I can also apply broadcasting to the last case
x4 = np.sum(sm1-sm1[:,None,:], 0).reshape(N,M,L)
# = np.sum(sm1[None,:,:]-sm1[:,None,:], 0).reshape(N,M,L)
With sm I have to expand (and sum) 2 dimensions:
x5 = np.sum(np.sum(sm[None,:,None,:,:]-sm[:,None,:,None,:],0),1)
len(sm[0]) and len(sm[1]) are not the sizes of the first and second dimensions of sm. They are the lengths of the first and second row of sm, and should both return the same value. You probably want to replace them with sm.shape[0] and sm.shape[1], which are equivalent to your Matlab code, although I am not sure that it will work as you expect it to.