I have two numpy arrays that are shaped (8760,1) that I want to combine into a single array that is (8760,2) and then from that, filter out any values of zero that might be in first index column, or gauge in the "data" so that I can do statistical manipulation with the temp array. I have tried np.stack and then attempted to filter out any zero values that way, but ended up with my temp array being 3D rather than still 2D.
data=np.stack((mb, gauge), axis=-1)
dta = data[:,data!=0]
idx = np.where(data[:,1]>0)
temp = data[idx,:]
I know I could filter out the zeros from gauge first, but I want to preserve the index values that go along with the mb array
np.stack joins along a newly created axis. Thus your arrays become 3D.
To join along an existing axis, you could use np.concatenate:
a1 = np.empty((100, 1))
a2 = np.empty((100, 1))
a3 = np.concatenate((a1, a2), axis=1) # will give a (100, 2) array
Related
I have a numpy array of size k, and a pandas dataframe with a column of size n>k that contains k missing values.
Is there an easy way to fill the k missing values from the numpy array correspondingly (that is, first occurred missing value in the column of the dataframe corresponds to the next value in the array)?
Something like this might work. You may also want to consider what order (i.e. sorting) you want to fill these values in.
fill_values = list(range(k)) #or whatever your array is
indicies_of_missing = df[df['myColumn'].isnull()].index # list of the missing indices
for fill_index, dataframe_index in enumerate(indicies_of_missing):
dataframe.loc[dataframe_index, 'myColumn'] = fill_values[fill_index]
I have a two arrays within a <1x2 cell>. I want to permute those arrays. Of course, I could use a loop to permute each one, but is there any way to do that task at once, without using loops?
Example:
>> whos('M')
Name Size Bytes Class Attributes
M 1x2 9624 cell
>> permute(M,p_matrix)
This does not permute the contents of the two arrays within M.
I could use something like:
>> for k=1:size(M,2), M{k} = permute(M{k},p_matrix); end
but I'd prefer not to use loops.
Thanks.
This seems to work -
num_cells = numel(M) %// Number of cells in input cell array
size_cell = size(M{1}) %// Get sizes
%// Get size of the numeric array that will hold all of the data from the
%// input cell array with the second dimension representing the index of
%// each cell from the input cell array
size_num_arr = [size_cell(1) num_cells size_cell(2:end)]
%// Dimensions array for permuting with the numeric array holding all data
perm_dim = [1 3:numel(size_cell)+1 2]
%// Store data from input M into a vertically concatenated numeric array
num_array = vertcat(M{:})
%// Reshape and permute the numeric array such that the index to be used
%// for indexing data from different cells ends up as the final dimension
num_array = permute(reshape(num_array,size_num_arr),perm_dim)
num_array = permute(num_array,[p_matrix numel(size_cell)+1])
%// Save the numeric array as a cell array with each block from
%// thus obtained numeric array from its first to the second last dimension
%// forming each cell
size_num_arr2 = size(num_array)
size_num_arr2c = num2cell(size_num_arr2(1:end-1))
M = squeeze(mat2cell(num_array,size_num_arr2c{:},ones(1,num_cells)))
Some quick tests show that mat2cell would prove to be the bottleneck, so if you don't mind indexing into the intermediate numeric array variable num_array and use it's last dimension for an equivalent indexing into M, then this approach could be useful.
Now, another approach if you would like to preserve the cell format would be with arrayfun, assuming each cell of M to be a 4D numeric array -
M = arrayfun(#(x) num_array(:,:,:,:,x),1:N,'Uniform',0)
This seems to perform much better than with mat2cell in terms of performance.
Please note that arrayfun isn't a vectorized solution as most certainly it uses loops behind-the-scenes and seems like mat2cell is using for loops inside its source code, so please do keep all these issues in mind.
I currently have a column vectors of different lengths and I want to insert another column vector at various points of the original array. i.e. I want to add my new array to the start of the old array skip 10 places add my new array again, skip another 10 spaces and add my new array again and so on till the end of the array. I can do this by using:
OffsetSign = [1:30]';
Extra = [0;0;0;0;0];
OffsetSign =[Extra;OffsetSign(1:10);Extra;OffsetSign(11:20);Extra;OffsetSign(21:30)];
However this is not suitable for longer arrays. Any tips on an easy way to do this for longer arrays?
here's one way to do it:
a = [1:30]';
b = [0;0;0;0;0];
a=reshape(a,10,[]);
b=repmat(b,[1 size(a,2)])
r=[b ; a]
r=r(:);
the trick is to reshape a to a matrix with columns of the right size (10 elements each). Replicate b to this # of columns , concatenate both and flatten the matrix to a vector...
I have a large structure array.
I would like to perform a sensitivity analysis on a function that processes this array.
So, say my structure array has name 's', 10,000 elements, and field names 'x' and 'y'.
I'd like to do the following:
xs = [s(:).x];
xs = xs + 5*randn(size(xs));
s(:).x = xs;
Sadly, the last step is not valid matlab. Any thoughts? Was hoping to avoid a loop.
From this answer and after playing around with deal. I think I have what you are looking for but it requires converting xs into a cell array using num2cell:
xs_cell = num2cell(xs); % convert matrix to cell array.
[S(:).X]=xs_cell{:}; % update the values in the field X
I have a loop that adds elements to a 1d array:
for i in range(0, 1000):
fvector[0, i] = function_value
after the loop finishes, I have a 1 x 1000 vector that I want to store in a multi-dimensional array fmatrix, which is 50 x 1000. I managed to do this using a loop and copying each element individually - but it is very slow. I've then tried to use slice to copy the whole vector in one go after the loop and then be ready to copy next vector at the next column. How do I make it go to the next column? I've tried:
s=slice([i], None)
fmatrix[s] = fvector
and various combinations for s, but I get error messages about setting an array element with a sequence, or invalid syntax.
I know this should be straight forward but I'm very new to python, numpy and arrays :-(
Try this. Allocate the matrix, here zero-initialized for effect:
>>> import numpy as np
>>> fmatrix = np.zeros((50, 1000))
Then index into it to obtain fvector:
>>> fvector = fmatrix[0]
Then assign to fvector's elements:
>>> for i in xrange(1000):
... fvector[i] = i
If you now inspect fmatrix[0], the first row of fmatrix, you'll find that it has been assigned to in the previous loop. That's because the NumPy row indexing creates fvector as a view on fmatrix's first row. This saves you a copy.
fvector has shape (1,1000). That's a 2D array, even if one axis has length 1.
You can slice it down to a 1D array with fvector[0,:]. This gives the first row.
fmatrix has shape (50,1000). You can slice it down to a 1D array with fmatrix[i,:]. This gives the ith row.
So to assign the values in the first row of fvector to the ith row of fmatrix:
fmatrix[i,:] = fvector[0,:]
Perhaps however there is no need for fvector to be a 2D array? Perhaps just make it a 1D array to begin with:
fvector = np.empty(1000)
for i in range(0, 1000):
fvector[i] = function_value
and then you could do the assignment with
fmatrix[i,:] = fvector