Logical indexing in matlab - need help to make faster - arrays

This is what I am trying to do, created a random array to demonstrate:
% all IDs
all_IDS = 1:216000000;
% Array 1
X = round(1550*rand(216000000,1));
Y = round(1550*rand(216000000,1));
Z = round(90*rand(216000000,1));
% Array 2
Xsub = round(1550*rand(160000,1));
Ysub = round(1550*rand(160000,1));
Zsub = round(90*rand(160000,1));
del_val =1;
% required o/p
reqd_op = zeros(1,10);
% boolean indexing
indx =1;
for jj = 1:160000
VID_X = Xsub(jj);
VID_Y = Ysub(jj);
VID_Z = Zsub(jj);
I2 = (X>VID_X-del_val & X<VID_X+del_val)& (Y>VID_Y-del_val & Y<VID_Y+del_val) & (Z>VID_Z-del_val & Z<VID_Z+del_val);
len = numel(all_IDS(I2));
reqd_op(1,indx:indx+len-1) = all_IDS(I2);
indx=indx+len;
end
The above code takes a lot of time as I am dealing with a very large array , Is there a way to eliminate the for loop, meaning, instead of doing Boolean indexing element by element - can I do it for the whole array at once ?

This will run x2.5 faster, anyway, array is too big so it still takes 0.3s per loop, so 160000 loops is like 13 hours on single cpu.
if ~exist('X','var')
% Array 1
X = round(1550*rand(216000000,1,'single'));
Y = round(1550*rand(216000000,1,'single'));
Z = round(90*rand(216000000,1,'single'));
% Array 2
Xsub = round(1550*rand(160000,1,'single'));
Ysub = round(1550*rand(160000,1,'single'));
Zsub = round(90*rand(160000,1,'single'));
end
del_val =single(1);
reqd_op = zeros(1,10,'single');% required o/p
tic
index =1;
for jj = 1:10
VID_X = Xsub(jj);
VID_Y = Ysub(jj);
VID_Z = Zsub(jj);
IdxFinal=[];
Idx1=find(abs(X-VID_X)<del_val); %little better than X>VID_X-del_val & X<VID_X+del_val)
if ~isempty(Idx1)
Idx2 = Idx1(Y(Idx1)>VID_Y-del_val & Y(Idx1)<VID_Y+del_val);
if ~isempty(Idx2)
Idx3= Idx2(Z(Idx2)>VID_Z-del_val & Z(Idx2)<VID_Z+del_val);
IdxFinal=Idx3;
end
end
len = length(IdxFinal);
index=index+len;
if len>0
reqd_op(1,index:index+len-1) = IdxFinal;
end
end
toc

Related

Get matrix consisting of every subsquare

I wish to obtain a matrix consisting of every 3x3 square within a matrix. The matrix is a #channels by m by n. The end result is a matrix of size #channels x (size1-2)*(size2-2) x 9I can do this in a non-vector fashion via the following code:
clear
size1 = 10;
size2 = 10;
matrix = 1:size1*size2;
matrix =reshape(matrix,[size1 size2]);
matrix_withdraw = 1:(88*size1*size2);
matrix_withdraw = reshape(matrix_withdraw,[88 size1 size2]);
tic
iter = 1;
for ii = 1:size1-2
for jj = 1:size2-2
locs(ii,jj,:,:) = matrix(ii:ii+2,jj:jj+2);
method1(:,iter,:) = reshape(matrix_withdraw(:,ii:ii+2,jj:jj+2),[88,9]);
iter = iter+1;
end
end
locs = permute(locs,[3 4 1 2]);
This is obviously fairly slow for large sizes of matrices. I'm looking to vectorize this. I managed to obtain a solution which works but it's not very clean
locs2 = ones(size1*(size2-2),1)+(0:size1*(size2-2)-1)';
temp = 1:size1*(size2-2);
temp = mod(temp,size1);
temp = (temp>(size1-2)) | (temp==0);
locs2(temp,:) = [];
locs3 = reshape(locs2,[1 1 (size1-2) (size2-2)]);
locs3(2,1,:,:) = reshape(locs2+1,[1 1 (size1-2) (size2-2)]);
locs3(3,1,:,:) = reshape(locs2+2,[1 1 (size1-2) (size2-2)]);
locs3(:,2,:,:) = locs3(:,1,:,:)+size1;
locs3(:,3,:,:) = locs3(:,2,:,:)+size1;
locs3 = permute(locs3,[1 2 4 3]);
locs3 = vec(locs3);
method2 = matrix_withdraw(:,locs3);
method2 = reshape(method2,[88,9,64]);
method2 = permute(method2,[1 3 2]);
Method1 and method2 are equivalent and render the exact same results. Method2 is also 10x faster than method1. My question is, is there a cleaner way to do this?
If you have the Image Processing Toolbox, you can use im2col to get each block as a column, and then reshape as a 4-D array:
blockSize = [3 3];
locs = reshape(im2col(matrix, blockSize), [blockSize size(matrix)-blockSize+1]);
Or you can do it directly, building a 4-D indexing array using implicit singleton expansion. This doesn't require any toolboxes:
m = 3; n = 3; % block size
[M, N] = size(matrix);
ind = (1:m).'+(0:n-1)*M + reshape(0:M-m, 1, 1, []) + reshape((0:N-n)*M, 1, 1, 1, []);
locs = matrix(ind);

Index Subset of Array in For Loop in MATLAB

I have a question regarding indexing and loops in MATLAB. I have a vector of length n (named data in the code below). I want to examine this vector 4 elements at a time inside of a for loop. How can I do this? My attempt included below does not work because it will exceed the array dimensions at the end of the loop.
for k = 1:length(data)
index = k:k+3;
cur_data = data(index);
pre_q_data1 = cur_data(1);
pre_q_data2 = cur_data(2);
% Interweaving the data
q = [pre_q_data1; pre_q_data2];
qdata = q(:)';
pre_i_data1 = cur_data(3);
pre_i_data2 = cur_data(4);
i = [pre_i_data1; pre_i_data2];
idata = i(:)';
end
You shouldn't have k go all the way to length(data) if you're planning on indexing up to k+3.
I've also taken the liberty of greatly simplifying your code, but feel free to ignore that!
for k = 1:length(data)-3
% maximum k = length(data)-3, so maximum index = length(data)-3+3=length(data)
index = k:k+3;
cur_data = data(k:k+3);
% Interweaving the data
q = cur_data(1:2); % transpose at end of line here if need be
i = cur_data(3:4); % could just use data(k+2:k+3) and not use cur_data
end

Mapping An Array To Logical Array In Matlab

Let's say an array a=[1,3,8,10,11,15,24], and a logical array b=[1,0,0,1,1,1,0,0,0,1,1,1,1,1], how to get [1,1,3,1,3,8,1,3,8,1,2,3,8,10], see where logic becomes 1 in b, the array index of a resets so it starts from the beginning, also the same where the logic becomes 0 a array starts from beginning and continues as 1,3,8,10..etc.
you can use diff to find where b changes, then use arrayfun to generate indexes for a:
a=[1,3,8,10,11,15,24];
b=[1,0,0,1,1,1,0,0,0,1,1,1,1,1];
idxs = find(diff(b) ~= 0) + 1; % where b changes
startidxs = [1 idxs];
endidxs = [idxs - 1,length(b)];
% indexes for a
ia = cell2mat(arrayfun(#(x,y) 1:(y-x+1),startidxs,endidxs,'UniformOutput',0));
res = a(ia);
You can use a for loop and track the state (0 or 1) of the b array:
a = [1,3,8,10,11,15,24];
b = [1,0,0,1,1,1,0,0,0,1,1,1,1,1];
final = []
index = 0;
state = b(1);
for i = 1:numel(b)
if b(i) ~= state
state = b(i);
index = 1;
else
index = index+1;
end
final = [final, a(index) ];
end

MATLAB: search for elements in an array matching multidimensional condition

I have a column vector (V1) of real numbers like:
123.2100
125.1290
...
954.2190
If I add, let's say, a number 1 to each row in this vector, I will get (V2):
124.2100
126.1290
...
955.2190
I need to find out how many elements from V2 are inside some error-window created from V1. For example the error-window = 0.1 (but in my case every element in V1 has it's own error window):
123.1100 123.3100
125.0290 125.2290
...
954.1190 954.3190
I can create some code like this:
% x - my vector
% ppm - a variable responsible for error-window
window = [(1-(ppm/1000000))*x, (1+(ppm/1000000))*x]; % - error-window
mdiff = 1:0.001:20; % the numbers I will iteratively add to x
% (like the number 1 in the example)
cdiff = zeros(length(mdiff),1); % a vector that will contain counts of elements
% corresponding to different mdiff temp = 0;
for i = 1:length(mdiff)
for j = 1:size(window,1)
xx = x + mdiff(i);
indxx = find( xx => window(j,1) & xx <= window(j,2) );
if any(indxx)
temp = temp + length(indxx); %edited
end
end
cdiff(i) = temp;
temp = 0;
end
So, at the end cdiff will contain all the counts corresponding to mdiff. The only thing, I would like to make the code faster. Or is there a way to avoid using the second loop (with j)? I mean to directly use a multidimensional condition.
EDIT
I decided to simpify the code like this (thanking to the feedback I got here):
% x - my vector
% ppm - a variable responsible for error-window
window = [(1-(ppm/1000000))*x, (1+(ppm/1000000))*x]; % - error-window
mdiff = 1:0.001:20; % the numbers I will iteratively add to x
% (like the number 1 in the example)
cdiff = zeros(length(mdiff),1); % a vector that will contain counts of elements
% corresponding to different mdiff temp = 0;
for i = 1:length(mdiff)
xx = x + mdiff(i);
cdiff(i) = sum(sum(bsxfun(#and,bsxfun(#ge,xx,window(:,1)'),bsxfun(#le,xx,window(:,2)'))));
end
In this case the code works faster and seems properly
add = 1; %// how much to add
error = .1; %// maximum allowed error
V2 = V1 + add; %// build V2
ind = sum(abs(bsxfun(#minus, V1(:).', V2(:)))<error)>1; %'// index of elements
%// of V1 satisfying the maximum error condition. ">1" is used to because each
%// element is at least equal to itself
count = nnz(ind);
Think this might work for you -
%%// Input data
V1 = 52+rand(4,1)
V2 = V1+1;
t= 0.1;
low_bd = any(abs(bsxfun(#minus,V2,[V1-t]'))<t,2); %%//'
up_bd = any(abs(bsxfun(#minus,V2,[V1+t]'))<t,2); %%//'
count = nnz( low_bd | up_bd )
One could also write it as -
diff_map = abs(bsxfun(#minus,[V1-t V1+t],permute(V2,[3 2 1])));
count = nnz(any(any(diff_map<t,2),1))
Edit 1:
low_bd = any(abs(bsxfun(#minus,V2,window(:,1)'))<t,2); %%//'
up_bd = any(abs(bsxfun(#minus,V2,window(:,2)'))<t,2); %%//'
count = nnz( low_bd | up_bd )
Edit 2: Vectorized form for the edited code
t1 = bsxfun(#plus,x,mdiff);
d1 = bsxfun(#ge,t1,permute(window(:,1),[3 2 1]));
d2 = bsxfun(#le,t1,permute(window(:,2),[3 2 1]));
t2 = d1.*d2;
cdiff_vect = max(sum(t2,3),[],1)';

Large Data Array manipulations in MATLAB

I have a large data set in an array <1x43 cell>. The data size is really large, these are some of the cell dimensions - 5 are <1x327680 double>, 11 are <1x1376256 double>
I am attempting to carry out a resample operation which I have a function for. (Function code shown below). I am trying to take an entire cell from the array, perform the Resample operation and store the result back in the same array location or a different one.
However, I get the following error in line 19 or the Resample function -
"Error using zeros
Maximum variable size allowed by the program is exceeded.
Error in Resample (line 19)
obj = zeros(t,1);
I run into an out of memory error when I comment our that line 19.
Please is there a more efficient way to manipulate such large data sets?
Thank you.
Actual Code:
%% To load each ".dat" file for the 51 attributes to an array.
a = dir('*.dat');
for i = 1:length(a)
eval(['load ' a(i).name ' -ascii']);
end
attributes = length(a);
% Scan folder for number of ".dat" files
datfiles = dir('*.dat');
% Count Number of ".dat" files
numfiles = length(datfiles);
% Read files in to MATLAB
for i = 1:1:numfiles
A{i} = csvread(datfiles(i).name);
end
% Remove discarded variables
ind = [1 22 23 24 25 26 27 32]; % Variables to be removed.
A(ind) = [];
% Reshape all the data into columns - (n x 1)
for i = 1:1:length(A)
temp = A{1,i};
[x,y] = size(temp);
if x == 1 && y ~= 1
temp = temp';
A{1,i} = temp;
end
end
% Retrieves the frequency data for the attributes from Excel spreadsheet
frequency = xlsread('C:\Users\aajwgc\Documents\MATLAB\Research Work\Data\testBig\frequency');
% Removing recorded frequency for discarded variables
frequency(ind) = [];
% Upsampling all the attributes to desired frequency
prompt = {'Frequency (Hz):'};
dlg_title = 'Enter desired output frequency for all attributes';
num_lines = 1;
def = {'50'};
answer= inputdlg(prompt,dlg_title,num_lines,def);
OutFreq = str2num(answer{1});
m = 1;
n = length(frequency);
A_resampled = cell(m,n);
A_resampled(:) = {''};
for i = length(frequency);
raw = cell2mat(A(1,i));
temp= Resample(raw, frequency(i,:), OutFreq);
A_resampled{i} = temp(i);
end
Resample Function:
function obj = Resample(InputData, InFreq, OutFreq, varargin)
%% Preliminary setup
% Allow for selective down-sizing by specifying type
type = 'mean'; %default to the mean/average
if size(varargin,2) > 0
type = varargin{1};
end
% Determine the necessary resampling factor
factor = OutFreq / InFreq;
%% No refactoring required
if (factor == 1)
obj = InputData;
%% Up-Sampling required
elseif (factor > 1)
t = factor * numel(InputData(1:end));
**obj = zeros(t,1); ----------------> Line 19 where I get the error message.**
for i = 1:factor:t
y = ((i-1) / factor) + 1;
z = InputData(y);
obj(i:i+factor) = z;
end
%% Down-Sampling required
elseif (factor < 1)
t = numel(InputData(1:end));
t = floor(t * factor);
obj = zeros(t,1);
factor = int32(1/factor);
if strcmp(type,'mean') %default is mean (process first)
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = mean(InputData(y:y+factor-1));
end
elseif strcmp(type,'min')
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = min(InputData(y:y+factor-1));
end
elseif strcmp(type,'max')
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = max(InputData(y:y+factor-1));
end
elseif strcmp(type,'mode')
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = mode(InputData(y:y+factor-1));
end
elseif strcmp(type,'sum')
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = sum(InputData(y:y+factor-1));
end
elseif strcmp(type,'single')
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = InputData(y);
end
else
obj = NaN;
end
else
obj = NaN;
end
If you have the DSP System Toolbox, you could use, e.g., the dsp.FIRInterpolator System Object (http://www.mathworks.co.uk/help/dsp/ref/dsp.firinterpolatorclass.html ) and call its step() function repeatedly to avoid processing all the data in one go.
By the way, up/downsampling (interpolation and decimation) are more complex concepts than you have assumed; in the most general sense they both require some form of filtering to remove artifacts that such processes generate.
You could design these filters yourself and convolve your signal with them, but doing this filter design requires a solid foundation in signal processing. If you want to go this route, I suggest picking up a textbook from somewhere as easy to get wrong without a reference text.

Resources