Large Data Array manipulations in MATLAB - arrays

Question

I have a large data set in an array <1x43 cell>. The data size is really large, these are some of the cell dimensions - 5 are <1x327680 double>, 11 are <1x1376256 double>
I am attempting to carry out a resample operation which I have a function for. (Function code shown below). I am trying to take an entire cell from the array, perform the Resample operation and store the result back in the same array location or a different one.
However, I get the following error in line 19 or the Resample function -
"Error using zeros
Maximum variable size allowed by the program is exceeded.
Error in Resample (line 19)
obj = zeros(t,1);
I run into an out of memory error when I comment our that line 19.
Please is there a more efficient way to manipulate such large data sets?
Thank you.
Actual Code:
%% To load each ".dat" file for the 51 attributes to an array.
a = dir('*.dat');
for i = 1:length(a)
eval(['load ' a(i).name ' -ascii']);
end
attributes = length(a);
% Scan folder for number of ".dat" files
datfiles = dir('*.dat');
% Count Number of ".dat" files
numfiles = length(datfiles);
% Read files in to MATLAB
for i = 1:1:numfiles
A{i} = csvread(datfiles(i).name);
end
% Remove discarded variables
ind = [1 22 23 24 25 26 27 32]; % Variables to be removed.
A(ind) = [];
% Reshape all the data into columns - (n x 1)
for i = 1:1:length(A)
temp = A{1,i};
[x,y] = size(temp);
if x == 1 && y ~= 1
temp = temp';
A{1,i} = temp;
end
end
% Retrieves the frequency data for the attributes from Excel spreadsheet
frequency = xlsread('C:\Users\aajwgc\Documents\MATLAB\Research Work\Data\testBig\frequency');
% Removing recorded frequency for discarded variables
frequency(ind) = [];
% Upsampling all the attributes to desired frequency
prompt = {'Frequency (Hz):'};
dlg_title = 'Enter desired output frequency for all attributes';
num_lines = 1;
def = {'50'};
answer= inputdlg(prompt,dlg_title,num_lines,def);
OutFreq = str2num(answer{1});
m = 1;
n = length(frequency);
A_resampled = cell(m,n);
A_resampled(:) = {''};
for i = length(frequency);
raw = cell2mat(A(1,i));
temp= Resample(raw, frequency(i,:), OutFreq);
A_resampled{i} = temp(i);
end
Resample Function:
function obj = Resample(InputData, InFreq, OutFreq, varargin)
%% Preliminary setup
% Allow for selective down-sizing by specifying type
type = 'mean'; %default to the mean/average
if size(varargin,2) > 0
type = varargin{1};
end
% Determine the necessary resampling factor
factor = OutFreq / InFreq;
%% No refactoring required
if (factor == 1)
obj = InputData;
%% Up-Sampling required
elseif (factor > 1)
t = factor * numel(InputData(1:end));
**obj = zeros(t,1); ----------------> Line 19 where I get the error message.**
for i = 1:factor:t
y = ((i-1) / factor) + 1;
z = InputData(y);
obj(i:i+factor) = z;
end
%% Down-Sampling required
elseif (factor < 1)
t = numel(InputData(1:end));
t = floor(t * factor);
obj = zeros(t,1);
factor = int32(1/factor);
if strcmp(type,'mean') %default is mean (process first)
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = mean(InputData(y:y+factor-1));
end
elseif strcmp(type,'min')
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = min(InputData(y:y+factor-1));
end
elseif strcmp(type,'max')
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = max(InputData(y:y+factor-1));
end
elseif strcmp(type,'mode')
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = mode(InputData(y:y+factor-1));
end
elseif strcmp(type,'sum')
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = sum(InputData(y:y+factor-1));
end
elseif strcmp(type,'single')
for i = 1:t
y = (factor * (i-1)) + 1;
obj(i) = InputData(y);
end
else
obj = NaN;
end
else
obj = NaN;
end

Answer 1

If you have the DSP System Toolbox, you could use, e.g., the dsp.FIRInterpolator System Object (http://www.mathworks.co.uk/help/dsp/ref/dsp.firinterpolatorclass.html ) and call its step() function repeatedly to avoid processing all the data in one go.
By the way, up/downsampling (interpolation and decimation) are more complex concepts than you have assumed; in the most general sense they both require some form of filtering to remove artifacts that such processes generate.
You could design these filters yourself and convolve your signal with them, but doing this filter design requires a solid foundation in signal processing. If you want to go this route, I suggest picking up a textbook from somewhere as easy to get wrong without a reference text.