matlab complex for-loop correlation calcul - arrays

This is the script that I have. It works till the ------ separation. Under I do not get any error from Matlab, but neither do I get a return of bestDx nor bestDy. Please help. (The first part is given just to put you in context)
%%
% Variables after running script Read_eA3_file.m
%date_time_UTC
%reflectivity
%clutter_mask
%Convert units
dBZ = reflectivity * 0.375 - 30;
dBZ_Mask = clutter_mask * 0.375 - 30;
%Replace clutter values with NaN
weather = NaN(size(dBZ)); %initialise to constant
weather(dBZ>=dBZ_Mask) = dBZ(dBZ>=dBZ_Mask); %copy values when A >= B
%Reduce to range -- those are 384x384 arrays
dBZ_range = dBZ(:,:,1:16); %16:18 to 16:23 included
weather_range = weather(:,:,1:16); %16:18 to 16:23 included
weather1618 = weather(:,:,1); %16:18 map only
weather1623 = weather(:,:,16); %16:23 map only
% Plot maps
image(imrotate(-weather1618,90)); %of 16:18
image(imrotate(-weather1623,90)); %of 16:23
%Find x,y of strongest dBZ
%Since the value are all negative. I look for their minimun
[M,I] = min(weather1618(:)); %for 16:18
[I_row, I_col] = ind2sub(size(weather1618),I); %values are 255 and 143
[M2,I2] = min(weather1623(:)); %for 16:23
[I2_row, I2_col] = ind2sub(size(weather1623),I2); %values are 223 and 7
%Calc displacement
%I get a value of 139.7140
max_displ=sqrt((I2_row-I_row)^2+(I2_col-I_col)^2); %between 1618 and 1623
%%
% -----Section below does not work; ONLY RUN the section ABOVE---------
%% Find Dx Dy for max_corr between two maps
maxCoeff=0;
weather1618Modified = zeros(384,384); %create weather array for time range
%weather1618Modified(:) = {NaN}; % Matlab cannot mix cell & double
%%
for x = 1:384
for y = 1:384
%30 pixel appx.
for Dx = -max_displ:30: max_displ
for Dy = -max_displ:30: max_displ
%Limit range of x+Dx and y+Dy to 1:384
if x+Dx<1 | y+Dy<1 | x+Dx>384 | y+Dy>384
continue
%weather1618Modified is the forecasted weather1823
weather1618Modified(x+Dx,y+Dy) = weather1618(x,y)
%Find the best correlation; Is corrcoef the right formula?
newCoeff=corrcoef(weather1623,weather1618Modified);
if newCoeff>maxCoeff
maxCoeff=newCoeff;
bestDx=Dx;
bestDy=Dy;
end
end
end
end
end
end
%% Calc displacement
bestDispl = sqrt(bestDx^2+bestDy^2); %bestDispl for a 5 min frame
%Calc speed
speed = bestDispl/time;

You have to delete the continue statement after the first if (or place it somewhere else).
The continue statement makes the program skip the remaining part of the for-loop and go directly to the next iteration. Therefore bestDx and bestDy will never be set.
Documentation: https://se.mathworks.com/help/matlab/ref/continue.html

Related

Summation based on unique entries of two arrays | Speed Issue

I have 3 arrays of size 803500*1 with the following details:
Rid: It can contain any number
RidID: It contains elements from 1 to 184 in random order. Each element appears multiple times.
r: It contains elements 0,1,2,...12. All elements (except zero) appear nearly 3400 to 3700 times at random indices in this array.
Following may be useful for generating sample data:
Rid = rand(803500,1);
RidID = randi(184,803500,1);
r = randi(13,803500,1)-1; %This may not be a good sample for r as per previously mentioned details?
What I want to do?
I want to calculate the sum of those entries of Rid which correspond to each positive unique entry of r and each unique entry of RidID.
This may be clearer with the code which I wrote for this problem:
RNum = numel(unique(RidID));
RSum = ones(RNum,12); %Preallocating for better speed
for i=1:12
RperM = r ==i;
for j = 1:RNum
RSum(j,i) = sum(Rid(RperM & (RidID==j)));
end
end
Issue:
My code works but it takes 5 seconds on average on my computer and I have to do this calculation nearly a thousand times. If this time be reduced from 5 seconds to atleast half of it, I'll be very happy. But how do I optimize this? I don't mind if it is made better with vectorization or any better written loop.
I am using MATLAB R2017b.
You can use accumarray :
u = unique(RidID);
A = accumarray([RidID r+1], Rid);
RSum = A(u, 2:13);
This is slower than accumarray as suggested by rahnema, but using findgroups and splitapply may save memory.
In your example, there may be thousands of zero-valued elements in the resulting matrix, where a combination of RidID and r does not occur. In this case a stacked result would be more memory efficient, like so:
RidID | r | Rid_sum
-------------------------
1 | 1 | 100
2 | 1 | 200
4 | 2 | 85
...
This can be achieved with the following code:
[ID, rn, RidIDn] = findgroups(r,RidID); % Get unique combo ID for 'r' and 'RidID'
RSum = splitapply( #sum, Rid, ID ); % Sum for each ID
output = table( RidIDn, rn, RSum ); % Nicely formatted table output
% Get rid of elements where r == 0
output( output.rn == 0, : ) = [];
You could convert this to the same output as the accumarray method, but it's already a slower method...
% Convert to 'unstacked' 2D matrix (optional)
RSum = full( sparse( 1:numel(Ridn), 1:numel(rn), RSum ) );

How to add additional zero arrrays

I have the following problem in my simulation.
A is an array 24 x 2. I am going to split it and get 4 or 12 array. It means that I group 6 or 2 array. It will be ok, if I use even "split" coefficient. If it is odd, I can"t split A.[ I can't group 5 or 7, because of 24/5=4*5 + 4 ( or 5*5 -1) or 24/7=7*3+3.
That's why I going to do the following:
If I have 24 x 2 and need group every 5 together:
block 1 : A(1,:), A(2,:),A(3,:),A(4,:),A(5,:)
block 2 : A(6,:), A(7,:),A(8,:),A(9,:),A(10,:)
block 3 : A(11,:), A(12,:),A(13,:),A(14,:),A(15,:)
block 4 : A(16,:), A(17,:),A(18,:),A(19,:),A(20,:)
block 5 : A(21,:), A(22,:),A(23,:),A(24,:), ?
As you can see the 5th block is not full, Matlab gives me an error. My idea is to create A(25,:)=0. For my simulation it will be ok.
I am going to simulate it as function:
A=rand(m,n)
w- # number of a vector that i would like group together ( in ex., it is `5`)
if mod(w,2)==0
if mod(m,2)==0
% do....
else
% remainder = 0
end
else
if mod(m,2)==0
% remainder = 0
else
%do...
end
I was going to simulate like above, but then I have noticed that it doesn't work. Because 24/10 = 2*10+4. So I should write something else
I can find the reminder as r = rem(24,5). As an example above, MatLab gives me r=4. then I can find a difference c= w-r =1 and after that, I don't know how to do that.
Could you suggest to me how to simulate such a calculation?
Determine the number of blocks needed, calculate the virtual amount of rows needed to fill these blocks, and add as many zero rows to A as the difference between the virtual and actual amount of rows. Since you didn't mention, what the actual output should look like (array, cell array, ...), I chose a reshaped array.
Here's the code:
m = 24;
n = 2;
w = 5;
A = rand(m, n)
% Determine number of blocks
n_blocks = ceil(m / w);
% Add zero rows to A
A(m+1:w*n_blocks, :) = 0
% Reshape A into desired format
A = reshape(A.', size(A, 1) / n_blocks * n, n_blocks).'
The output (shortened):
A =
0.9164959 0.1373036
0.5588065 0.1303052
0.4913387 0.6540321
0.5711623 0.1937039
0.7231415 0.8142444
0.9348675 0.8623844
[...]
0.8372621 0.4571067
0.5531564 0.9138423
A =
0.91650 0.13730
0.55881 0.13031
0.49134 0.65403
0.57116 0.19370
0.72314 0.81424
0.93487 0.86238
[...]
0.83726 0.45711
0.55316 0.91384
0.00000 0.00000
A =
0.91650 0.13730 0.55881 0.13031 0.49134 0.65403 0.57116 0.19370 0.72314 0.81424
0.93487 0.86238 0.61128 0.15006 0.43861 0.07667 0.94387 0.85875 0.43247 0.03105
0.48887 0.67998 0.42381 0.77707 0.93337 0.96875 0.88552 0.43617 0.06198 0.80826
0.08087 0.48928 0.46514 0.69252 0.84122 0.77548 0.90480 0.16924 0.82599 0.82780
0.49048 0.00514 0.99615 0.42366 0.83726 0.45711 0.55316 0.91384 0.00000 0.00000
Hope that helps!

How to do SUM on array from outside file?

I'm newbie college student for programming studies,
so recently i have task to calculate matrix from outside files for Gauss Jordan Numeric Method, in the txt file i provide has 10 (x) and (y) data, and declare with do functions to calculate the 10 data from the txt file each for x^2, x^3, x^4, xy, x^2y
my question is : how to SUM (calculate total) each x^2, x^3 ... that was calculated by program ? i try do sum file in below and still got errors (the first argument of sum must not a scalar.)
the Fortran apps i use was Plato cc from Silverfrost.
I apologize if my english bad and my pogram looks funny.
i have 10 data in my txt looks like these :
(x) (y)
12 10
5 6
28 8
9 11
20 17
6 24
32 9
2 7
1 30
26 22
in program below i open these files and want each x and y i provide read and calculate to get x^2, x^3, x^4, xy, x^2y
Program Gauss_Jordan
Real x(10),y(10),xj,yj,xj2,xj3,xj4,xjyj,xj2yj
Open (10, file='Data.txt')
Do j = 1,10
Read(10,*) x(j), y(j)
xj2 = x(j)**2
xj3 = x(j)**3
xj4 = x(j)**4
xjyj = x(j)*y(j)
xj2yj = (x(j)**2)*y(j)
Do k = 1,10
T(xj2) = SUM( xj2, dim=1)
T(xj3) = SUM (xj3, dim=1)
T(xj4) = SUM (xj4, dim=1)
T(xjyj) = SUM (xjyj, dim=1)
T(xj2yj) = SUM (xj2yj, dim=1)
End Do
End Do
Close(10)
End
for T(xj2) I want to get one result scalar result from SUM the all xj^2 that program has been calculated.
Like in excel was expected :
(A) is 1st xj^2 value that has been calculated
.
.
.
until (J) is 10th xj^2 value that has been calculated
sxj^2 = SUM(Xj^2)
SUM (A-J)
The 'sum' intrinsic needs an array argument, which we can compute from the input arrays without using a loop, so your program could be:
Program Gauss_Jordan
Real x(10), y(10), x2(10), x3(10), x4(10), xy(10), x2y(10)
Open(10, file='Data.txt')
Do j = 1, 10
Read (10, *) x(j), y(j)
End Do
Close(10)
x2 = x**2
x3 = x**3
x4 = x**4
xy = x*y
x2y = (x**2)*y
sx2 = SUM(x2)
sx3 = SUM(x3)
sx4 = SUM(x4)
sxy = SUM(xy)
sx2y = SUM(x2y)
End
From what I see I think you are misunderstanding what the SUM intrinsic does. Since your example isn't storing xj2, xj3 etc. in arrays, SUM isn't going to be useful to you. Instead you could declare totals as scalars (as you described you wanted) and simply add the individual xj2 variables in a loop as in the example below.
Also, you should get in the habit of using the implicit none declaration. It will save you from unexpected errors due to spelling mistakes.
Program Gauss_Jordan
implicit none
Real x(10),y(10),xj,yj,xj2,xj3,xj4,xjyj,xj2yj
real :: Txj2,Txj3,Txj4,Txjyj,Txj2yj
integer :: j
Txj2 = 0
Txj3 = 0
Txj4 = 0
Txjyj= 0
Txj2yj= 0
Open (10, file='Data.txt')
Do j = 1,10
Read(10,*) x(j), y(j)
xj2 = x(j)**2
xj3 = x(j)**3
xj4 = x(j)**4
xjyj = x(j)*y(j)
xj2yj = (x(j)**2)*y(j)
Txj2 = Txj2 + xj2
Txj3 = Txj3 + xj3
Txj4 = Txj4 + xj4
Txjyj = Txjyj + xjyj
Txj2yj = Txj2yj + xj2yj
End Do
print *, 'Txj2 = ', Txj2
Close(10)
End
When I ran this I got the output below which is what I believe you intended:
3175

Saving mixed data cell array to ascii file in MATLAB

I get some data from an instrument that is formatted in a specific way. I need to load the data into MATLAB, manipulate some values, then save it back with the same format to load back into the instrument software for further analysis...
The issue I am having is the data is of mixed value types and they are kind of all over the place.
The file is tab delimited, I have added arrows eg --> to show the location of the tabs (like notepad++ does)
Scan-42/01
Temperature [K] :--> 295.00
Time [s] :--> 60
"Linspace"
0.01--> 0.96
0.02--> 0.95
0.03--> 0.95
"Logspace"
0.01--> 0.96
0.02--> 0.95
0.04--> 0.94
The data keeps going down but I have cut it off after 3 rows.
The data I need to manipulate will be the Temperature, and some of the values under Linspace and Logspace.
I am currently importing the data like this:
filename = 'test.asc';
delimiter = '\t';
formatSpec = '%s%s%[^\n\r]';
fileID = fopen(filename,'r');
dataArray = textscan(fileID, formatSpec, 'Delimiter', delimiter, 'ReturnOnError', false);
Data in MATLAB looks like this:
Even if I could set up some kind of template in MATLAB where I could get the values nesessary, and then save them in excactly this format would work fine. The file must be saved as .asc, or the instrument will reject it.
Help is greatly appreciated.
Thanks
Hope this would work for you.
Code
%%// Note: file1 is your input .asc filename and file2 is the output .asc.
%%// Please specify their names before running this.
%%// **** Read in file data ****
fid = fopen(file1,'r');
A = importdata(file1,'\n')
%%// Delimiters (mind these assumptions)
linlog_delim1 = '--> ';
temperature_delim1 = 'Temperature [K] :--> ';
sep1 = cellfun(#(x) isequal(x,''),A)
sep1 = [sep1 ;1]
sep_ind = find(sep1)
full_data = regexp(A,linlog_delim1,'split')
%%// Temperature value
temp_ind = find(~cellfun(#isempty,strfind(A,'Temperature [K] :-->')))
temp_val = str2num(cell2mat(full_data{temp_ind,:}(1,2)))
%%// Linspace values
sep_linspace = cellfun(#(x) isequal(x,'"Linspace"'),A)
lin_start_ind = find(sep_linspace)+1
lin_stop_ind = sep_ind(find(sep_ind>lin_start_ind,1,'first'))-1
linspace_data = vertcat(full_data{lin_start_ind:lin_stop_ind})
linspace_valid_ind = cellfun(#str2num,linspace_data(:,1))
linspace_valid_val = cellfun(#str2num,linspace_data(:,2))
%%// Logspace values
sep_linspace = cellfun(#(x) isequal(x,'"Logspace"'),A)
log_start_ind = find(sep_linspace)+1
log_stop_ind = sep_ind(find(sep_ind>log_start_ind,1,'first'))-1
logpace_data = vertcat(full_data{log_start_ind:log_stop_ind})
logspace_valid_ind = cellfun(#str2num,logpace_data(:,1))
logspace_valid_val = cellfun(#str2num,logpace_data(:,2))
%%// **** Let us modify some data ****
temp_val = temp_val + 10;
linspace_valid_val_mod1 = linspace_valid_val+[1 2 3]'; %%//'
logspace_valid_val_mod1 = logspace_valid_val+[1 20 300]'; %%//'
%%// **** Write back file data ****
%%// Write back temperature data
A(temp_ind) = {[temperature_delim1,num2str(temp_val)]}
%%// Write back linspace data
mod_lin_val = cellfun(#strtrim,cellstr(num2str(linspace_valid_val_mod1)),'uni',0)
mod_lin_ind = cellstr(num2str(linspace_valid_ind))
sep_lin = repmat({linlog_delim1},numel(mod_lin_val),1)
A(lin_start_ind:lin_stop_ind)=cellfun(#horzcat,mod_lin_ind,sep_lin,mod_lin_val,'uni',0)
%%// Write back logspace data
mod_log_val = cellfun(#strtrim,cellstr(num2str(logspace_valid_val_mod1)),'uni',0)
mod_log_ind = cellstr(num2str(logspace_valid_ind))
sep_log = repmat({linlog_delim1},numel(mod_log_val),1)
A(log_start_ind:log_stop_ind)=cellfun(#horzcat,mod_log_ind,sep_log,mod_log_val,'uni',0)
%%// Remove leading whitespaces
A = strtrim(A)
%%// Write the modified data
fid2 = fopen(file2,'w');
for row = 1:numel(A)
fprintf(fid2,'%s\n',A{row,:});
end
fclose(fid);
fclose(fid2);
Changes for the demo:
Temperature has 10 added.
"Linspace" has 1 2 and 3 added to it's elements respectively.
"Logspace" has 1 20 and 300 added to it's elements respectively.
Results
Before -
Scan-42/01
Temperature [K] :--> 295.00
Time [s] :--> 60
"Linspace"
0.01--> 0.96
0.02--> 0.95
0.103--> 0.95
"Logspace"
0.01--> 0.96
0.02--> 0.95
0.04--> 0.94
After -
Scan-42/01
Temperature [K] :--> 305
Time [s] :--> 60
"Linspace"
0.01--> 1.96
0.02--> 2.95
0.103--> 3.95
"Logspace"
0.01--> 1.96
0.02--> 20.95
0.04--> 300.94
Edit 1:
Code
%%// I-O filenames
input_filename = 'gistfile1.txt';
output_file = 'gistfile1_out.txt';
%%// Get data from input filename
delimiter = '\t';
formatSpec = '%s%s%[^\n\r]';
fid = fopen(input_filename,'r');
dataArray = textscan(fid, formatSpec, 'Delimiter', delimiter, 'ReturnOnError', false);
%%// Get data into A
A(:,1) = dataArray{1,1}
A(:,2) = dataArray{1,2}
%%// Find separator indices
ind1 = find([cellfun(#(x) isequal(x,''),A(:,2));1])
temperature_ind = find(~cellfun(#isempty,strfind(A,'Temperature')))
temperature_val = str2num(cell2mat(A(temperature_ind,2)))
%%// Linspace values
sep_linspace = cellfun(#(x) isequal(x,'"Linspace"'),A(:,1))
lin_start_ind = find(sep_linspace)+1
lin_stop_ind = ind1(find(ind1>lin_start_ind,1,'first'))-1
linspace_valid_ind = cellfun(#str2num,A(lin_start_ind:lin_stop_ind,1))
linspace_valid_val = cellfun(#str2num,A(lin_start_ind:lin_stop_ind,2))
%%// Logspace values
sep_logspace = cellfun(#(x) isequal(x,'"Logspace"'),A(:,1))
log_start_ind = find(sep_logspace)+1
log_stop_ind = ind1(find(ind1>log_start_ind,1,'first'))-1
logspace_valid_ind = cellfun(#str2num,A(log_start_ind:log_stop_ind,1))
logspace_valid_val = cellfun(#str2num,A(log_start_ind:log_stop_ind,2))
%%// **** Let us modify some data ****
temp_val_mod1 = temperature_val + 10;
linspace_valid_val_mod1 = linspace_valid_val+[1:numel(linspace_valid_val)]';
logspace_valid_val_mod1 = logspace_valid_val+10.*[1:numel(logspace_valid_val)]';
%%// **** Write back file data into A ****
A(temperature_ind,2) = cellstr(num2str(temp_val_mod1))
A(lin_start_ind:lin_stop_ind,2) = cellstr(num2str(linspace_valid_val_mod1))
A(log_start_ind:log_stop_ind,2) = cellstr(num2str(logspace_valid_val_mod1))
%%// Write the modified data
fid2 = fopen(output_file,'w');
for row = 1:size(A,1)
fprintf(fid2,'%s\t%s\n',A{row,1},A{row,2});
end
%%// Close files
fclose(fid);
fclose(fid2);
Results
Before -
Scan-42/01
Temperature [K] : 295.00
Time [s] : 60
"Linspace"
0.01 0.96
0.02 0.95
0.03 0.95
"Logspace"
0.01 0.96
0.02 0.95
0.04 0.94
After -
Scan-42/01
Temperature [K] : 305
Time [s] : 60
"Linspace"
0.01 1.96
0.02 2.95
0.03 3.95
"Logspace"
0.01 10.96
0.02 20.95
0.04 30.94
Please note that the only formatting difference between input and output files is that there is no whitespaced row between "Linspace" and the previous row in the output file, as was there in the input file. This is seen similarly for "Logspace".
I've solved a nearly identical problem once before. The solution goes something like this:
First, you're already splitting your data up into chunks, so that's good. Judging by your comment, it seems that the data is consistently formatted from file to file, but inconsistently formatted in each individual file. That's fine.
What you need to do is iterate through dataArray, and find each unique label (Such as "Linspace") and track that labels index. What you'll end up with is a vector of indices that tell you exactly where in dataArray these labels appear. Once you have all of the labels indices, you need to look at the dataArray, and see how the data between each label is formatted. Then you'll write some code to break dataArray into sub-arrays. You'll need to write a different sub-array parser for each format.
I know that's a little abstract, so let me try to give you an example.
timeIndex = find(strcmp(dataArray, 'Time'), 1);
linespaceIndex = find(strcmp(dataArray, '"linSpace"'), 1);
logespaceIndex = find(strcmp(dataArray, '"logSpace"'), 1);
linSpaceData = dataArray(linspaceIndex+3:logspaceIndex-1); % This is the "sub-array" I was refering to. It's a little piece of dataArray that contains only the linspace data values.
This is just an example, and will probably not plug-and-play, it's just meant to be a thought-provoker. Note the +3 and -1, those were just guessed. You'll have to empirically determine those for each range, as lings like tabs, colons, and spaces can get in the way. That should be enough to get you started on your problem. Let me know if you need clarification, or if this isn't helpful. Good luck!
-Fletch

How do I make this specific code run faster in Matlab?

I have an array with a set of chronological serial numbers and another source array with random serial numbers associated with a numeric value. The code creates a new cell array in MATLAB with the perfectly chronological serial numbers in one column and in the next column it inserts the associated numeric value if the serial numbers match in both original source arrays. If they don't the code simply copies the previous associated value until there is a new match.
j = 1;
A = {random{1:end,1}};
B = cell2mat(A);
value = random{1,2};
data = cell(length(serial), 1);
data(:,1) = serial(:,1);
h = waitbar(0,'Please Wait...');
steps = length(serial);
for k = 1:length(serial)
[row1, col1, vec1] = find(B == serial{k,1});
tf1 = isempty(vec1);
if (tf1 == 0)
prices = random{col1,2};
data(j,2) = num2cell(value);
j = j + 1;
else
data(j,2) = num2cell(value);
j = j + 1;
end
waitbar(k/steps,h,['Please Wait... ' num2str(k/steps*100) ' %'])
end
close(h);
Right now, the run-time for the code is approximately 4 hours. I would like to make this code run faster. Please suggest any methods to do so.
UPDATE
source input (serial)
1
2
3
4
5
6
7
source input (random)
1 100
2 105
4 106
7 107
desired output (data)
SR No Value
1 100
2 105
3 105
4 106
5 106
6 106
7 107
Firstly, run the MATLAB profiler (see 'doc profile') and see where the bulk of the execution time is occuring.
Secondly, don't update the waitbar on every iteration> Particularly if serial contains a large (> 100) number of elements.
Do something like:
if (mod(k, 100)==0) % update on every 100th iteration
waitbar(k/steps,h,['Please Wait... ' num2str(k/steps*100) ' %'])
end
Some points:
Firstly it would help a lot if you gave us some sample input and output data.
Why do you initialize data as one column and then fill it's second in the loop? Rather initialize it as 2 columns upfront: data = cell(length(serial), 2);
Is j ever different from k, they look identical to me and you could just drop both the j = j + 1 lines.
tf1 = isempty(vec1); if (tf1 == 0)... is the same as the single line: if (!isempty(vec1)) or even better if(isempty(vec1)) and then swap the code from your else and your if.
But I think you can probably find a fast vecotrized solution if you provide some (short) sample input and output data.

Resources