I am trying to loop through a column with 50000 rows. I would like to compare the value in say i with (i+1). The only way I know how to do this is by defining an array. However, there is only one variable i.e. The variables column name e.g. Col but 50000 observations within the column. When I use:
array Transform {50000} Col
where Transform is the name of the array and Col is the column name in my dataset, I receive a subscript error as there are too few variables i.e. Only 1 vs 50000. I have tried replacing {50000} with {50000,1} (and even {*}) so the compiler recognizes that there are 50k observations and only one column. Further I have attempted to transpose the dataset but this seems difficult as I need to add on another variable on to the dataset later which depends on the values of i and (i+1).
Is there a method to loop through the column to compare i and (i+1) using any method (not necessarily an array)? Thanks for the help :)
Example of using LAG:
data input;
infile cards;
input transform;
cards;
3
5
8
12
16
;
run;
data comp;
set input;
transform_change = transform - lag1(transform);
run;
For reversed order of rows:
data input_rownum / view=input_rownum;
set input;
rownum = _N_;
run;
proc sort data=input_rownum out=input_reversed;
by descending rownum;
run;
data comp_reverse;
set input_reversed;
transform_change = transform - lag1(transform);
run;
LAG1 means previous value of the variable. LAG2 is for pre-previous, and so on. Consult the documentation for more.
Arrays work across variables, so aren't suitable for your task here. There's a couple of options for you, given the small number of rows the easiest is probably to just join the dataset on itself, with the row number offset by one. You can then do your comparison.
data want;
merge have have (firstobs=2 rename=(col=col_plus1));
run;
If you only want to compare row i with i+1 you could use the lag function. This pulls the value from the previous row read (beware when using this with loops as not all rows will be processed in a loop)
Related
I would like to set the length of an array depending on what value i obtain from reading a dataset:number which has one variable num with one numeric value. But I am getting an error message: saying that I cannot initiate the probs array. Can i get any suggestion on how to solve this issue? (I really don't want to hardcode the length of the probs array)
data test;
if _n_=1 then do;
set work.number;
i = num +1;
end;
array probs{i} _temporary_ .....
SAS Data step arrays can not be dynamically sized during step run-time.
One common approach is to place the computed number of rows of the data set into a macro variable before the data step.
I'm not sure what you are doing with probs.
What values will be going into the array elements ?
Do you need all prob data while iterating through each row of the data set ?
Is a single result computed from the probs data ?
Example - Compute row count in a data null using nobs set option:
data _null_;
if 0 then set work.number nobs=row_count;
call symputx ('ROW_COUNT', row_count);
stop;
run;
data test;
array probs (&ROW_COUNT.) _temporary_;
* something with probs[index] ;
* maybe fill it ?
do index = 1 by 1 until (last_row);
set work.number;
probs[index] = prob; * prob from ith row;
end;
* do some computation that a procedure isn't able to;
…
result_over_all_data = my_magic; * one result row from processing all prob;
output;
stop;
run;
Of course your actual use of the array will vary.
The many other ways to get row_count include dictionary.table views, sql select count(*) into and a variety of ATTRN invocations.
I have a cell array that I need to split into several matrices so that I can take the sum of subsets of the data. This is a sample of what I have:
A = {'M00.300', '1644.07';...
'M00.300', '9745.42'; ...
'M00.300', '2232.88'; ...
'M00.600', '13180.82'; ...
'M00.600', '2755.19'; ...
'M00.600', '15800.38'; ...
'M00.900', '18088.11'; ...
'M00.900', '1666.61'};
I want the sum of the second columns for each of 'M00.300', 'M00.600', and 'M00.900'. For example, to correspond to 'M00.300' I would want 1644.07 + 9745.42 + 2232.88.
I don't want to just hard code it because each data set is different, so I need the code to work for different size cell arrays.
I'm not sure of the best way to do this, I was going to begin by looping through A and comparing the strings in the first column and creating matrices within that loop, but that sounded messy and not efficient.
Is there a simpler way to do this?
Classic use of accumarray. You would use the first column as an index and the second column as the values associated with each index. accumarray works where you group values that belong to the same index together and you apply a function to those values. In your case, you'd use the default behaviour and sum things together.
However, you'll need to convert the first column into numeric labels. The third output of unique will help you do this. You'll also need to convert the second column into a numeric array and so str2double is a perfect way to do this.
Without further ado:
[val,~,id] = unique(A(:,1)); %// Get unique values and indices
out = accumarray(id, str2double(A(:,2))); %// Aggregate the groups and sum
format long g; %// For better display of precision
T = table(val, out) %// Display on a nice table
I get this:
>> T = table(val, out)
T =
val out
_________ ________
'M00.300' 13622.37
'M00.600' 31736.39
'M00.900' 19754.72
The above uses the table class that is available from R2013b and onwards. If you don't have this, you can perhaps use a for loop and print out each cell and value separately:
for idx = 1 : numel(out)
fprintf('%s: %f\n', val{idx}, out(idx));
end
We get:
M00.300: 13622.370000
M00.600: 31736.390000
M00.900: 19754.720000
I have a big SAS table, let's describe the columns as, A nd B columns in character format and all other columns are vairable in numerical format (every variable has a different name) with unknow amounth length N, like:
A B Name1 Name2 Name3 .... NameN
-------------------------------------------------
Char Char Number1 Number2 Number3 ..... NumberN
.................................................
.................................................
The goal is that the numerical array Name1-NameN will sum up downward through the Class=B (By B),
So the final table will look like this:
A B Name1 Name2 Name3 .... NameN
----------------------------------------
Char Char Sum1 Sum2 Sum3 ..... SumN
........................................
........................................
To do this sum-up, I described 2 arrays. The first one is:
array Varr {*} _numeric_; /* it reads only numerical columns */
Then I described another array with the same length (Summ1-SummN) to do the sum-up process.
The thing is that I can only describe the length of this new array manually. For example, if there are 80 numerical values, then I have to write manually like:
array summ {80} Summ1-Summ80;
The code works when I write it manually. But instead I want to write something like
array summ {&N} Summ1-Summ&N; /* &N is the dimension of the array Varr */
I tried with do-loop and dim(Varr) under the array in many different ways like:
data want;
array Varr {*} _numeric_;
do i=1 to dim(Varr);
N+1 ;
end;
%put &N;
array Summ {&N} Summ1-Summ&N;
retain Summ;
if first.B then do i=1 to dim(varr); summ(i)=varr(i) ;end;
else do i =1 to dim(varr); summ(i) = summ(i) + varr(i) ; varr(i)=summ(i); end;
drop Summ1-Summ&N;
run;
But it doesn't work. Any idea about how to bring the length of the first array to the second array?
You need to calculate and store the number of numeric variables in a previous step. The easiest way is to use the dictionary.columns metadata table, available in proc sql. This contains all column details for a given dataset, including the type (num or char), you therefore just need to count the number of columns where the type is 'num'.
The code below does just that and stores the result in a macro variable, &N. using the into : functionality. I've also used the functions left and put to remove leading blanks from the macro variable, otherwise you'll encounter problems when putting summ1-summ&N.
I've also added a 2nd solution based on your answer, but will be more efficient as it doesn't read in any records, only the column details
proc sql noprint;
select left(put(count(*),best12.)) into :N
from dictionary.columns
where libname='SASHELP' and memname='CLASS' and type='num';
quit;
%put Numeric variables = &N.;
/*****************************************/
/* alternative solution */
data _null_;
set sashelp.class (obs=0);
array temp{*} _numeric_;
call symputx('N',dim(temp));
run;
%put Numeric variables = &N.;
Now I found another solution with a little modification of the solution from #kl78
Before when I tried with call symput ('N',dim(varr)); I forgot to change the numeric format and to remove the uneccessary spaces. When I run it without format, the code tried to find Summ_____87, so it gave error.
Now I run it with format, call symput ('N',put(dim(varr),2.)); the code can find Summ87, so it is totally sucessfull now.
I'm quite new to MatLab and this problem really drives me insane:
I have a huge array of 2 column and about 31,000 rows. One of the two columns depicts a spatial coordinate on a grid the other one a dependent parameter. What I want to do is the following:
I. I need to split the array into smaller parts defined by the spatial column; let's say the spatial coordinate are ranging from 0 to 500 - I now want arrays that give me the two column values for spatial coordinate 0-10, then 10-20 and so on. This would result in 50 arrays of unequal size that cover a spatial range from 0 to 500.
II. Secondly, I would need to calculate the average values of the resulting columns of every single array so that I obtain per array one 2-dimensional point.
III. Thirdly, I could plot these points and I would be super happy.
Sadly, I'm super confused since I miserably fail at step I. - Maybe there is even an easier way than to split the giant array in so many small arrays - who knows..
I would be really really happy for any suggestion.
Thank you,
Arne
First of all, since you wish a data structure of array of different size you will need to place them in a cell array so you could try something like this:
res = arrayfun(#(x)arr(arr(:,1)==x,:), unique(arr(:,1)), 'UniformOutput', 0);
The previous code return a cell array with the array splitted according its first column with #(x)arr(arr(:,1)==x,:) you are doing a function on x and arrayfun(function, ..., 'UniformOutput', 0) applies function to each element in the following arguments (taken a single value of each argument to evaluate the function) but you must notice that arr must be numeric so if not you should map your values to numeric values or use another way to select this values.
In the same way you could do
uo = 'UniformOutput';
res = arrayfun(#(x){arr(arr(:,1)==x,:), mean(arr(arr(:,1)==x,2))), unique(arr(:,1)), uo, 0);
You will probably want to flat the returning value, check the function cat, you could do:
res = cat(1,res{:})
Plot your data depends on their format, so I can't help if i don't know how the data are, but you could try to plot inside a loop over your 'res' variable or something similar.
Step I indeed comes with some difficulties. Once these are solved, I guess steps II and III can easily be solved. Let me make some suggestions for step I:
You first define the maximum value (maxValue = 500;) and the step size (stepSize = 10;). Now it is possible to iterate through all steps and create your new vectors.
for k=1:maxValue/stepSize
...
end
As every resulting array will have different dimensions, I suggest you save the vectors in a cell array:
Y = cell(maxValue/stepSize,1);
Use the find function to find the rows of the entries for each matrix. At each step k, the range of values of interest will be (k-1)*stepSize to k*stepSize.
row = find( (k-1)*stepSize <= X(:,1) & X(:,1) < k*stepSize );
You can now create the matrix for a stepk by
Y{k,1} = X(row,:);
Putting everything together you should be able to create the cell array Y containing your matrices and continue with the other tasks. You could also save the average of each value range in a second column of the cell array Y:
Y{k,2} = mean( Y{k,1}(:,2) );
I hope this helps you with your task. Note that these are only suggestions and there may be different (maybe more appropriate) ways to handle this.
I have a 60,000-by-2 array. The first column is data 1 and second column is data 2; both of equal length. I'm not sure how to properly write the syntax to compare data 1 to data 2, and if data 1 is larger than data 2 then write that to the third column. Or vice versa if data 2 is larger than data 1. I have begun constructing a for loop, but I'm having syntax issues comparing the columns.
No loops are needed. If you simply want to create a vector containing the largest element in each row of your 60,000-by-2 matrix you can use the max function:
A = rand(6e4,2); % Random demo data
B = max(A,[],2);
Or if you then want to put the result directly in a third column of A:
A(:,3) = max(A,[],2);
Read the documentation for max. You'll see that the 2 in the third argument applied the max function across each row of the input, A.