I have a SAS code that works fine to read to datasets and merge them. The data sets are named according to the quarter and year of the data, e.g.: "data1_Q11999" and "data2_Q11999". The code I use to do this is below.
Now I want to loop over several of this datasets by increasing the year from 1999 to 2014 and the quarter from 1 to 4 (i.e. two loops).
My understanding is that I need to create a macro to do this, but I am having some issues with the syntax.
The code is below. I tried to wrap the code around a %macro statement with do loops but keep getting a bunch of syntax errors. Is there a straightforward way of achieving this?
data origfile;
infile "D:/data1_Q11999.txt" dlm= '|' MISSOVER DSD lrecl=32767 firstobs=1 ;
input
fico : 8.
dt_first_pi : 8.
id : $16.
run;
data svcgfile;
infile "D:/data2_Q11999.txt" dlm= '|' MISSOVER DSD lrecl=32767 firstobs=1 ;
input
id : $12.
Period : 8.
actual_loss : 12.
;
run;
PROC SORT DATA=origfile OUT=origfile;
BY id;
RUN;
PROC SORT DATA=svcgfile OUT=svcgfile;
BY id;
RUN;
DATA mergedata;
MERGE origfile svcgfile;
BY id;
RUN;
Assuming that you want to generate a separated merged file for year quarter you could use a macro like this.
%macro read(first_yr,last_yr);
%local year qtr;
%do year=&first_yr %to &last_yr ;
%do qtr=1 %to 4 ;
data data1;
infile "D:\data1_Q&qtr.&year..txt" dsd dlm= '|' truncover ;
length id $16 fico dt_first_pi 8 ;
input fico dt_first_pi id ;
run;
proc sort data=data1; by id; run;
data data2;
infile "D:\data2_Q&qtr.&year..txt" dsd dlm= '|' truncover ;
length id $16 period actual_loss 8 ;
input id period actual_loss ;
run;
proc sort data=data2; by id; run;
data result_q&qtr.&year. ;
merge data1 data2 ;
by id;
run;
%end;
%end;
%mend read ;
Then you could call it like this to generate 64 separate datasets.
%read(1999,2014)
But you probably will really want to have those 64 dataset combined into one so that you can use it more easily for your next steps. You could probably fix the process that reads the data to generate it all at once, but here is a simple data step to combine any dataset that starts with RESULT_ like the macro above generates and combine them into a single dataset.
data want ;
length year qtr 8 dsname $41 ;
set result_: indsname=dsname ;
year = input(substr(scan(dsname,-1,'.'),9),4.);
qtr = input(substr(scan(dsname,-1,'.'),8),1.);
run;
Related
Can anyone help with this issue I'm having where the macro is only taking the final row value of the data?
I have some data that looks like this:
data data1 ;
infile datalines dsd dlm='|' truncover;
input id :$2. year_age_15 EDU_2000 EDU_2001 EDU_2002 ;
datalines4;
10|2000|3|4|5
11|2000|5|5|6
12|2001|1|2|3
13|2002|5|5|6
14|2001|2|2|2
15|2000|3|3|4
;;;;
However I need it to use the year variable to determine which data to keep, and then change all the values for the years after that value to missing, like so:
data data1 ;
infile datalines dsd dlm='|' truncover;
input id :$2. year_age_15 EDU_2000 EDU_2001 EDU_2002 ;
datalines4;
10|2000|3|.|.
11|2000|5|.|.
12|2001|1|2|.
13|2002|5|5|6
14|2001|2|2|.
15|2000|3|.|.
;;;;
I've been trying to get this macro to work, but it only works intermittently and works just for the final row of the data rather than looping through the rows.
%macro macro2 (output=, input=);
data &output;
set &input;
%DO I = 1 %TO 6;
%do; call symput('value2',trim(left(put(year_age_15,8.))));
temp_col=&value2.;
%let year_end=&value2.;
%put YEAR END IS: &year_end.;
%put EDU YEAR IS: EDU_&year_end.;
%do year = &year_end. %TO 2002;
%put &year.;
EDU_&year.=.;
%end;
%end;
%end;
run;
%MEND macro2;
%macro1(input=testset, output=output_testset);
In R it could be something simple like :
for(i in 1:6){.
do this
}
Any advice? I can't figure out which bit is going wrong, thanks!
So, I think the issue here is your data is at the wrong level. You certainly can do what Reeza suggests, and I think it's probably reasonable to do so, but the reason why this is a bit complicated is that you have data in your variable name. That's not a best practice - your variable name should be "education" and your data should have a row for each year. Then this would be a simple WHERE statement!
Here's a simple PROC TRANSPOSE that turns it to the right structure, and then if you really need it the other way, a second one will turn it back. The where statement can be in the proc transpose or could be used somewhere else.
proc transpose data=data1 out=data_t (where=(year_Age_15 ge input(scan(_NAME_,2,'_'),4.)));
by id year_Age_15;
var edu_:;
run;
proc transpose data=data_t out=want;
by id year_age_15;
id _name_;
var col1;
run;
Create an array and index it by years rather than default 1:n
Loop through your array starting at year+1 and set to missing
data want;
set data1;
array educ(2000:2002) edu_2000-edu_2002;
if (year_age_15 +1) <= hbound(educ) then do i= (year_age_15 +1) to hbound(educ);
call missing(educ(i));
end;
run;
As #Joe mentions, the year to match is part of a variable name, which is tremor inducing 'data in the metadata'
You can use the VNAME to retrieve the variable name of an index accessed array element. Use that feature to compare to expected variable name whilst looping over a variable array based on variables named EDU*.
Example:
data have ;
infile datalines dsd dlm='|' truncover;
input id :$2. year_age_15 EDU_2000 EDU_2001 EDU_2002 ;
datalines4;
10|2000|3|4|5
11|2000|5|5|6
12|2001|1|2|3
13|2002|5|5|6
14|2001|2|2|2
15|2000|3|3|4
;;;;
data want;
set have;
array edus edu_:;
* find index of element corresponding to variable name having year;
do _n_ = 1 to dim(edus) until (upcase(vname(edus(_n_))) = cats('EDU_',year_age_15));
end;
* fill in elements at indices post the found one with missing values;
do _n_ = _n_+1 to dim(edus);
call missing(edus(_n_));
end;
run;
Is there any form to keep variables with a doop loop in data step?
will be something as:
data test;
input id aper_f_201501 aper_f_201502 aper_f_201503 aper_f_201504
aper_f_201505 aper_f_201506;
datalines;
1 0 1 2 3 5 7
2 -1 5 4 8 7 9
;
run;
%macro test;
%let date = '01Jul2015'd;
data test2;
set test(keep=do i = 1 to 3;
aper_f_%sysfunc(intnx(month,&date,-i,begin),yymmn6.);
end;)
run;
%mend;
%test;
I need to iterate several dates.
Thank you very much.
You need to use macro %do loop instead of the data step do loop, which is not going to be valid in the middle of a dataset option. Also do not generate those extra semi-colons into the middle of your dataset options. And do include a semi-colon to end your SET statement.
%macro test;
%local i date;
%let date = '01Jul2015'd;
data test2;
set test(keep=
%do i = 1 %to 3;
aper_f_%sysfunc(intnx(month,&date,-i,begin),yymmn6.)
%end;
);
run;
%mend;
%test;
You can use the colon shortcut to reference variables with the same prefix, anything in front of the colon will be kept.
keep ID aper_f_2015: ;
There's also a hyphen when you have sequential lists
keep ID aper_f_201501-aper_f_201512;
You can use a macro but not sure it adds a lot of value here.
I have a bunch of character variables which I need to sort out from a large dataset. The unwanted variables all have entries that are the same or are all missing (meaning I want to drop these from the dataset before processing the data further). The data sets are very large so this cannot be done manually, and I will be doing it a lot of times so I am trying to create a macro which will do just this. I have created a list macro variable with all character variables using the following code (The data for my part is different but I use the same sort of code):
data test;
input Obs ID Age;
datalines;
1 2 3
2 2 1
3 2 2
4 3 1
5 3 2
6 3 3
7 4 1
8 4 2
run;
proc contents
data = test
noprint
out = test_info(keep=name);
run;
proc sql noprint;
select name into : testvarlist separated by ' ' from test_info;
quit;
My idea is then to just use a data step to drop this list of variables from the original dataset. Now, the problem is that I need to loop over each variable, and determine if the observations for that variable are all the same or not. My idea is to create a macro that loops over all variables, and for each variable counts the occurrences of the entries. Since the length of this table is equal to the number of unique entries I know that the variable should be dropped if the table is of length 1. My attempt so far is the following code:
%macro ListScanner (org_list);
%local i next_name name_list;
%let name_list = &org_list;
%let i=1;
%do %while (%scan(&name_list, &i) ne );
%let next_name = %scan(&name_list, &i);
%put &next_name;
proc sql;
create table char_occurrences as
select &next_name, count(*) as numberofoccurrences
from &name_list group by &next_name;
select count(*) as countrec from char_occurrences;
quit;
%if countrec = 1 %then %do;
proc sql;
delete &next_name from &org_list;
quit;
%end;
%let i = %eval(&i + 1);
%end;
%mend;
%ListScanner(org_list = &testvarlist);
Though I get syntax errors, and with my real data I get other kinds of problems with not being able to read the data correctly but I am taking one step at a time. I am thinking that I might overcomplicate things so if anyone has an easier solution or can see what might be wrong to I would be very grateful.
There are many ways to do this posted around.
But let's just look at the issues you are having.
First for looping through your space delimited list of names it is easier to let the %do loop increment the index variable for you. Use the countw() function to find the upper bound.
%do i=1 %to %sysfunc(countw(&name_list,%str( )));
%let next_name = %scan(&name_list,&i,%str( ));
...
%end;
Second where is your input dataset in your SQL code? Add another parameter to your macro definition. Where to you want to write the dataset without the empty columns? So perhaps another parameter.
%macro ListScanner (dsname , out, name_list);
%local i next_name sep drop_list ;
Third you can use a single query to count all of variables at once. Just use count( distinct xxxx ) instead of group by.
proc sql noprint;
create table counts as
select
%let sep=;
%do i=1 %to %sysfunc(countw(&name_list,%str( )));
%let next_name = %scan(&name_list,&i,%str( ));
&sep. count(distinct &next_name) as &next_name
%let sep=,;
%end;
from &dsname
;
quit;
So this will get a dataset with one observation. You can use PROC TRANSPOSE to turn it into one observation per variable instead.
proc transpose data=counts out=counts_tall ;
var _all_;
run;
Now you can just query that table to find the names of the columns with 0 non-missing values.
proc sql noprint ;
select _name_ into :drop_list separated by ' '
from counts_tall
where col1=0
;
quit;
Now you can use the new DROP_LIST macro variable.
data &out ;
set &dsname ;
drop &drop_list;
run;
So now all that is left is to clean up after your self.
proc delete data=counts counts_tall ;
run;
%mend;
As far as your specific initial question, this is fairly straightforward. Assuming &testvarlist is your macro variable containing the variables you are interested in, and creating some test data in have:
%let testvarlist=x y z;
data have;
call streaminit(7);
do id = 1 to 1e6;
x = floor(rand('Uniform')*10);
y = floor(rand('Uniform')*10);
z = floor(rand('Uniform')*10);
if x=0 and y=4 and z=7 then call missing(of x y z);
output;
end;
run;
data want fordel;
set have;
if min(of &testvarlist.) = max(of &testvarlist.)
and (cmiss(of &testvarlist.)=0 or missing(min(of &testvarlist.)))
then output fordel;
else output want;
run;
This isn't particularly inefficient, but there are certainly better ways to do this, as referenced in comments.
I have a data set that needs to be blown out a certain number of rows according to a dynamic value. Take the dataset below for example:
DATA HAVE;
LENGTH ID $3 COUNT 3;
INPUT ID $ COUNT;
DATALINES;
A 4
B 3
C 1
D 2
;
RUN;
ID=A needs to be blown out 4 rows, ID=B needs to be blown out 3 rows, etc. The resulting dataset would look as such (minus a bunch of other variables I have):
A 1
A 2
A 3
A 4
B 1
B 2
B 3
C 1
D 1
D 2
The following code works to an extent, but I'm having trouble dynamically setting the &COUNT. macro. I tried to insert a CALL SYMPUTX("COUNT",COUNT) statement so that as it loops over each row, the count is placed into the macro and the row is blown at that number of rows.
** THIS CODE ONLY WORKS IF YOU SET COUNT= TO SOME VALUE **;
%MACRO LOOPOVER();
DATA WANT; SET HAVE;
DO UNTIL(LAST.ID);
BY ID;
%DO I=1 %TO &COUNT.;
COUNT = &I.; OUTPUT;
%END;
END;
RUN;
%MEND;
%LOOPOVER;
** THIS CODE DOESN'T WORK BUT I'M NOT SURE WHY?? **;
%MACRO LOOPOVER();
DATA WANT; SET HAVE;
DO UNTIL(LAST.ID);
BY ID;
CALL SYMPUTX("COUNT",COUNT); /* NEW LINE HERE */
%DO I=1 %TO &COUNT.;
COUNT = &I.; OUTPUT;
%END;
END;
RUN;
%MEND;
%LOOPOVER;
It is unnecessary to use macro.
data want(rename=(_count=count));
set have;
do i=1 to count;
_count=i;
output;
end;
drop count;
run;
I have lots of tables which I would like to sort with Proc Sort. (The names of the tables are written in a text file.) To avoid repeating the same code all over again I have tried creating a macro that would import the text file, create an array consisting of those table names and finally sort all the tables. However, I came across a few problems. In Python, I would easily be able to loop through an array. But in SAS, I am not sure how to do it.
%MACRO SORT_TABLES();
PROC IMPORT
DATAFILE = 'TABLES_LIST.txt'
OUT = WORK.TABLES_LIST (RENAME = VAR1 = TABLE_NAME)
DBMS = TAB
REPLACE;
GETNAMES = NO;
QUIT;
/* GET THE LIST OF TABLE NAMES: */
PROC SQL NOPRINT;
SELECT
DISTINCT TABLE_NAME
INTO :TABLEVAR1 - :TABLEVAR&SYSMAXLONG
FROM
WORK.TABLES_LIST;
QUIT;
DATA _NULL_;
ARRAY TABLE_NAMES $ &TABLEVAR1 - &TABLEVAR&SYSMAXLONG;
RUN;
%DO %OVER TABLE_NAMES
PROC SORT
DATA = &TABLEVAR1 /* how can I iterate here???? */
OUT = 'WORK.'||&TABLEVAR1;
BY A B C;
QUIT;
%END;
%MEND;
Just use an iterative %DO loop to loop over your "array" of macro variables.
proc sql noprint ;
select distinct table_name
into :tablevar1 -
from table_list
;
quit;
%do i=1 %to &sqlobs ;
proc sort data=&&tablevar&i ; by _all_ ; run;
%end;
But you don't need a macro for this. There are easier ways to generate code.
filename code temp;
data _null_;
set table_list ;
put 'PROC SORT DATA = ' table_name '; BY _all_; run;' ;
run;
%include code / source2 ;