I have a dataset where I have different names in one column, the names can be duplicate. My task here is to compare each and every name with the rest of the names in the column.For example if I take the name 1 "Vishal" I have to compare it with all the names from 2 to 13. If there is a matching name from row 2 to 13 there will be different column made "flag" with value of Y if there is a duplicate if no duplicate then a value of N.I have to perform this operation with all the names in the group
I have written a code which looks like this:
data Name;
input counter name $50.;
cards;
1 vishal
2 swati
3 sahil
4 suman
5 bindu
6 bindu
7 vishal
8 tushar
9 sahil
10 swati
11 gudia
12 priyansh
13 priyansh
;
proc sql;
select count(name) into: n from swati;
quit;
proc sql;
select name into: name1 -:name13 from swati;
quit;
options mlogic mprint symbolgen;
%macro swati;
data name1;
set swati;
%do i = 1 %to 1;
%do j= %eval(&i.+1) %to &n.;
if &&name&i. =&&name&j. then flag="N";
else flag="Y";
%end;
%end;
run;
%mend;
%swati;
the code gives me the vale N for all the names even if there is a name matching, also it makes a different variable with using all the variable names.*
The desired output is shown below
Name Flag
vishal N
swati N
sahil N
suman Y
bindu N
bindu Y
vishal Y
tushar Y
sahil Y
swati Y
gudia Y
priyansh N
priyansh Y
So basically we started finding vishal (the first name) from 2 to 13 and see if there is a duplicate, if there is the flag is N i.e. there is a duplicate. Let us see the name "Suman" which is the fourth name in the list, and we start searching for its matching from 5 to 13. Since there isn't any duplicate for that we have flagged it as "Y".
WE HAVE TO DO THIS USING A DO LOOP
Sort data by Name
Use a data step with BY to identify duplicates
Resort by Order if desired
proc sort data=name;
by name;
run;
data want;
set name;
by name;
if first.name and last.name then unique='Y';
else unique='N';
run;
proc sort data=want;
by counter;
run;
Your answer for the last observation does not look right. Is there another condition such that if it is the last record the flag should be 'N' instead of 'Y'?
I really see no reason why you have to use a DO loop. But you could place a DO loop around a SET statement with the POINT= option to look for matching names.
data want ;
set name nobs=nobs ;
length next $50;
next=' ';
do p=_n_+1 to nobs until (next=name) ;
set name(keep=name rename=(name=next)) point=p;
end;
if next=name then flag='N'; else flag='Y';
drop next;
run;
You could also take advantage of the COUNTER variable and do it using GROUP BY in a SELECT statement in PROC SQL.
proc sql ;
create table want2 as
select *
, case when (counter = max(counter)) then 'Y' else 'N' end as flag
from name
group by name
order by counter
;
quit;
Related
I have the following data
DATA HAVE;
input yr_2001 yr_2002 yr_2003 area;
cards;
1 1 1 3
0 1 0 4
0 0 1 3
1 0 1 6
0 0 1 4
;
run;
I want to do the following proc freq for variable yr_2001 to yr_2003.
proc freq data=have;
table yr_2001*area;
where yr_2001=1;
run;
Is there a way I can do it without having to repeat it for each year, may be using a loop for proc freq??
Two ways:
1. Transpose it
Add a counter variable to your data, n, and transpose it by n area, then only keep values where the year flag is equal to 1. Because we set an index on the transposed group year, we do not need to re-sort it before doing by-group processing.
data have2;
set have;
n = _N_;
run;
proc transpose data=have
name=year
out=have2_tpose(rename = (COL1 = year_flag)
where = (year_flag = 1)
index = (year)
drop = n
);
by n area;
var yr_:;
run;
proc freq data=have2_tpose;
by year;
table area;
run;
2. Macro loop
Since they all start with yr_, it will be easy to get all the variable names from dictionary.columns and loop over all the variables. We'll use SQL to read the names into a |-separated list and loop over that list.
proc sql noprint;
select name
, count(*)
into :varnames separated by '|'
, :nVarnames
from dictionary.columns
where memname = 'HAVE'
AND libname = 'WORK'
AND name LIKE "yr_%"
;
quit;
/* Take a look at the variable names we found */
%put &varnames.;
/* Loop over all words in &varnames */
%macro freqLoop;
%do i = 1 %to &nVarnames.;
%let varname = %scan(&varnames., &i., |);
title "&varname.";
proc freq data=have;
where &varname. = 1;
table &varname.*area;
run;
title;
%end;
%mend;
%freqLoop;
So I have a dataset with one primary key: unique_id and 1200 variables. This dataset is generated from a macro so the number of columns will not be fixed. I need to split this dataset into 4 or more datasets of 250 variables each, and each of these smaller datasets should contain the primary key so that I can merge them back later. Can somebody help me with either a sas function or a macro to solve this?
Thanks in advance.
A simple way to split a datasets in the way you request is to use a single data step with multiple output datasets where each one has a KEEP= dataset option listing the variables to keep. For example:
data split1(keep=Name Age Height) split2(keep=Name Sex Weight);
set sashelp.class;
run;
So you need to get the list of variables and group then into sets of 250 or less. Then you can use those groupings to generate code like above. Here is one method using PROC CONTENTS to get the list of variables and CALL EXECUTE() to generate the code.
I will use macro variables to hold the name of the input dataset, the key variable that needs to be kept on each dataset and maximum number of variables to keep in each dataset.
So for the example above those macro variable values would be:
%let ds=sashelp.class;
%let key=name;
%let nvars=2;
So use PROC CONTENTS to get the list of variable names:
proc contents data=&ds noprint out=contents; run;
Now run a data step to split them into groups and generate a member name to use for the new split dataset. Make sure not to include the KEY variable in the list of variables when counting.
data groups;
length group 8 memname $41 varnum 8 name $32 ;
group +1;
memname=cats('split',group);
do varnum=1 to &nvars while (not eof);
set contents(keep=name where=(upcase(name) ne %upcase("&key"))) end=eof;
output;
end;
run;
Now you can use that dataset to drive the generation of the code:
data _null_;
set groups end=eof;
by group;
if _n_=1 then call execute('data ');
if first.group then call execute(cats(memname,'(keep=&key'));
call execute(' '||trim(name));
if last.group then call execute(') ');
if eof then call execute(';set &ds;run;');
run;
Here are results from the SAS log:
NOTE: CALL EXECUTE generated line.
1 + data
2 + split1(keep=name
3 + Age
4 + Height
5 + )
6 + split2(keep=name
7 + Sex
8 + Weight
9 + )
10 + ;set sashelp.class;run;
NOTE: There were 19 observations read from the data set SASHELP.CLASS.
NOTE: The data set WORK.SPLIT1 has 19 observations and 3 variables.
NOTE: The data set WORK.SPLIT2 has 19 observations and 3 variables.
Just another way of doing it using macro variables:
/* Number of columns you want in each chunk */
%let vars_per_part = 250;
/* Get all the column names into a dataset */
proc contents data = have out=cols noprint;
run;
%macro split(part);
/* Split the columns into 250 chunks for each part and put it into a macro variable */
%let fobs = %eval((&part - 1)* &vars_per_part + 1);
%let obs = %eval(&part * &vars_per_part);
proc sql noprint;
select name into :cols separated by " " from cols (firstobs = &fobs obs = &obs) where name ~= "uniq_id";
quit;
/* Chunk up the data only keeping those varaibles and the uniq_id */
data want_part∂
set have (keep = &cols uniq_id);
run;
%mend;
/* Run this from 1 to whatever the increment required to cover all the columnns */
%split(1);
%split(2);
%split(3);
this is not a complete solution but some help to give you another insight into how to solve this. The previous solutions have relied much on proc contents and data step, but I would solve this using proc sql and dictionary.columns. And I would create a macro that would split the original file into as many parts as needed, 250 cols each. The steps roughly:
proc sql; create table as _colstemp as select * from dictionary.columns where library='your library' and memname = 'your table' and name ne 'your primary key'; quit;
Count the number of files needed somewhere along:
proc sql;
select ceil(count(*)/249) into :num_of_datasets from _colstemp;
select count(*) into :num_of_cols from _colstemp;
quit;
Then just loop over the original dataset like:
%do &_i = 1 %to &num_of_datasets
proc sql;
select name into :vars separated by ','
from _colstemp(firstobs=%eval((&_i. - 1)*249 + 1) obs = %eval(min(249,&num_of_cols. - &_i. * 249)) ;
quit;
proc sql;
create table split_&_i. as
select YOUR_PRIMARY_KEY, &vars from YOUR_ORIGINAL_TABLE;
quit;
%end;
Hopefully this gives you another idea. The solution is not tested, and may contain some pseudocode elements as it's written from my memory of doing things. Also this is void of macro declaration and much of parametrization one could do.. This would make the solution more general (parametrize your number of variables for each dataset, your primary key name, and your dataset names for example.
I am trying to create an array of strings and want to insert a value in it, if it does not exist already in the array.
I read somewhere that we can use 'IN' operator with Array. So, coded it as follows:
DATA WANT;
SET HAVE;
BY ID;
ARRAY R_PROS_SCRN_ID {2} $4. R_PROS_SCRN_ID_1 - R_PROS_SCRN_ID_2;
RETAIN R_PROS_SCRN_ID_1 - R_PROS_SCRN_ID_2;
IF NOT PROS_SCRN_ID IN R_PROS_SCRN_ID THEN DO;
DO I=1 to 2 ;
IF MISSING( R_PROS_SCRN_ID{i}) THEN DO;
R_PROS_SCRN_ID{i} = PROS_SCRN_ID;
LEAVE;
END;
END;
END;
IF LAST.ID THEN OUTPUT;
RUN;
In Array R_PROS_SCRN_ID, I want only the unique values from field PROS_SCRN_ID.
It is throwing error:
NOTE: Invalid numeric data, PROS_SCRN_ID='MED' , at line 17352 column 201.
I think it is because I did not initialize the Array before comparing and hence it is considering it as Numeric Array. But, I have specified the format as $4. Why is it throwing error?
Also, I am not sure if this is the best way get unique values in an Array. Is there any better way to implement this?
Your code appears to be collecting unique values by group, pivoting from a tall data structure to a wide data structure.
One of the clearest DATA step ways is to use what we call DOW loop in which SET is within the loop. This sample code presumes no more than 10 unique satellite values per group. (The by variables can be thought of as key variables, and all other variables would be satellites)
data have;
input user_id screen_id ;
datalines;
1 1
1 2
1 1
1 1
1 1
1 3
2 1
2 1
2 1
3 0
4 1
4 2
4 3
5 11
5 11
5 11
5 5
5 1
5 5
5 6
5 1
run;
data want;
_index = 0;
do until (last.user_id);
set have;
by user_id;
array ids screen_id1-screen_id10;
if screen_id not in ids then do;
_index + 1;
ids(_index) = screen_id;
end;
end;
drop _index screen_id;
run;
One of the clearest procedural ways is to select the unique values and transpose them.
proc sql;
create view uniqueScreenByUser as
select distinct user_id, screen_id
from have
order by user_id
;
proc transpose data=uniqueScreenByUser prefix=screen_id out=wantWide(drop=_name_);
by user_id;
var screen_id;
run;
Is there any form to keep variables with a doop loop in data step?
will be something as:
data test;
input id aper_f_201501 aper_f_201502 aper_f_201503 aper_f_201504
aper_f_201505 aper_f_201506;
datalines;
1 0 1 2 3 5 7
2 -1 5 4 8 7 9
;
run;
%macro test;
%let date = '01Jul2015'd;
data test2;
set test(keep=do i = 1 to 3;
aper_f_%sysfunc(intnx(month,&date,-i,begin),yymmn6.);
end;)
run;
%mend;
%test;
I need to iterate several dates.
Thank you very much.
You need to use macro %do loop instead of the data step do loop, which is not going to be valid in the middle of a dataset option. Also do not generate those extra semi-colons into the middle of your dataset options. And do include a semi-colon to end your SET statement.
%macro test;
%local i date;
%let date = '01Jul2015'd;
data test2;
set test(keep=
%do i = 1 %to 3;
aper_f_%sysfunc(intnx(month,&date,-i,begin),yymmn6.)
%end;
);
run;
%mend;
%test;
You can use the colon shortcut to reference variables with the same prefix, anything in front of the colon will be kept.
keep ID aper_f_2015: ;
There's also a hyphen when you have sequential lists
keep ID aper_f_201501-aper_f_201512;
You can use a macro but not sure it adds a lot of value here.
I have a bunch of character variables which I need to sort out from a large dataset. The unwanted variables all have entries that are the same or are all missing (meaning I want to drop these from the dataset before processing the data further). The data sets are very large so this cannot be done manually, and I will be doing it a lot of times so I am trying to create a macro which will do just this. I have created a list macro variable with all character variables using the following code (The data for my part is different but I use the same sort of code):
data test;
input Obs ID Age;
datalines;
1 2 3
2 2 1
3 2 2
4 3 1
5 3 2
6 3 3
7 4 1
8 4 2
run;
proc contents
data = test
noprint
out = test_info(keep=name);
run;
proc sql noprint;
select name into : testvarlist separated by ' ' from test_info;
quit;
My idea is then to just use a data step to drop this list of variables from the original dataset. Now, the problem is that I need to loop over each variable, and determine if the observations for that variable are all the same or not. My idea is to create a macro that loops over all variables, and for each variable counts the occurrences of the entries. Since the length of this table is equal to the number of unique entries I know that the variable should be dropped if the table is of length 1. My attempt so far is the following code:
%macro ListScanner (org_list);
%local i next_name name_list;
%let name_list = &org_list;
%let i=1;
%do %while (%scan(&name_list, &i) ne );
%let next_name = %scan(&name_list, &i);
%put &next_name;
proc sql;
create table char_occurrences as
select &next_name, count(*) as numberofoccurrences
from &name_list group by &next_name;
select count(*) as countrec from char_occurrences;
quit;
%if countrec = 1 %then %do;
proc sql;
delete &next_name from &org_list;
quit;
%end;
%let i = %eval(&i + 1);
%end;
%mend;
%ListScanner(org_list = &testvarlist);
Though I get syntax errors, and with my real data I get other kinds of problems with not being able to read the data correctly but I am taking one step at a time. I am thinking that I might overcomplicate things so if anyone has an easier solution or can see what might be wrong to I would be very grateful.
There are many ways to do this posted around.
But let's just look at the issues you are having.
First for looping through your space delimited list of names it is easier to let the %do loop increment the index variable for you. Use the countw() function to find the upper bound.
%do i=1 %to %sysfunc(countw(&name_list,%str( )));
%let next_name = %scan(&name_list,&i,%str( ));
...
%end;
Second where is your input dataset in your SQL code? Add another parameter to your macro definition. Where to you want to write the dataset without the empty columns? So perhaps another parameter.
%macro ListScanner (dsname , out, name_list);
%local i next_name sep drop_list ;
Third you can use a single query to count all of variables at once. Just use count( distinct xxxx ) instead of group by.
proc sql noprint;
create table counts as
select
%let sep=;
%do i=1 %to %sysfunc(countw(&name_list,%str( )));
%let next_name = %scan(&name_list,&i,%str( ));
&sep. count(distinct &next_name) as &next_name
%let sep=,;
%end;
from &dsname
;
quit;
So this will get a dataset with one observation. You can use PROC TRANSPOSE to turn it into one observation per variable instead.
proc transpose data=counts out=counts_tall ;
var _all_;
run;
Now you can just query that table to find the names of the columns with 0 non-missing values.
proc sql noprint ;
select _name_ into :drop_list separated by ' '
from counts_tall
where col1=0
;
quit;
Now you can use the new DROP_LIST macro variable.
data &out ;
set &dsname ;
drop &drop_list;
run;
So now all that is left is to clean up after your self.
proc delete data=counts counts_tall ;
run;
%mend;
As far as your specific initial question, this is fairly straightforward. Assuming &testvarlist is your macro variable containing the variables you are interested in, and creating some test data in have:
%let testvarlist=x y z;
data have;
call streaminit(7);
do id = 1 to 1e6;
x = floor(rand('Uniform')*10);
y = floor(rand('Uniform')*10);
z = floor(rand('Uniform')*10);
if x=0 and y=4 and z=7 then call missing(of x y z);
output;
end;
run;
data want fordel;
set have;
if min(of &testvarlist.) = max(of &testvarlist.)
and (cmiss(of &testvarlist.)=0 or missing(min(of &testvarlist.)))
then output fordel;
else output want;
run;
This isn't particularly inefficient, but there are certainly better ways to do this, as referenced in comments.