Data set have has some columns with prefix Dex. But I don't know how many columns exactly with that prefix.
I want to create an array with values equal to those columns.
data want;
set have;
array Dex[100];
for i = 1 to 100;
[assign values]
end;
run;
Is there a way to do this without knowing those columns' names?
Yes, you could define your array such as:
array vars Dex:;
do i=1 to dim(vars);
[assign values]
end;
run;
Use dictionary.columns to find out the number of columns which has prefix - dex
/Sample dataset/
data have;
dex_random1=1;
dex_1=2;
dex_3=4;
dex_dex_random=2;
run;
proc sql;
select count(*) into: Number_of_vars from dictionary.columns where
upcase(libname)="WORK" and upcase(memname)="HAVE" and upcase(name) like "DEX%";
select name into: All_vars separated by " " from dictionary.columns where
upcase(libname)="WORK" and upcase(memname)="HAVE" and upcase(name) like "DEX%";
quit;
data want(drop=i);
set have;
array dex[&Number_of_vars.] &All_vars. ;
do i=1 to &Number_of_vars.;
dex[i]=1;
end;
run;
Related
Can anyone help with this issue I'm having where the macro is only taking the final row value of the data?
I have some data that looks like this:
data data1 ;
infile datalines dsd dlm='|' truncover;
input id :$2. year_age_15 EDU_2000 EDU_2001 EDU_2002 ;
datalines4;
10|2000|3|4|5
11|2000|5|5|6
12|2001|1|2|3
13|2002|5|5|6
14|2001|2|2|2
15|2000|3|3|4
;;;;
However I need it to use the year variable to determine which data to keep, and then change all the values for the years after that value to missing, like so:
data data1 ;
infile datalines dsd dlm='|' truncover;
input id :$2. year_age_15 EDU_2000 EDU_2001 EDU_2002 ;
datalines4;
10|2000|3|.|.
11|2000|5|.|.
12|2001|1|2|.
13|2002|5|5|6
14|2001|2|2|.
15|2000|3|.|.
;;;;
I've been trying to get this macro to work, but it only works intermittently and works just for the final row of the data rather than looping through the rows.
%macro macro2 (output=, input=);
data &output;
set &input;
%DO I = 1 %TO 6;
%do; call symput('value2',trim(left(put(year_age_15,8.))));
temp_col=&value2.;
%let year_end=&value2.;
%put YEAR END IS: &year_end.;
%put EDU YEAR IS: EDU_&year_end.;
%do year = &year_end. %TO 2002;
%put &year.;
EDU_&year.=.;
%end;
%end;
%end;
run;
%MEND macro2;
%macro1(input=testset, output=output_testset);
In R it could be something simple like :
for(i in 1:6){.
do this
}
Any advice? I can't figure out which bit is going wrong, thanks!
So, I think the issue here is your data is at the wrong level. You certainly can do what Reeza suggests, and I think it's probably reasonable to do so, but the reason why this is a bit complicated is that you have data in your variable name. That's not a best practice - your variable name should be "education" and your data should have a row for each year. Then this would be a simple WHERE statement!
Here's a simple PROC TRANSPOSE that turns it to the right structure, and then if you really need it the other way, a second one will turn it back. The where statement can be in the proc transpose or could be used somewhere else.
proc transpose data=data1 out=data_t (where=(year_Age_15 ge input(scan(_NAME_,2,'_'),4.)));
by id year_Age_15;
var edu_:;
run;
proc transpose data=data_t out=want;
by id year_age_15;
id _name_;
var col1;
run;
Create an array and index it by years rather than default 1:n
Loop through your array starting at year+1 and set to missing
data want;
set data1;
array educ(2000:2002) edu_2000-edu_2002;
if (year_age_15 +1) <= hbound(educ) then do i= (year_age_15 +1) to hbound(educ);
call missing(educ(i));
end;
run;
As #Joe mentions, the year to match is part of a variable name, which is tremor inducing 'data in the metadata'
You can use the VNAME to retrieve the variable name of an index accessed array element. Use that feature to compare to expected variable name whilst looping over a variable array based on variables named EDU*.
Example:
data have ;
infile datalines dsd dlm='|' truncover;
input id :$2. year_age_15 EDU_2000 EDU_2001 EDU_2002 ;
datalines4;
10|2000|3|4|5
11|2000|5|5|6
12|2001|1|2|3
13|2002|5|5|6
14|2001|2|2|2
15|2000|3|3|4
;;;;
data want;
set have;
array edus edu_:;
* find index of element corresponding to variable name having year;
do _n_ = 1 to dim(edus) until (upcase(vname(edus(_n_))) = cats('EDU_',year_age_15));
end;
* fill in elements at indices post the found one with missing values;
do _n_ = _n_+1 to dim(edus);
call missing(edus(_n_));
end;
run;
I am working with a table with more than 50 columns. I am trying to replace the value of multiple columns using a lookup table.
Table:
data have;
infile datalines delimiter=",";
input ID $1. SUB_ID :$2. COUNTRY :$2. A $1. B $1.;
datalines;
1,A,FR,A,B
2,B,CH,,B
3,C,DE,B,A
4,D,CZ,,B
5,E,GE,A,
6,F,EN,B,
7,G,US,,A
;
run;
Lookup table:
data lookup;
infile datalines delimiter=",";
input value_before $1. value_after :$2.;
datalines;
A,1
B,2
C,3
;
run;
Actual code:
data want;
if 0 then set lookup;
if _n_ = 1 then do;
declare hash lookup(dataset:'lookup');
lookup.defineKey('value_before');
lookup.defineData('value_after');
lookup.defineDone();
end;
set have;
if (lookup.find(key:A) = 0) then
A = value_after;
if (lookup.find(key:B) = 0) then
B = value_after;
/* ... */
/* if (lookup.find(key:Z) = 0) then
Z = value_after; */
drop value_before value_after;
run;
I guess this code would do the job if I would hardcode the 50 columns.
I wonder if there is a way to "apply" the hash.find() to all variables except the first three (ID, SUB_ID and Country) (maybe by indexing ?) without having to hardcode them or to use macros. For the sake of example I only computed 2 variables to replace the value (A and B) but there are more than 50 (with really different names and no pattern like var1,var2,...,varn).
In cases like this, I like to use proc sql and the dictionary table to fill in the column names for me to create an array. The below code will pull the variable names from dictionary.columns and save them as space-delimited into the macro variable varnames. We can feed this into an array and then use array logic to do the rest.
proc sql noprint;
select name
into :varnames separated by ' '
from dictionary.columns
where libname = 'WORK'
AND memname = 'HAVE'
AND name NOT IN('ID', 'SUB_ID', 'COUNTRY')
;
quit;
data want;
if 0 then set lookup;
if _n_ = 1 then do;
declare hash lookup(dataset:'lookup');
lookup.defineKey('value_before');
lookup.defineData('value_after');
lookup.defineDone();
end;
set have;
array vars[*] &varnames.;
do i = 1 to dim(vars);
if lookup.Find(key:vars[i])=0 then vars[i] = value_after;
end;
drop value_before value_after i;
run;
So I have a dataset with one primary key: unique_id and 1200 variables. This dataset is generated from a macro so the number of columns will not be fixed. I need to split this dataset into 4 or more datasets of 250 variables each, and each of these smaller datasets should contain the primary key so that I can merge them back later. Can somebody help me with either a sas function or a macro to solve this?
Thanks in advance.
A simple way to split a datasets in the way you request is to use a single data step with multiple output datasets where each one has a KEEP= dataset option listing the variables to keep. For example:
data split1(keep=Name Age Height) split2(keep=Name Sex Weight);
set sashelp.class;
run;
So you need to get the list of variables and group then into sets of 250 or less. Then you can use those groupings to generate code like above. Here is one method using PROC CONTENTS to get the list of variables and CALL EXECUTE() to generate the code.
I will use macro variables to hold the name of the input dataset, the key variable that needs to be kept on each dataset and maximum number of variables to keep in each dataset.
So for the example above those macro variable values would be:
%let ds=sashelp.class;
%let key=name;
%let nvars=2;
So use PROC CONTENTS to get the list of variable names:
proc contents data=&ds noprint out=contents; run;
Now run a data step to split them into groups and generate a member name to use for the new split dataset. Make sure not to include the KEY variable in the list of variables when counting.
data groups;
length group 8 memname $41 varnum 8 name $32 ;
group +1;
memname=cats('split',group);
do varnum=1 to &nvars while (not eof);
set contents(keep=name where=(upcase(name) ne %upcase("&key"))) end=eof;
output;
end;
run;
Now you can use that dataset to drive the generation of the code:
data _null_;
set groups end=eof;
by group;
if _n_=1 then call execute('data ');
if first.group then call execute(cats(memname,'(keep=&key'));
call execute(' '||trim(name));
if last.group then call execute(') ');
if eof then call execute(';set &ds;run;');
run;
Here are results from the SAS log:
NOTE: CALL EXECUTE generated line.
1 + data
2 + split1(keep=name
3 + Age
4 + Height
5 + )
6 + split2(keep=name
7 + Sex
8 + Weight
9 + )
10 + ;set sashelp.class;run;
NOTE: There were 19 observations read from the data set SASHELP.CLASS.
NOTE: The data set WORK.SPLIT1 has 19 observations and 3 variables.
NOTE: The data set WORK.SPLIT2 has 19 observations and 3 variables.
Just another way of doing it using macro variables:
/* Number of columns you want in each chunk */
%let vars_per_part = 250;
/* Get all the column names into a dataset */
proc contents data = have out=cols noprint;
run;
%macro split(part);
/* Split the columns into 250 chunks for each part and put it into a macro variable */
%let fobs = %eval((&part - 1)* &vars_per_part + 1);
%let obs = %eval(&part * &vars_per_part);
proc sql noprint;
select name into :cols separated by " " from cols (firstobs = &fobs obs = &obs) where name ~= "uniq_id";
quit;
/* Chunk up the data only keeping those varaibles and the uniq_id */
data want_part∂
set have (keep = &cols uniq_id);
run;
%mend;
/* Run this from 1 to whatever the increment required to cover all the columnns */
%split(1);
%split(2);
%split(3);
this is not a complete solution but some help to give you another insight into how to solve this. The previous solutions have relied much on proc contents and data step, but I would solve this using proc sql and dictionary.columns. And I would create a macro that would split the original file into as many parts as needed, 250 cols each. The steps roughly:
proc sql; create table as _colstemp as select * from dictionary.columns where library='your library' and memname = 'your table' and name ne 'your primary key'; quit;
Count the number of files needed somewhere along:
proc sql;
select ceil(count(*)/249) into :num_of_datasets from _colstemp;
select count(*) into :num_of_cols from _colstemp;
quit;
Then just loop over the original dataset like:
%do &_i = 1 %to &num_of_datasets
proc sql;
select name into :vars separated by ','
from _colstemp(firstobs=%eval((&_i. - 1)*249 + 1) obs = %eval(min(249,&num_of_cols. - &_i. * 249)) ;
quit;
proc sql;
create table split_&_i. as
select YOUR_PRIMARY_KEY, &vars from YOUR_ORIGINAL_TABLE;
quit;
%end;
Hopefully this gives you another idea. The solution is not tested, and may contain some pseudocode elements as it's written from my memory of doing things. Also this is void of macro declaration and much of parametrization one could do.. This would make the solution more general (parametrize your number of variables for each dataset, your primary key name, and your dataset names for example.
I have a bunch of character variables which I need to sort out from a large dataset. The unwanted variables all have entries that are the same or are all missing (meaning I want to drop these from the dataset before processing the data further). The data sets are very large so this cannot be done manually, and I will be doing it a lot of times so I am trying to create a macro which will do just this. I have created a list macro variable with all character variables using the following code (The data for my part is different but I use the same sort of code):
data test;
input Obs ID Age;
datalines;
1 2 3
2 2 1
3 2 2
4 3 1
5 3 2
6 3 3
7 4 1
8 4 2
run;
proc contents
data = test
noprint
out = test_info(keep=name);
run;
proc sql noprint;
select name into : testvarlist separated by ' ' from test_info;
quit;
My idea is then to just use a data step to drop this list of variables from the original dataset. Now, the problem is that I need to loop over each variable, and determine if the observations for that variable are all the same or not. My idea is to create a macro that loops over all variables, and for each variable counts the occurrences of the entries. Since the length of this table is equal to the number of unique entries I know that the variable should be dropped if the table is of length 1. My attempt so far is the following code:
%macro ListScanner (org_list);
%local i next_name name_list;
%let name_list = &org_list;
%let i=1;
%do %while (%scan(&name_list, &i) ne );
%let next_name = %scan(&name_list, &i);
%put &next_name;
proc sql;
create table char_occurrences as
select &next_name, count(*) as numberofoccurrences
from &name_list group by &next_name;
select count(*) as countrec from char_occurrences;
quit;
%if countrec = 1 %then %do;
proc sql;
delete &next_name from &org_list;
quit;
%end;
%let i = %eval(&i + 1);
%end;
%mend;
%ListScanner(org_list = &testvarlist);
Though I get syntax errors, and with my real data I get other kinds of problems with not being able to read the data correctly but I am taking one step at a time. I am thinking that I might overcomplicate things so if anyone has an easier solution or can see what might be wrong to I would be very grateful.
There are many ways to do this posted around.
But let's just look at the issues you are having.
First for looping through your space delimited list of names it is easier to let the %do loop increment the index variable for you. Use the countw() function to find the upper bound.
%do i=1 %to %sysfunc(countw(&name_list,%str( )));
%let next_name = %scan(&name_list,&i,%str( ));
...
%end;
Second where is your input dataset in your SQL code? Add another parameter to your macro definition. Where to you want to write the dataset without the empty columns? So perhaps another parameter.
%macro ListScanner (dsname , out, name_list);
%local i next_name sep drop_list ;
Third you can use a single query to count all of variables at once. Just use count( distinct xxxx ) instead of group by.
proc sql noprint;
create table counts as
select
%let sep=;
%do i=1 %to %sysfunc(countw(&name_list,%str( )));
%let next_name = %scan(&name_list,&i,%str( ));
&sep. count(distinct &next_name) as &next_name
%let sep=,;
%end;
from &dsname
;
quit;
So this will get a dataset with one observation. You can use PROC TRANSPOSE to turn it into one observation per variable instead.
proc transpose data=counts out=counts_tall ;
var _all_;
run;
Now you can just query that table to find the names of the columns with 0 non-missing values.
proc sql noprint ;
select _name_ into :drop_list separated by ' '
from counts_tall
where col1=0
;
quit;
Now you can use the new DROP_LIST macro variable.
data &out ;
set &dsname ;
drop &drop_list;
run;
So now all that is left is to clean up after your self.
proc delete data=counts counts_tall ;
run;
%mend;
As far as your specific initial question, this is fairly straightforward. Assuming &testvarlist is your macro variable containing the variables you are interested in, and creating some test data in have:
%let testvarlist=x y z;
data have;
call streaminit(7);
do id = 1 to 1e6;
x = floor(rand('Uniform')*10);
y = floor(rand('Uniform')*10);
z = floor(rand('Uniform')*10);
if x=0 and y=4 and z=7 then call missing(of x y z);
output;
end;
run;
data want fordel;
set have;
if min(of &testvarlist.) = max(of &testvarlist.)
and (cmiss(of &testvarlist.)=0 or missing(min(of &testvarlist.)))
then output fordel;
else output want;
run;
This isn't particularly inefficient, but there are certainly better ways to do this, as referenced in comments.
User PomPazz post this answer for creating a list from input variables:
"You need to use a macro to "write" the SAS code for you.
This should do what you are looking for. It takes a space delimited list of values, and loops over them doing what your code specifies. Post a comment if you have a question on it.
%macro doit(list);
proc sql noprint;
%let n=%sysfunc(countw(&list));
%do i=1 %to &n;
%let val = %scan(&list,&i);
create table somlib._&val as
select * from somlib.somtable
where item=&val;
%end;
quit;
%mend;
%doit(100 101 102);
Note, data sets cannot start with a number so I have these starting with '_'"
My questions is, how can this be applied to creating a list from a variable in a dataset that can be used in an IF statement such as "IF Telephone in(List) then Invalid=1"
This is needed to validate a list of telephone numbers from a predetermined list of invalid numbers.
The basic answer to your question is that you need to pull it into a macro variable or an include file.
proc sql;
select distinct telephone into :tellist separated by ','
from invalid_phones;
quit;
data want;
set have;
if telephone in (&tellist.) then invalid=1;
run;
This is limited to about 32k characters, so if you have more than ~3000 phone numbers it may not work. If telephone is character you need to select distinct quote(telephone) instead.
The more detailed answer is that this, usually, is inefficient. Better is to use a format.
data for_fmt;
set invalid_phones;
start=telephone;
label='INVALID';
fmtname='TELCHECKF'; *add $ if telephone is a character field;
output;
if _n_=1 then do; *this block adds a line to deal with non-matching records (hlo=o means other);
start=.;
label='VALID';
hlo='o';
output;
end;
run;
proc format cntlin=for_fmt;
quit;
data want;
set have;
if put(telephone,TELCHECKF.)='INVALID' then invalidflag=1;
run;
This can be faster than the list method, and doesn't have the length issues.
Joe's answer is great, but I just wanted to add that you can do this in one SQL step
proc sql noprint;
create table want as
select *, case
when telephone in (select distinct telephone from invalid_phones) then 1
else 0
end as invalid
from have
;
quit;