I'm working with different databases. All of them contain information about 1000+ companies. A company is defined by its ticker code (the short version of the name (i.e. Ford as F) usually seen on stock quotation boards).
Aside from the ticker code to merge on I also have to merge on the time. I used month as a count variable throughout my time series. The final purpose is to have a regression in the kind of
Y(jt) = c + X(jt) +X1(jt) etc with j = company (ticker) and t = time (month).
So imagine I have 2 databases, one of which is the base database with variables such as Tickers, months, betas of a company (risk measure) etc. and a second database which has an extra variable (let's say market capitalisation).
What I want to do then is to merge these 2 databases based on the ticker and the month.
Example: Base database:
Ticker ____ Month ____ Betas
AA ____ 4 ____ 1.2
BB ____ 8 ____ 1.18
Second database:
Ticker ____ Month ____ MCAP
AA ____ 4 ____ 8542
BB ____ 6 ____ 1245
Then after merge I would like to have something like this:
Ticker ____ Month ____ Betas ____ MCAP
AA ____ 4 ____ 1.2 ____ 8542
So all observations that do not match BOTH the date and ticker have to be dropped. I'm sure this is possible, just can't find the right type of code.
PS: I'm guessing the underscores have something to do with font layout but both the bold as italic is supposed to be normal :)
Agree with Jonathan... after sorting both datasets independently by ticker and time, the data step of merging is what I would use..... little modification
data want;
merge base(in = b) mcap(in = m);
by ticker time;
if m & b;
run;
Records that don't have common ticker and time in both datasets would be dropped automatically..
Calling the two datasets base and mcap, and assuming that they have both been sorted by ticker and month, you can do it this way:
data want;
merge base(in = b)
mcap(in = m);
if m & b;
run;
The subsetting if will not accept any row that does not match in bath datasets.
Ok so it appears you can just do it very easily by:
proc sort data=work;
by ticker month;
run;
proc sort data=wsize;
by ticker month;
run;
data test;
merge work(in=a) wsize(in=b);
by ticker month;
frommerg=a;
fromwtvol=b;
run;
data test;
set test;
if frommerg=0 then delete;
run;
data test;
set test;
if fromwtvol = 0 then delete;
run;
data test;
set test;
drop frommerg fromwtvol;
run;
That's the code I used, I tried this before posting because I didn't want to look like a leecher but it so happens that the 2 databases i tried had nothing in common (what are the odds with 70.000 observations :D), I retried it and it works (for now!)
Thanks anyway!
proc sort data=database1;
by ticker month;
run;
proc sort data=database2;
by ticker month;
run;
data gh;
merge database1(in=a) database2(in=b);
by ticker month;
if a and b;
run;
Related
I have a bunch of time series data (sas-files) which I like to merge / combine up to a larger table (I am fairly new to SAS).
Filenames:
cq_ts_SYMBOL, where SYMBOL is the respective symbol for each file
with the following structure:
cq_ts_AAA.sas7bdat: file1
SYMBOL DATE TIME BID ASK MID
AAA 20100101 9:30:00 10.375 10.4 .
AAA 20100101 9:31:00 10.38 10.4 .
.
.
AAA 20150101 15:59:00 15 15.1 .
cq_ts_BBB.sas7bdat: file2
SYMBOL DATE TIME BID ASK MID
BBB 20120101 9:30:00 12.375 12.4 .
BBB 20120102 9:31:00 12.38 12.4 .
.
.
BBB 20170101 15:59:00 20 20.1 .
Key characteristics:
- They have the same variable name
- They have different number of observations
- They are all saved in the same folder
So what I want to do is:
- Create 3 tables: BID-table, ASK-table, Mid-table with the following structure, ie for bid-table, cq_ts_bid.sas7bdat:
DATE TIME AAA BBB ...
20100101 9:30:00 10.375 .
20100102 9:31:00 10.38 .
.
.
20120101 9:30:00 9.375 12.375
20120102 9:31:00 9.38 12.38
.
.
20150101 15:59:00 15 17
.
.
20170101 15:59:00 . 20
It is not all to difficult to do it for 2 stock time series, however, I was wondering whether there is the possibility to do the following:
From data set cq_ts_AAA take DATE TIME BID and rename BID to AAA (either from the values in symbol? does this make sense? or get the name from the filename).
Do the same for cq_ts_BBB.
In fact, loop through the folder to get the number of files and filenames (this part I got more or less, see below).
Merge cq_ts_BBB and cq_ts_BBB having DATE TIME AAA (former bid price of AAA) BBB (former bid price of BBB), for all the files in the folder.
Do this for BID, then for ASK and finally MID (actually I couldn't get the midpoint variable from bid and ask (i.e. mid= (bid + ask) / 2;) just gives me the "." in the previous data steps when creating the files).
I think a macro to first get each single file then rename (when should this step take place?) it and merge them together - like a double loop.
Here the renaming and merging part:
data ALDW_short (rename=(iprice = ALDW));
set output.cq_ts_aldw
retain date time ALJ;
run;
data ALJ_short (rename= (iprice = ALJ));
set output.cq_ts_alj;
retain date time datetime ALJ;
run;
data ALDW_ALJ_merged (keep= date itime ALDW ALJ);
merge ALDW_short ALJ_short;
by datetime;
run;
This is the part to loop through the folder and get a list of names:
proc contents data = output._all_ out = outputcont(keep = memname) noprint;
run;
proc sort data = outputcont nodupkey;
by memname;
run;
data _null_;
set outputcont end = last;
by memname;
i+1;
call symputx('name'||trim(left(put(i,8.))),memname);
if last then call symputx('count',i);
run;
Would it make sense to extract the symbol (and how? they have different length) from the filename or just to take them from the variable SYMBOL (and how can I get the one value to rename my column?)?
Somehow I have difficulty changing the order of columns, ie. I tried with retain and format.
Looks like you could do this easily with PROC TRANSPOSE. Combine your datasets into a single dataset.
data all ;
set set output.cq_ts_: ;
by date time;
run;
Then use PROC TRANSPOSE for each of your source variables/target tables.
proc transpose data=all out=bid ;
by date time ;
id symbol;
var bid;
run;
Given your example data a formula for MID of
mid = (bid + ask)/2 ;
Should work. Most likely if you got all missing values you probable put the assignment statement before the SET or INPUT statement. In other words you were trying to calculate using values that had not been read in yet.
I am tryinng to loop over a series of dates in order to create the dates inbetween. This is to be done in steps of month, always displaying the last day of the respective month. The start and end dates are given (first_date and last_date), while the last_date should always refer to the end of the previous month.
The original dataset looks like the following:
customer id first_date last_date
xy 135 01.01.2000 25.03.2005
xy 247 19.03.2003 25.03.2005
ab 387 01.06.2010 30.12.2012
ab 128 01.05.2010 28.02.2011
...
My goal is to have a dataset which looks like this:
customer id date
xy 135 31.01.2000
xy 135 28.02.2000
...
xy 135 28.02.2005
xy 247 31.03.2003
xy 247 30.04.2003
...
xy 247 28.02.2005
I found the solution to iterate over days quite straightforward (see below), but I am struggling to implement the monthly steps and the end of month dates.
data want;
set have;
by customer id;
do day = first_date to last_date;
output;
end;
format day date9.;
run;
Thanks for your help!!
First, lets get some data:
data have;
attrib customer length=$10 informat=$10.
id informat=best.
first_date informat=ddmmyy10. format=ddmmyy10.
last_date informat=ddmmyy10. format=ddmmyy10.
;
input customer $
id
first_date
last_date
;
datalines;
xy 135 01.01.2000 25.03.2005
xy 247 19.03.2003 25.03.2005
ab 387 01.06.2010 30.12.2012
ab 128 01.05.2010 28.02.2011
;
run;
The intnx() function will come to the rescue here. We are going to create a new variable called date, and then use the intnx function to return the end of the month for that date. As long as that date is less than the end date, we will continue to output it to a dataset and then increment to the end of the following month.
data want;
format date ddmmyy10.;
set have;
date = intnx('month',first_date,0,'end');
do while (date le last_date);
output;
date = intnx('month',date,1,'end');
end;
drop first_date last_date;
run;
While I think Rob's answer is the right way to do this, it's probably helpful to see how to do it the way you were trying to.
Starting with this:
data want;
set have;
by customer id;
do day = first_date to last_date;
output;
end;
format day date9.;
run;
This gives you too many rows, right? So what you need to do is identify where in the month you are. There are a bunch of ways to do this. Several date functions (like INTNX and INTCK) could be used to tell you where you are; but the easiest is just to compare month(date) with month(date+1). When they're different, you're on the last day of a month!
data want;
set have;
by customer id notsorted;
do day = first_date to last_date;
if month(day) ne month(day+1) then output;
end;
format day date9.;
run;
(I added notsorted since Rob's example data was not sorted, and I'm lazy. Probably not needed in your real case.)
I would note that this probably isn't your ideal solution - Rob's is probably that, in terms of data steps - in terms of speed. This of course will iterate through every day rather than just once per month.
Another option if you have the dataset you created above - with one row per day - is to use PROC EXPAND, if you have the ETS module. It's very handy for things like this.
data intermediate;
set have;
by customer id notsorted;
do day = first_date to last_date;
output;
end;
format day date9.;
run;;;
Here's your day-level data. Then below is the PROC EXPAND statement, asking for monthly data, aligned at the end. id day; identifies the time series variable, and by customer id notsorted; is the normal by statement (what variables identify the observations), with notsorted so they don't have to be in order relative to each other.
proc expand data=intermediate out=want from=day to=month align=end;
id day;
by customer id notsorted;
run;
This gives a slightly different solution than Rob's and my other solution, because it does give you the final row for each if it's not at the end of a month (and does set that final row to the end of the month). If that's desired, great, and our solutions can easily be adapted to give that; if it's not desired, you'll have to remove it afterwards.
You can do this with a simple iterative DO loop by using the date interval functions. Subtract one from the number of intervals to make it end at the last day of the previous month.
data want ;
set have ;
do offset=0 to intck('month',first_date,last_date)-1;
date=intnx('month',first_date,offset,'e');
output;
end;
format date yymmdd10.;
run;
I have a dataset that has a list of contributions of members of a sales organization by day. What I want to ultimately end up with is the following information:
For each day:
How much the entire team sold. ($200 for day one, $350 for day two..)
How much a designated subset ("Joe"...for example) of that team sold (Joe sold $100 day one, $200 day two...)
the difference in the above two calculations ($200-$100 for day one, $350-$200 for day two....)
how many total people contributed that day (2 in day 1, 3 in day two, 5 in day 3)
how many of my designated subset contributed that day (1 every day in this case, since Joe was there every day)
In the example below, Joe is my designated subset. The problem I am having is directing SAS to only sum up Joe's contributions. The method I have below works, but only if Joe is the only contributor AND if he contributes every day. I basically force him to be the first entry, then point to him. This fails if he is not there one day, or if my subset has multiple people.
Below is my attempt I've been working on, but I think I'm going down the wrong path, since this will not be dynamic enough when I add more people. For example, if the subset now becomes Joe and Sue....the calculation will still just point to Joe. If I point it two first two obs, it may select hal accidentally from day one. Is there a way to specify by rom "Only add the Amount column if the name next to it is either Joe or Sue? Help!
*declare team;
/*%let team=('joe','sue');*/
%let team=('joe');
*input data;
data have;
input day name $ amount;
cards;
1 hal 100
1 joe 100
2 joe 80
2 sue 70
2 jim 200
3 joe 50
3 sue 100
3 ted 200
3 tim 100
3 wen 5000
;
run;
*getting my team to float to top of order list;
data have;
set have;
if name in &team. then order=1;
else order=2;
run;
*order;
proc sort data=have;
by day order name;
run;
*add running count by day;
data have;
set have;
by day;
x+1;
if first.day then x=1;
run;
*get number of people on team;
proc sql noprint;
select count(distinct name) into :count
from have
where name in &team.;
quit;
*get max of people per day;
proc sql noprint;
select max(x) into :max_freq from have;
quit;
*pre transpose...set labels;
data have;
set have;
varname=cats('Name_',x);
value=name;
output;
varname=cats('Amount_',x);
value=amount;
output;
keep day value varname;
run;
*transpose;
proc transpose data=have out=have_transp(drop=_NAME_);
by day;
id varname;
var value;
run;
data want;
set have_transp;
array Amount {*} Amount:;
TOT_Amount=0;
NUM_TOTAL_PEOPLE=0;
do i=1 to dim(Amount);
if Amount[i]>0
then
do;
TOT_Amount+Amount[i];
NUM_TOTAL_PEOPLE+1;
end;
end;
TEAM_CONTRIB=Amount_1;
NON_TEAM_CONTRIB=TOT_Amount-TEAM_CONTRIB;
run;
A few other things:
Every member of the team will not always be present every day
There are very many possibilities for how many people might be on the total team and/or subset
Here's a way using proc means that doesn't use arrays. Proc means will calculate data at different levels by default when using the CLASS and TYPES statements. The data can then be merged into the appropriate level. In this solution it doesn't matter how many people are in the group/subset or that everyone is present for every day.
/*Subset group*/
data subteam;
input name $;
cards;
joe
sue
;
run;
/*Sample data*/
data have;
input day name $ amount;
cards;
1 hal 100
1 joe 100
2 joe 80
2 sue 70
2 jim 200
3 joe 50
3 sue 100
3 ted 200
3 tim 100
3 wen 5000
;
run;
*Set group variable for subset team;
data have;
set have;
group=0;
run;
*Set group variable=1 to subset;
proc sql;
update have
set group=1
where name in (select name from subteam);
quit;
*Calculate sums;
proc means data=have;
class day group;
types day day*group;
var amount;
output out=want1 sum=total n=count;
run;
*Reformat into desired format;
data want2;
merge want1 (where=(group=.) rename=(total=total_overall count=count_overall))
want1 (where=(group=1) rename=(total=total_group count=count_group));
by day;
run;
I have 55 weeks of sales data of a certain item. I created two SAS datasets from the original data. The first dataset has the date and the sum of quantity sold in each date. Therefore, I have 385 observations (55 x 7). The second table has detailed transaction data. Specifically, for each date, I have the time between transactions, which is the time between the arrival of one customer and the next one who purchased that item (I call it the interarrival times). What I need to do next is as follows:
For the first table (daily sales) I need to take the sales data for
each week, fit a number of distributions to find the parameters of
each one, and record those parameters in a separate table. Note that
each week has eaxctly 7 observations
For the second table (interarrival times) I also need to fit a
number of distributions to find the parameters of each one, and
record those parameters in the same table above, but here, I don’t
have an exact number of observations in each week
Note: I already labeled the week number for the observations in each of the two datasets and I wrote the code that fits the distributions to the data. The only area in which I am struggling is how to tell SAS to take the data for one week, do the calculations, fit the distributions, and then move to the next week (i.e. group the data by week and perform multiple statements on each group).
I tried so many methods and none of them worked including nested loops. I know how to get the weekly sales using other methods and procedures such as PROC SQL, but I am not sure whether I can fit distributions with PROC SQL.
I am using proc nlp to estimate the parameters of each distribution using the maximum likelihood method. For example, if I need to estimate Mu and Sigma for the normal distribution, I am using the following code:
proc nlp data= temp vardef=n covariance=h outest=parms;
title "Normal";
max loglik;
parms mu=0, sigma=1;
bounds sigma > 1e-12;
loglik=-log(sigma*(2*constant('PI'))**.5) - 0.5*((x-mu)/sigma)**2;
run;
This method will find Mu and Sigma that most likely produced the data.
For others wishing to use SAS's internal grouping the nlm code would become:
/* Ensure that the data is sorted to allow group processing */
proc sort data = temp;
by week;
run;
proc nlp data = temp vardef = n covariance = h outest = parms;
/* Produce separate output for each week */
by week;
title "Normal";
max loglik;
parms mu = 0, sigma = 1;
bounds sigma > 1e-12;
loglik = -log(sigma * (2 * constant('PI'))**.5) - 0.5 * ((x - mu) / sigma)**2;
run;
And here is a method using proc univariate:
/* Suppress printed output (remove to see all the details) */
ods select none;
proc univariate data = temp;
/* Produce separate output for each week */
by week;
histogram x /
/* Request fitting to normal distribution */
normal
/* You can select other distributions too */
lognormal;
/* Put the fitted parameters in a dataset */
ods output ParameterEstimates = parms;
/* Put the fit statistics in a dataset */
ods output GoodnessOfFit = quality;
run;
/* Restore printing output */
ods select all;
Here's what I used
%macro weekly;
%do i=1 %to 55;
proc sql;
create table temp as
select location, UPC, date, x, week
from weeks
where week = &i;
quit;
/* I have here the rest of the code where I do my calculations and I fit the distributions to the data of each week */
%end;
%mend;
%weekly;
I knew that proc sql would work initially but I was wondering whether there may be a more efficient way to do it.
For the purpose of simplicity I'll try to take an example from everyday life. Let's say I have a table in CSV file loaded in a table called dataOriginal with 3 columns - names, jobs , dates.
Let's take a closer look at the column "date":
date
____
'13.01.2014 20:34'
'22.03.2014 11:17'
...
I want to split date in a date-vector and add this vector (along with the variable names for each of it's columns (since we have multiple dates we have de facto a matrix)) to a column in a new table again named "Date" but with all the naming goodies in it such as year, month etc.
Here is what I have done so far (sorry for the poor code quality but I've just started learning MATLAB :-/):
I split each date in a date-vector and also add names to each element like this:
dateFormat = 'dd.mm.yy HH:MM';
[year,month,day,hour,minute,second] = datevec(datesRaw, dateFormat);
so that I have this:
year(1) % returns '2014' since this is the first date in my column
year % returns all years in my entire column
Then I converted the above to a table:
dates = array2table([year,month,day,hour,minute,second],'VariableNames',{'year','month',...,'second'});
so I get a nice output like this
year month second
____ _____ ... ______
2014 1 0
2014 3 0
... ... ... ...
This allows me an easy-to-read access to each column by simply calling for example:
year % returns all years
year(1) % returns first entry's year (here: '2014' from '13.01.2014 20:34')
I've processed my other columns too doing various operations on those and at the end I'm trying to horizontally concatenate all like this:
name job date
____ _____________________ _____________________
year month ... second
____ _____ ______
"Bob" "Construction worker" 2014 1 ... 0
"Alice" "Waitress" 2014 3 ... 0
... ... ... ... ... ...
I'm struggling exactly with the part with the nesting of year,month etc. in a single column named "date". I'd like to address a date's element in the table above as follows:
myData.name(1) % will return 'Bob'
myData.job(1) % will return 'construction worker'
myData.date(1).year(1) % should return '2014' for Bob, the construction worker
Currently I'm having the following code after some sweating and swearing:
dataFinal =
horzcat(array2table([dataProcessed(:,1),dataProcessed(:,2)],'VariableNames',[dataOriginal.Properties.VariableNames(1),dataOriginalProperties,VariableNames(2)]],
array2table([year,month,day,hour,minute,second],'VariableNames',{'year','month','day','hour','minute','second'}))
where
dataProcessed(:,1) are my processed names
dataProcessed(:,2) are my processed jobs
dataOriginal.Properties.VariableNames(1) is the name of the first column in my original table - "name"
dataOriginal.Properties.VariableNames(2) is the name of the second column in my original table - "job"
I do not know how to insert
array2table([year,month,day,hour,minute,second],'VariableNames',{'year','month','day','hour','minute','second'})
in a named column "date" in order to accomplish my goal.
Thanks!
Try the following, it may be what you're looking for:
data = table(names, jobs, table(years, months, ...), 'VariableNames', {'name', 'job', 'date'})
Though you will address as follows, which is slightly different from what you said you want; it may still work for your purposes:
data.name(1);
data.job(1);
data.date.year(1);
EDIT: To see your output, do
disp([data(:, ~strcmp(data.Properties.VariableNames, 'date')), data.date])
names ids years months
_____ ___ _____ ______
'Bob' 1 2014 4
'Max' 2 2013 8
(when editing the comment I didn't exactly replicate the data and fields from the answer, but I think you should get the point here).