In matlab I have a 720x360x365 matrix (let's call it A) of daily precipitation for one year. 365 stands for days in a year. I need to write a code to convert these daily data to the monthly sum. If I start from January, I need to do mean (A,3) of the first 31 days, then the mean (A,3) of February, the next 28 or 29 days. Because the days alternate between 31 and 30 (and 28 or 29 for February), I don't know how to write a code to do this.
please help me I don't know how to do it.
thank you
You can use mat2cell to divide your data in cells per month. First make a vector with the number of days per month (not taking into account leap years), and then use this to divide the data. Then you can use cellfun on each cell (i.e. month) to get any metric you define per month:
data = rand(720, 360, 365);
days_per_month = [31 28 31 30 31 30 31 31 30 31 30 31];
% divide months in cells
data_cell = mat2cell(data, size(data,1), size(data,2), days_per_month);
mean_cell = cellfun(#(A) mean(A,3), data_cell, 'UniformOutput', false)
To use this in a loop, and account for leap years, you can use the function leapyear(year):
days_per_month = [31 28 31 30 31 30 31 31 30 31 30 31];
years = 1984:2015
for k = 1:numel(years)
if leapyear(years(k))
days_per_month(2) = 29;
else
days_per_month(2) = 28;
end
% rest of what you want to do
end
Related
I am trying to group by dataset in three month groups, or quarters, but as I'm starting from an arbitrary date, I cannot use the quarter function in sas.
Example data below of what I have and quarter is the column I need to create in SAS.
The start date is always the same, so my initial quarter will be 3rd Sep 2018 - 3rd Dec 2018 and any active date falling in that quarter will be 1, then quarter 2 will be 3rd Dec 2018 - 3rd Mar 2019 and so on. This cannot be coded manually as the start date will change depending on the data, and the number of quarters could be up to 20+.
The code I have attempted so far is below
data test_Data_op;
set test_data end=eof;
%let j = 0;
%let start_date = start_Date;
if &start_Date. <= effective_dt < (&start_date. + 90) then quarter = &j.+1;
run;
This works and gives the first quarter correctly, but I can't figure out how to loop this for every following quarter? Any help will be greatly appreciated!
No need for a DO loop if you already have the start_date and actual event dates. Just count the number of months and divide by three. Use the continuous method of the INTCK() function to handle start dates that are not the first day of a month.
month_number=intck('month',&start_date,mydate,'cont')+1;
qtr_number=floor((month_number-1)/3)+1;
Based on the comment by #Lee. Edited to match the data from the screenshot.
The example shows that May 11 would be in the 3rd quarter since the seed date is September 3.
data have;
input mydate :yymmdd10.;
format mydate yymmddd10.;
datalines;
2018-09-13
2018-12-12
2019-05-11
;
run;
%let start_date='03sep2018'd;
data want;
set have;
quarter=floor(mod((yrdif(&start_date,mydate)*4),4))+1;
run;
If you want the number of quarters to extend beyond 4 (e.g. September 4, 2019 would be in quarter 5 rather than cycle back to 1), then remove the "mod" from the function:
quarter=floor(yrdif(&start_date,mydate)*4)+1;
The traditional use of quarter means a 3 month time period relative to Jan 1. Make sure your audience understands the phrase quarter in your data presentation actually means 3 months relative to some arbitrary starting point.
The funky quarter can be functionally computed from a months apart derived using a mix of INTCK for the baseline months computation and a logical expression for adjusting with relation to the day of the month of the start date. No loops required.
For example:
data have;
do startDate = '11feb2019'd ;
do effectiveDate = startDate to startDate + 21*90;
output;
end;
end;
format startDate effectiveDate yymmdd10.;
run;
data want;
set have;
qtr = 1
+ floor(
( intck ('month', startDate, effectiveDate)
-
(day(effectiveDate) < day(startDate))
)
/ 3
);
format qtr 4.;
run;
Extra
Comparing my method (qtr) to #Tom (qtr_number) for a range of startDates:
data have;
retain seq 0;
do startDate = '01jan1999'd to '15jan2001'd;
seq + 1;
do effectiveDate = startDate to startDate + 21*90;
output;
end;
end;
format startDate effectiveDate yymmdd10.;
run;
data want;
set have;
qtr = 1
+ floor( ( intck ('month', startDate, effectiveDate)
- (day(effectiveDate) < day(startDate))
) / 3 );
month_number=intck('month',startDate,effectiveDate,'cont')+1;
qtr_number=floor((month_number-1)/3)+1;
format qtr: month: 4.;
run;
options nocenter nodate nonumber;title;
ods listing;
proc print data=want;
where qtr ne qtr_number;
run;
dm 'output';
-------- OUTPUT ---------
effective month_ qtr_
Obs seq startDate Date qtr number number
56820 31 1999-01-31 1999-04-30 1 4 2
57186 31 1999-01-31 2000-04-30 5 16 6
57551 31 1999-01-31 2001-04-30 9 28 10
57916 31 1999-01-31 2002-04-30 13 40 14
58281 31 1999-01-31 2003-04-30 17 52 18
168391 90 1999-03-31 1999-06-30 1 4 2
168483 90 1999-03-31 1999-09-30 2 7 3
168757 90 1999-03-31 2000-06-30 5 16 6
168849 90 1999-03-31 2000-09-30 6 19 7
169122 90 1999-03-31 2001-06-30 9 28 10
169214 90 1999-03-31 2001-09-30 10 31 11
169487 90 1999-03-31 2002-06-30 13 40 14
169579 90 1999-03-31 2002-09-30 14 43 15
169852 90 1999-03-31 2003-06-30 17 52 18
169944 90 1999-03-31 2003-09-30 18 55 19
280510 149 1999-05-29 2001-02-28 7 22 8
280875 149 1999-05-29 2002-02-28 11 34 12
281240 149 1999-05-29 2003-02-28 15 46 16
282035 150 1999-05-30 2000-02-29 3 10 4
282400 150 1999-05-30 2001-02-28 7 22 8
282765 150 1999-05-30 2002-02-28 11 34 12
I have some columns as shown below,
1st Month 2nd Month 3rd Month 4th Month 5th Month 6th Month Current_Month
10 20 30 40 50 60 70
10 10 10 10 10 10 10
And i need output like this,
Feb March April May June July August
10 20 30 40 50 60 70
10 10 10 10 10 10 10
And the data varies every month, So current month column should be replaced by the Name of the Month and accordingly the last months data with Names of the month.
Please note i'm a beginner.
Thanks in advance.
I have 32 bit encoded value for date and time value.I know the sample 32bit encoded value date and time but i don't know how to convert and get this value using c program or any other language or script. The sample data as below
54 FE C0 72 =(25-Oct-13 20:34:58)
55 01 DC 8B =(26-Oct-13 22:34:51)
57 01 DC 8B =(14-Apr-14 14:34:51)
42 23 8F 96 =(02-Jun-09 17:06:30)
3C F5 28 4B= (31-Mar-00 18:51:55)
3A F4 28 49 =(12-Oct-99 18:51:53)
For the above sample data i am tried using unix timestamp method but i am getting wrong date time value.The answers using unix time stamp method as below
54 FE C0 72 =(10 Mar 2015 09:59:14 GMT
55 01 DC 8B =(12 Mar 2015 18:35:55 GMT)
57 01 DC 8B =(04 Apr 2016 03:16:27 GMT)
Please give me your guidance to convert the above 32bit encoded value to correct date and time value. And please share any other methods other than unix timestamp.And i think some algorithms are encrypted inside the hex code
EDIT:
Adding examples from chat:
30067004 =24-mar-1997 07:57:56
2C067004=Tue Apr 16 23:57:56 1996
29567004=Thu Aug 31 15:57:56 1995
13567004=9-jul-90 7:57:56
17567004=15-jun-91 15:57:56
1C567004=15-aug-92 07:57:56
20567004=23-jul-93 15:57:56
24867004=15-jul-94 23:57:56
10067004=29 sep-89 15:57:56
2B067004=21-jan-96 15:57:56
00000000 = 1-Jan-1986
00030000 = 1-jan-1986
00038000 = 1-jan-1994
Let's start by looking at the first two:
54 FE C0 72 =(25-oct-13 20:34:58)
54 FE C0 78 =(25-oct-13 20:35:04)
These two dates are 6 seconds apart, and the values differ by 6. So we know that at least the last byte specifies seconds.
55 01 DC 8B =(26-oct-13 22:34:51)
55 01 E3 93 =(26-oct-13 23:04:51)
Similarly, these two are are 30 minutes (1800 seconds) apart, and the values differ by 1800. So at least the last two bytes specify seconds.
54 FE C0 78 =(25-oct-13 20:35:04)
55 01 DC 8B =(26-oct-13 22:34:51)
There's a larger range in the values, but note that the last two bytes seem to be fairly close in value. Taking DC8Bh - C078h gives us 1C13h = 7187d. That's a difference of 2 hours (7200 seconds) minus 13 seconds, which is how far apart the time portions of the two dates are. So it looks like the last two bytes specify the time. However, there are 86400 seconds in a day, and C078h = 49272d which would be closer to around 13:00:00 than 20:35:04, that and the largest value you can store in 16 bits is 65535. Also, the first two bytes differ by 3 but the dates differ by 1. Let's come back to that in a bit.
55 01 DC 8B =(26-oct-13 22:34:51)
57 01 DC 8B =(14-Apr-14 14:34:51)
Note here that the last two bytes are the same, and that the minutes and seconds are the same, although the hours differ by 8. So perhaps the last two bytes specify seconds in part of a day. Going back to the prior example, the first two bytes differed by 3 when the dates differed by 1. So perhaps the first two bytes specify an 8 hour interval. This would account for the last two bytes being the same when the time differs by 8 or 16 hours. If we take 5701h - 5501h we get 200h = 512d. Dividing by 3 we get 169 2/3 days. The two dates above differ by 170 days, and if you take the hours into account it's about 169 2/3.
So now we have dates. 5501h is the third 8-hour interval in 26-oct-13, so the start of the day is 54FFh = 21759d. Dividing by 3 gives us 7253. Counting back days, that gives us an epoch date of 1993-12-17.
Now lets go back to the time. Assuming the last two bytes are seconds in an 8-hour interval. That gives us a maximum value of 28800d. Note that this value only needs 15 bits to store. DC8Bh has the highest bit set, so let's see what we get if we mask that bit out. That gives us 5C8Bh = 23691d, and 23691 seconds is 6 hours 34 minutes 51 seconds. That matches the third and fifth examples with a difference of 8 hours.
As for the most significant bit in the third byte, my guess is that is specifies DST. All the examples have this bit set, and all the dates are when DST is active.
So to summarize:
The first two bytes divided by 3 is number of days since 1993-12-17.
The first two bytes mod 3 is the 8-hour interval in the day. Multiply this value by 28800 (i.e. seconds in 8 hours) to set the initial time in seconds.
The last two bytes with the most significant bit masked out are seconds from the start of the 8-hour interval. Add this value to the value from the prior step to get seconds from midnight.
Check the most significant bit in the third byte to set the DST flag.
EDIT:
So it seems the result this algorithm gives for 57 01 DC 8B =(14-Apr-14 14:34:51) is ahead by one day. Let's look at one of the new examples:
42 23 8F 96 =(02-Jun-09 17:06:30)
Our algorithm give a date of 30-May-09, so it's behind by 3 days. This is interesting because it differs from what we got for 25-Oct-13 and 26-Oct-13 by about 4 years. What's different is that there's a leap year in between. So perhaps this encoding is assuming all years have 366 days. If we go back to the epoch date of 1993-12-17, we see that there are 15 non-leap years from 1994 to 2013 inclusive. That give us a new epoch date of 1994-1-1, which makes more sense. So after doing the initial conversion with 1994-1-1 as the epoch, we need to count the number of non-leap years and subtract that many days.
Now let's look at this one:
3A F4 28 49 =(12-Oct-99 18:51:53)
The time is still correct, but the date is way off. Notice however that the most significant bit of byte 3 is NOT set. This seems to indicate a different epoch. The start of 12-Oct-99 is 3AF2h = 15090d. Dividing by 3 gives us 5030. Counting backward gives us an epoch of 1986-01-03. But then there's this:
00000000 = 1-Jan-1986
00030000 = 1-jan-1986
So it looks like 1986-1-1 is the epoch, but there's a special case in place for this date, so the actual epoch is 1985-12-31.
But, we're off by 3 days. If all years had 366 days, this would not be the case. It would work however if all years had 365 days. This means that for the 1985-12-31 epoch, we need to count the number of leap years and add that many days. This is the opposite of what we need to do with the 1994-1-1 epoch.
This now works for almost everything. Everything except these:
3C F5 28 4B= (31-Mar-00 18:51:55)
1C567004=15-aug-92 07:57:56
But it does work for this:
2B067004=21-jan-96 15:57:56
So it looks like this encoding does do a leap year check, but only for the current year.
Taking these changes to the algorithm and applying them to the code provided by LPs, we now have this:
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <time.h>
int isleap(int year)
{
if (year % 4 != 0) return 0;
if (year % 100 != 0) return 1;
if (year % 400 != 0) return 0;
return 1;
}
int main(int argc, char *argv[])
{
// Read encoding from command line
uint32_t datetime = strtoul(argv[1],NULL,16);
uint16_t mydate = datetime >> 16;
uint16_t mytime = datetime & 0xFFFF;
int new_encoding = (mytime & 0x8000) != 0;
// Calculate days
time_t linuxSeconds = (mydate/3);
// Calculate the 8 hours on current day of date
uint8_t third_Count = mydate % 3;
// Add days from 1/1/1970, that is the base of time in linux
if (new_encoding)
{
// Days between 1970-1-1 and 1994-1-1 minus 1
linuxSeconds += 8765;
}
else
{
// Days between 1970-1-1 and 1986-1-1 minus 1
linuxSeconds += 5843;
}
// Calculate total amount of hours
linuxSeconds *= 24;
// Calculate total amount of seconds
linuxSeconds *= 3600;
// Add seconds of last 8 hours group
linuxSeconds += (mytime & 0x7FFF);
// Add alla seconds of grups of 8 hours of date
linuxSeconds += (third_Count * 28800);
// Add or subtract days depending on whether new_encoding is set
struct tm *mytm = gmtime(&linuxSeconds);
int daydiff = 0, year;
for (year = new_encoding ? 1994 : 1986; year <= mytm->tm_year + 1900; year++) {
if (year < mytm->tm_year + 1900) {
if (new_encoding) {
// remove a day for non-leap years
if (!isleap(year)) {
daydiff--;
}
} else {
// add a day for leap years unless it's the current year
if (year != (mytm->tm_year + 1900) && isleap(year)) {
daydiff++;
}
}
}
}
if (mydate < 0x0003) {
// special case for day 0
linuxSeconds += 86400;
} else {
linuxSeconds += daydiff * 86400;
}
// Print the date with actual GMT
printf(ctime(&linuxSeconds));
// Print Greenwich time
printf(asctime(gmtime(&linuxSeconds)));
return 0;
}
One caveat about this code: if it's run on a system where time_t is 32-bit, it won't be able to properly display dates after 2038. If time_t is 64-bit, those dates will display properly.
EDIT 2:
There was an issue with code 30068000 being one day ahead. There was a bug in the code when checking the current month. The tm_mon field in struct tm ranges from 0 to 11, not 1 to 12. Fixed.
EDIT 3:
So it seems the month/day check when adding/subtracting days was just plain wrong, as it was causing Feb 28 to appear twice. When I removed that, I found that the 1994 scheme was a day ahead, so it looks like it has the same special case for day 0 that the 1986 scheme has. Fixed again.
Using the perfect explanation of #dbush, below you can find a simple linux gcc compiled code.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <time.h>
void decrypt ( uint32_t data )
{
uint16_t mydate = data>>16;
uint16_t mytime = data&0x0000FFFF;
// Calculate days
time_t linuxSeconds = (mydate/3);
// Calculate the 8 hours on current day of date
uint8_t third_Count = mydate % 3;
// Add days from 1/1/1970, that is the base of time in linux
if (mytime&0x8000)
{
// Days from 171/1970 to 17/12/1993
linuxSeconds += 8751;
}
else
{
// Days from 171/1970 to 19/03/1984
linuxSeconds += 5846;
}
// Calculate total amount of hours
linuxSeconds *= 24;
// Calculate total amount of seconds
linuxSeconds *= 3600;
// Add seconds of last 8 hours group
linuxSeconds += (mytime&0x7FFF);
// Add alla seconds of grups of 8 hours of date
linuxSeconds += (third_Count * 28800);
// Print the date with actual GMT
// printf(ctime(&linuxSeconds));
// Print Greenwich time
printf(asctime(gmtime(&linuxSeconds)));
}
int main(int argc, char *argv[])
{
decrypt(0x54FEC072);
decrypt(0x5501DC8B);
decrypt(0x5701DC8B);
decrypt(0x42238F96);
decrypt(0x3CF5284B);
decrypt(0x3AF42849);
return 0;
}
As #rmrps pointed out the last example is out of 1 day.
OUTPUT OF THE EXAMPLE ABOVE
Fri Oct 25 20:34:58 2013
Sat Oct 26 22:34:51 2013
Tue Apr 15 14:34:51 2014
Sat May 30 17:06:30 2009
Fri Mar 31 18:51:55 2000
Tue Oct 12 18:51:53 1999
Building on dbush' excellent analysis, there still one issue regarding the days.
This can be explained by the format, instead of counting days since epoch counts years since epoch and day of year, but ignoring leap years and using a fixed 366 day year. This sets the epoch to 31 December 1993.
So from the days_since_epoch part in dbush' answer, calculate years_since_epoch = days_since_epoch / 366 and day_of_year = days_since_epoch % 366. Set the epoch at 1993-12-31, and add the lapsed time since epoch (taking leap days into account), and you'll get the correct dates from the timestamp.
The last two timestamps you added doesn't fit dbush' analysis and so seem follow a different format. Note that the bit that dbush assumed was a DST indicator is no longer set - it actually appears to be a date scheme selector rather than DST indicator.
The date part of the last two examples is simpler - it's simply days since epoch (but a different epoch - 3 January 1986).
Edit: following discussion in chat, I'm updating my code samples.
Because my implementation - and some of the results differ from dbush' implementation and results, I'm adding an explanation as well.
If the format is based on 4 hex bytes AA BB CC DD, we treat this as a big-endian 32-bit number (AA msb = bit 31, DD lsb = bit 0), the hex timestamp is decoded as follows:
bits 0-14: number of seconds into a 8-hour window
bit 15: date scheme/epoch selector (see below)
bits 31-16 mod 3: the 8-hour window of the day.
bits 31-16 div 3: an indication of days since epoch (let's call it day_count)
day_count is not the actual number of days as both schemes treats every year as having a fixed number of days. It also includes the current (partial) day, so we should remove the partial day by subtracting one from day_count. This also means that a zero day_count is probably invalid (This has been confirmed for one of the two schemes 00000000 and 00030000 yield the same date - if 00008000 and 00038000 yield the same value it also holds for the other scheme)
For each scheme, there are only two parameters that differ between the schemes, epoch and days_per_year. Given these parameters, the calculation is the same - work out the following:
years_since_epoch = day_count / days_per_year
days_since_new_year = day_count % days_per_year // whole days
Then calculate the actual number of days since epoch as days_since_new_year plus the number of days in each year since epoch, taking leap days into account.
The two schemes are selected by bit 15:
If 1, epoch is 1994-01-01 and days_per_year is 366
If 0, epoch is 1986-01-01 and days_per_year is 365
The following code decodes both timestamp formats:
#include <stdint.h>
#include <time.h>
int isLeapYear( int y )
{
if ( y % 400 == 0 ) return 1;
if ( y % 100 == 0 ) return 0;
if ( y % 4 == 0 ) return 1;
return 0;
}
time_t decodeTimestamp (uint32_t timestamp)
{
time_t result = 0;
int y, days_since_new_year, years_since_epoch, epoch_year;
int day_count = (timestamp >> 16 ) /3;
int part_of_day = (timestamp >> 16 ) %3;
int seconds_in_day = part_of_day * 8 * 3600 + ( (timestamp & 0x7FFF) % 28800 ) ;
if ( day_count > 0 )
{
--day_count; // remove current (partial) day from day_count
}
if ( ((timestamp >> 15) & 1) == 1 ) // bit 15 is date scheme
{
days_since_new_year = day_count % 366;
years_since_epoch = day_count / 366;
epoch_year = 1994;
result = 757382400; //1994-01-01
}
else
{
days_since_new_year = day_count % 365;
years_since_epoch = day_count / 365;
epoch_year = 1986;
result = 504921600;//1986-01-01
}
result += years_since_epoch * 365 *24*60*60;
for ( y = epoch_year ; y < epoch_year + years_since_epoch; ++y )
{
if ( isLeapYear( y ) )
{
result += 24 * 60 * 60;
}
}
result += days_since_new_year * 24 * 60 * 60;
result += seconds_in_day;
return result;
}
I've put this up on http://codepad.org/K4JC0zmf, which also includes a main function which test against all of the examples I've seen in this thread. The only one it falls over is C0068000, which is explained by time_t being 32 bits not 64 bits.
Edit 2: Updated dbush' implementation in the side-by-side on codepad.
I've also put both mine and dbush' (current) implementation on http://codepad.org/vMYzNM4g to see the differences.
Both methods give the same correct results, apart from the 2038 overflow case (C0068000). I expect that they would give the same result with a 64-bit time_t however, so I think both algorithms are now correct.
E393 - DC8B = 1800
23:04:51 - 22:34:51 = 1800s
(10 Mar 2015 09:59:14 GMT - 25-oct-13 20:34:58) = 43248256 seconds
Try subtracting 43248256 decimal (293EA80 hex) from the number and then use it as if it was a unix time stamp:
54FEC072h - 293EA80h = 526AD5F2h = 1382733298d = "Fri, 25 Oct 2013 20:34:58 GMT"
I`ve a small app that returns is week even or not.
time_t now = time(0);
tm *ltm = localtime(&now);
int twin=(ltm->tm_yday/7)%2
But independently from the 1st day of the year so it returns
mon, thu, we, etc
0,1,1,1,1,1,1
in the next week
1,0,0,0,0,0,0
In the next year
mon, thu, we, etc
0,0,1,1,1,1,1
in the next week
1,1,0,0,0,0,0
and so on..
Twin- if number modulo 2 = 0
So I have to add shift to change week number in each sunday or monday. Any suggestions?
You are assuming that first week has exactly 7 days which is incorrect.
For example Jan 1st 2013 was Tuesday, so the first week is only 5 days long.
How about using strftime? Something like:
time_t now = time(0);
tm *ltm = localtime(&now);
char weekNr[3];
strftime(weekNr, sizeof(weekNr), "%W", ltm);
int isOdd = atoi(weeknr) % 2;
What you call twin, in English is usually called even.
About your question, the issue here is that you are not calculating the week number correctly: you are simply dividing by 7, and that's not enough because the start of year and the start of week vary each year.
Moreover, there are several different ways to decide which one is week 1. See for example this code, to get started.
UPDATE: Copying shamelessly from the eglibc source code:
1) The week number of the current year as a decimal number, range 00 to 53, starting with the first Sunday as the first day of week 01 (strftime("%U")):
tp->tm_yday - tp->tm_wday + 7) / 7
2) The week number of the current year as a decimal number, range 00 to 53, starting with the first Monday as the first day of week 01 (strftime("%W")):
(tp->tm_yday - (tp->tm_wday - 1 + 7) % 7 + 7) / 7
3) The ISO 8601 week number (see NOTES) of the current year as a decimal number, range 01 to 53, where week 1 is the first week that has at least 4 days in the new year (strftime("%V")):
Well this is complicated... so you are better with the idea by #MaikuMori of using strftime``, but with"%V", and then parse the result, withatoi()`.
How to segregate ddmmyyhhmmss (day, month, year, hour, minute, second) 12 digit number and put in the separate 6 column in MATLAB? I need 7 column now in total. For example, I have following string
051210151255 (which is 05 day, 12 month, 2010 year, 15 hours, 12 minute, 55 second)
Now I need 7 column (whole 12 digit number, dd, mm, yy, hh, mm, ss)
d = '051210151255';
output = zeros(1, 6);
for i=1:6
output(1, i) = str2num(d(i*2-1:i*2));
end
output = [str2num(d) output];
gives:
>> output =
051210151255 5 12 10 15 12 55