I am trying to loop through dates from a selected near date, going a period of four weeks back each time, for 12 periods.
I am doing something wrong in the arrays to push the data to the Results sheet, and it is not working.
Thanks in advance for your help.
function loopDates() {
var sss = SpreadsheetApp.getActiveSpreadsheet();
var ss = sss.getSheetByName('Main');
var results = sss.getSheetByName('Results');
// gets values from Main sheet to calculate new Date
var firstYear = ss.getRange('Main!C3').getValue();
var firstMonth = ss.getRange('Main!E3').getValue();
var firstDay = ss.getRange('Main!G3').getValue();
var target = new Array();// this is a new array to collect data
var headers = [];//inner array
var periods = 13;
var counter = 0;
//loop thru 12 dates and get just the dates for the header
for (var i = 0; i < periods-1; i++) {
counter = counter + 1;
// yearValue,monthValue,day Calculate new Date
var date = [firstYear,firstMonth,firstDay];
var d = new Date(firstYear,firstMonth,firstDay); // must have commas day,
month numbers start at 0
var numberDays = 28;
var countBack = 2*numberDays * 24 * 60 * 60 * 1000 + 3* 24 * 60 * 60 *
1000;
//sign must be '+' since 'offset' is subtracted
var NewDate = new Date(d.getTime() - countBack);
ss.getRange('C5').setValue(new Date(NewDate)).setNumberFormat('mm dd
yyyy');
headers.push([NewDate[i]]); // inner array
}
target.push(headers); // outer array
//Write to Results sheet
var ss = sss.getSheetByName('Main').getRange('A1').setValue(counter);
results.getRange("A1").offset(0, 1, target.length).setValue(target);
return target;
}
Related
There is start date and end date. Using moment ,will get difference between 2 dates in hours.
var now = moment(sessionData.StartTime);
var end = moment(sessionData.EndTime);
var duration = moment.duration(end.diff(now));
var days = duration.asHours();
it returns : 3.08 .
I want to show that difference like this- 3 hrs 15 m.
Is this possible to achieve
Try get the difference between two days using diff followed by expressing it terms of duration. Then you would be able to get exact number of days, hours, minutes
let now = moment("2017-01-26T14:21:22+0000");
let expiration = moment("2017-01-29T17:24:22+0000");
let diff = expiration.diff(now);
let diffDuration = moment.duration(diff);
let dayDiff = diffDuration.days() + "d";
let hoursDiff = diffDuration.hours() + "hrs";
let minDiff = diffDuration.minutes() + "m";
console.log(`${dayDiff} ${hoursDiff} ${minDiff}`);
In the code below cohort_counts_4 is a dataframe that has 3 columns g,samplingRate and samplingRate1. In the rowDF variable I am collecting the columns samplingRate and samplingRate1 (which are percentages). And in the
percentages variable I am converting it to Array[Double].
When I am trying to run this, I am getting the error below during runtime in the percentages. I need it to be Array[Double] as I have to randomSplit in the next step.
java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to java.lang.Double.
Please let me know your thoughts.
sample data of percentages -
percentages: Array[Seq[Double]] =
Array(WrappedArray(0.06449504858964898, 0.9355049514103511)
, WrappedArray(0.015861918725594032, 0.9841380812744059)
, WrappedArray(0.22082241578907924, 0.7791775842109208)
, WrappedArray(0.14416119376185044, 0.8558388062381496)
, WrappedArray(0.10958692395592619, 0.8904130760440738)
, WrappedArray(1.0, 0.0)
, WrappedArray(0.6531128743810083, 0.3468871256189917)
, WrappedArray(0.04880653326943304, 0.9511934667305669))
val cohortList = cohort_counts_4.select("g").collect().map(_(0)).toList
var cohort_list = new ListBuffer[org.apache.spark.sql.DataFrame]()
var total_rows: Int = 0
for (igroupid<-cohortList){
val sample_rate = cohort_counts_4.filter(col("g")===igroupid).select("samplingRate","samplingRate1")
cohort_list += sample_rate
val curr_rows = sample_rate.count().toInt
total_rows += curr_rows
}
val customers_new = cohort_list.reduce(_ union _)
val rowDF = customers_new.select(array(customers_new.columns.map(col):_*) as "row")
var percentages =Array(rowDF.collect.map(_(0)).asInstanceOf[Double])
// var percentages = rowDF.collect.map(_.getSeq[Double](0))
val uni = customers_2.select("x","g").distinct
.randomSplit(percentages)
I changed the code from
var percentages =Array(rowDF.collect.map(_(0)).asInstanceOf[Double])
to below
var percentages =rowDF.collect.map(_(0).asInstanceOf[Seq[Double]]).flatten
and it worked.
I'm trying to map the max number of consecutive days with rain <1 mm in Google Earth Engine.
This is the link to the code
https://code.earthengine.google.com/22b5c20d2700a2ffb5989f892838ac58
First I reclassify the collection with 0 if rain <=1 and 1 if >1.
Then I run the code that should count the days of the longest dry period, but it is able to do so only if the dry period reach the end of the time period.
For instance if I am looking for the longest dry period in 4 days timestep i get the following series:
rain days 1 2 3 4 output
0,0,1,1 = 0 dry days
0,1,0,0 = 2 dry days
0 = rain<=1 and
1 = rain>1 (as per the first step)
Does anyone can help in solving this?
Thanks
I don't think you were far off in your code that you provided. To keep track of the dry spells you have to use .iterate(). I took a stab at your application in a little different way where instead of classifying the data before the iteration, I calculate which pixels are dry each day and carry over the accumulated days that a pixel is dry, otherwise it is set to zero:
// DATA
var collection = ee.ImageCollection("UCSB-CHG/CHIRPS/DAILY");
// Define time range
var startyear = 2000;
var endyear = 2017;
var startmonth = 1;
var endmonth = 12;
// Set date in ee date format
var startdate = ee.Date.fromYMD(startyear,startmonth,1);
var enddate = ee.Date.fromYMD(endyear,endmonth,31);
// Filter data
var datain_t = collection.filterDate(startdate, enddate)
.filter(ee.Filter.calendarRange(startmonth,endmonth, 'month'))
.select("precipitation").map(function(img){
return img.addBands(ee.Image.constant(0).uint8().rename('counter'));
})
.sort('system:time_start');
// // START
var dataset = datain_t
.filterDate("2016-08-01","2016-08-30")
.sort('system:time_start:');
print(dataset,"dataset");
var precipThresh = 1; // mm
function drySpells(img, list){
// get previous image
var prev = ee.Image(ee.List(list).get(-1));
// find areas gt precipitation threshold (gt==0, lt==1)
var dry = img.select('precipitation').lt(precipThresh);
// add previous day counter to today's counter
var accum = prev.select('counter').add(dry).rename('counter');
// create a result image for iteration
// precip < thresh will equal the accumulation of counters
// otherwise it will equal zero
var out = img.select('precipitation').addBands(
img.select('counter').where(dry.eq(1),accum)
).uint8();
return ee.List(list).add(out);
}
// create first image for iteration
var first = ee.List([ee.Image(dataset.first())]);
// apply dry speall iteration function
var maxDrySpell = ee.ImageCollection.fromImages(
dataset.iterate(drySpells,first)
).max(); // get the max value
// display results
Map.addLayer(maxDrySpell.select('counter'),{min:0,max:30,palette:'#9ecae1,#ffffff,#ffeda0,#feb24c,#f03b20'},'Max Dry Spells');
Here is the link to the code: https://code.earthengine.google.com/80b4c0f7e82a5f0da316af1d2a55dd59
Don't try to run this analysis for too long of a time period or Earth Engine will give an error. I hope this helps!
Ok, I realize this is a very niche issue, but I'm hoping the process is straight forward enough...
I'm tasked with creating a data file out of Customer/Order information. Problem is, the datafile has a 5 product max limit.
Basically, I get my data, group by cust_id, create the file structure, within that loop, group by product_id, rewrite the fields in previous file_struct with new product info. That's worked all well and good until a user exceeded that max.
A brief example.. (keep in mind, the structure of the array is set by another process, this CANNOT change)
orderArray = arranyew(2);
set order = 1;
loop over cust_id;
field[order][1] = "field(1)"; // cust_id
field[order][2] = "field(2)"; // name
field[order][3] = "field(3)"; // phone
field[order][4] = ""; // product_1
field[order][5] = ""; // quantity_1
field[order][6] = ""; // product_2
field[order][7] = ""; // quantity_2
field[order][8] = ""; // product_3
field[order][9] = ""; // quantity_3
field[order][10] = ""; // product_4
field[order][11] = ""; // quantity_4
field[order][12] = ""; // product_5
field[order][13] = ""; // quantity_5
field[order][14] = "field(4)"; // trx_id
field[order][15] = "field(5)"; // total_cost
counter = 0;
loop over product_id
field[order[4+counter] = productCode;
field[order[5+counter] = quantity;
counter = counter + 2;
end inner loop;
order = order + 1;
end outer loop;
Like I said, this worked fine until I had a user who ordered more than 5 products.
What I basically want to do is check the number of products for each user if that number is greater than 5, start a new line in the text field, but I'm stufk on how to get there.
I've tried numerous fixes, but nothing gives the results I need.
I can send the entire file if It can help, but I don't want to post it all here.
You need to move the inserting of the header and footer fields into product loop eg. the custid and trx_id fields.
Here's a rough idea of one why you can go about this based on the pseudo code you provided. I'm sure that there are more elegant ways that you could code this.
set order = 0;
loop over cust_id;
counter = 1;
order = order + 1;
loop over product_id
if (counter == 1 || counter == 6) {
if (counter == 6) {
counter == 1;
order= order+1;
}
field[order][1] = "field(1)"; // cust_id
field[order][2] = "field(2)"; // name
field[order][3] = "field(3)"; // phone
}
field[order][counter+3] = productCode; // product_1
field[order][counter+4] = quantity; // quantity_1
counter = counter + 1;
if (counter == 6) {
field[order][14] = "field(4)"; // trx_id
field[order][15] = "field(5)"; // total_cost
}
end inner loop;
if (counter == 6) {
// loop here to insert blank columns and the totals field to fill out the row.
}
end outer loop;
One thing goes concern me. If you start a new line every five products then your transaction id and total cost is going to be entered into the file more than once. You know the receiving system. It may be a non-issue.
Hope this helps
As you put the data into the row, you need check if there are more than 5 products and then create an additional line.
loop over product_id
if (counter mod 10 == 0 and counter > 0) {
// create the new row, and mark it as a continuation of the previous order
counter = 0;
order = order + 1;
field[order][1] = "";
...
field[order][15] = "";
}
field[order[4+counter] = productCode;
field[order[5+counter] = quantity;
counter = counter + 2;
end inner loop;
I've actually done the export from an ecommerce system to MOM, but that code has since been lost. I have samples of code in classic ASP.
I have an algorithm which scans through data read from a .csv file(approx 3700 lines) and assess's which trading week of the year each entry is in by running a count++ for every Sunday of that year and assigning the count value as the trading week when the date falls within that week.
It's working but performance is lagging. It is the 3rd function running using Task.Factory.StartNew (I have also tried parallel.Invoke).
Results of timing tests.
before: 00:00:05.58
after: 00:00:23.27
UPDATE
Added break after each trading week is set. Time improved but still slow.
new time: 00:00:15.74
For our purposes the 1st week of the year is week 1(not 0) and is defined as from the first day of the year until the Sunday. If the first day of the year is a Sunday the length of week 1 is 1 day.
private void SetDefiniteWeeks()
{
string FileLoc = FilePath + Market + ".csv";
string[] Data = File.ReadAllLines(FileLoc);
var FileData = from D in Data
let DataSplit = D.Split(',')
select new
{
Date = DateTime.Parse(DataSplit[0]),
ClosingPrice = double.Parse(DataSplit[4])
};
//assign each date to it's relevant week
TradingWeek TW;
List<TradingWeek> tradingWeek = new List<TradingWeek>();
foreach (var pe in FileData)
{
// DateTime dt = pe.Date;
int Year = pe.Date.Year;
string End_of_Week = "Sunday";
int WeekCount = 0;
DateTime LoopDate_Begin = new DateTime(Year,1,1);
DateTime LoopDate_End = new DateTime(Year,12,31);
do
{
if (LoopDate_Begin.DayOfWeek.ToString() == End_of_Week)
{
WeekCount++;
if (LoopDate_Begin.DayOfYear > pe.Date.DayOfYear && LoopDate_Begin.DayOfYear < (pe.Date.DayOfYear + 7))
{
TW = new TradingWeek { Week = WeekCount, Date = pe.Date };
tradingWeek.Add(TW);
break;
}
}
LoopDate_Begin = LoopDate_Begin.AddDays(1);
} while (LoopDate_Begin.Date.ToString() != LoopDate_End.Date.ToString());
}
}
Please help.
UPDATE
NEW TIME
00:00:06.686
A vast improvement. Thanks all for your help.
Revised code:
CalendarWeekRule cw = CalendarWeekRule.FirstDay;
var calendar = CultureInfo.CurrentCulture.Calendar;
var trad_Week = (from pe in FileData
select new TradingWeek
{
Date = pe.Date,
Week = (calendar.GetWeekOfYear(pe.Date, cw,DayOfWeek.Sunday))
}).ToList();
Im not sure if this is what you want but after reading the comments I got the feeling that this might work (?)
var calendar = CultureInfo.CurrentCulture.Calendar;
var tradingWeek = (from pe in FileData
select new TradingWeek
{
Date = pe.Date,
Week = calendar.GetWeekOfYear(pe.Date, CalendarWeekRule.FirstDay, DayOfWeek.Sunday);
}).ToList();
Edit: Changed to CalendarWeekRule.FirstDay since it's (more?) what OP is looking for.
Three quick thoughts:
Why are you only adding one day each time and checking to see if it's Sunday. Surely once you have found your first Sunday you can add seven days to find the next one?
If you order your pes by DateTime before you start then you don't need to restart at the beginning of the year for each one, you can pick up where you left off.
As Nicolas says, break after adding the trading week. No need to go through the rest of the year after you already know what the answer is.
I guess you'll end up with something like this (may or may not actually work, but should be close enough)
TradingWeek TW;
List<TradingWeek> tradingWeek = new List<TradingWeek>();
string End_of_Week = "Sunday";
var orderedData = FileData.OrderBy(x => x.Date)
DateTime LoopDate_Begin = new DateTime(orderedData[0].Date.Year,1,1);
int WeekCount = 1;
while (LoopDate_Begin.DayOfWeek.ToString() != End_of_Week)
{
LoopDate_Begin = LoopDate_Begin.AddDays(1);
}
foreach (var pe in orderedData)
{
do
{
if (LoopDate_Begin.DayOfYear > pe.Date.DayOfYear && LoopDate_Begin.DayOfYear < (pe.Date.DayOfYear + 7))
{
TW = new TradingWeek { Week = WeekCount, Date = pe.Date };
tradingWeek.Add(TW);
break;
}
WeekCount++;
LoopDate_Begin = LoopDate_Begin.AddDays(7);
} while (true); //need to be careful here
}
if I get you correctly, you don't need to look any further as soon as you've added your TradingWeek
So, you can
break;
after
tradingWeek.Add(TW);
You could then even leave out the
&& LoopDate_Begin.DayOfYear < (pe.Date.DayOfYear + 7)
condition since the first part is going to be true only once: for your desired interval.
You might even go for a loopless approach by dividing the number of days since your starting week by 7 - and doing some cleaning up work ;)
Can you get rid of your do loop altogether by calculating the Week Number directly? Something like the accepted answer here.
Following #nicolas78's response, something like this should work
int Year = pe.Date.Year;
DateTime Year_Begin = new DateTime(Year,1,1);
int Jan1DayOfWeek = Year_Begin.DayOfWeek;
foreach (var pe in FileData)
{
int WeekCount = (pe.Date.DayOfYear - Jan1DayOfWeek) % 7 + 1;
TradingWeek TW = new TradingWeek { Week = WeekCount, Date = pe.Date };
tradingWeek.Add(TW);
}
Depending on how DayOfWeek and DayOfYear count, that is from 0 or 1, and how your mod operation work, you may need to tweak the WeekCount computation a bit.
There's a built-in feature to get the week of the year based on the date in .NET. An example is shown below, but it may need some tweaking to fit your business scenario:
System.Globalization.CultureInfo myCI = new System.Globalization.CultureInfo("en-US");
int week = myCI.Calendar.GetWeekOfYear(DateTime.Now.ToUniversalTime(), System.Globalization.CalendarWeekRule.FirstFourDayWeek, System.DayOfWeek.Sunday);
You don't need to count at all - just do a quick calculation. This assumes that a partial week at the start of the year is week 1 and week 2 begins on the first Monday.
List<TradingWeek> tradingWeek = new List<TradingWeek>();
foreach (var pe in FileData)
{
var date = pe.Date;
while (date.DayOfWeek != DayOfWeek.Sunday)
date = date.AddDays(1);
var week = date.DayOfYear/7+1;
var TW = new TradingWeek {Week = week, Date = pe.Date};
tradingWeek.Add(TW);
}