In the given example, I have two lines to be displayed and datasets/time series data records for them are here dataset1: mock-data/i.js & dataset2: mock-data/iv.
Time range is between 2015 to 2021 years. In each data set there are around 60K to 70K records.
In reality it would be millions of records in each dataset and there will be multiple dataset/lines(40 to 60 datasets/lines).
Question: 1The part which I don't understand is, when I load the chart for the first time, which data should I display ??????? (as there are many records in the dataset)
I checked highcharts example: https://www.highcharts.com/demo/stock/lazy-loading and I feel like, from given time range, it shows each month's first date data when being loaded for the first time.
function getInitialDataPointsFromBackEnd() {
const onLoadDataset = [];
let timestamp;
for (let i = 0; i < dataset.length; i++) {
const { name, datapoints } = dataset[i];
const tempDataset = { name: name, datapoints: [] };
for (let j = 0; j < datapoints.length; j++) {
// push 0th record blindly to onLoadDataset
if (j === 0) {
tempDataset.datapoints.push(datapoints[j]);
timestamp = datapoints[j][0];
timestamp = timestamp + 2.628e9; // timestamp with one month difference
}
// push last record blindly to onLoadDataset
if (j === datapoints.length - 1) {
tempDataset.datapoints.push(datapoints[j]);
onLoadDataset.push(tempDataset);
}
// start finding next month timestamp record
const filteredMonthlyRecord = datapoints.find(
(x) => x[0] === timestamp
);
if (filteredMonthlyRecord) {
// if record is found increse time stamp by one month
timestamp = timestamp + 2.628e9; // timestamp with one month difference
tempDataset.datapoints.push(datapoints[j]);
}
}
}
return onLoadDataset;
}
So I'm trying to apply the similar kind of logic using getInitialDataPointsFromBackEnd function. Assume this is BE functionality or BE implementation in reality but just to make example more understandable I'm using in FE side, which should extract out each month's first record for given range(In my case time range is from 2011 to 2021).
But, since, in each dataset, there are around 60K to 70K records, I have to loop through all of them and prepare my first-load data.
This is very time consuming process. If I have 60 to 70 datasets and from each dataset If I have to extract out each month record, it will take forever to serve the data to front-end.
I'm sure, I'm missing something or making some basic mistakes.
Please help me understand what should be the onLoad dataset if you have two time series datasets as shown below.
My efforts: https://stackblitz.com/edit/js-srglig?file=index.js
Question 2: Also, bottom navigator keeps on updating every time when you try to change the selection and redraws lines in wrong way. How can I fix it too ? Ideally it should not update lining part only scroll trasparent window should be updated. isn't it?
Related
I have a transactions sheet,
In column A, there is the type of the transaction.
In column D, there is the description of the transaction
I would like all the rows where the column A is "ATM" to have the same description.
This is what I wrote, a typical loop. However it takes AGES :D So if you know another technique with let's say, an indexation of all the rows to modify with a bulk change of all the values, it would be wonderful :D
Thanks a lot !
function ATM() {
// changes the description of the ATM lines into Cash withdrawal
var spreadsheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1");
var rowCount = spreadsheet.getDataRange().getNumRows();
for (i=1; i<=rowCount; i++) {
var ATM = spreadsheet.getRange(i,1).getValue() ; // IF A column cell is ATM then
if (ATM=="ATM") {
spreadsheet.getRange(i,4).setValue("Cash withdrawal"); // Changes the 4th column of the row into cash withdrawal
}
}
spreadsheet.getRange('A1').activate();
};
I tried the code I wrote above. It works perfectly but it takes ages. I would like to speed the process up :)
I believe your goal is as follows.
You want to reduce the process cost of your script.
In this case, how about the following modifications?
Modied script 1:
In this pattern, the column "D" is overwritten.
function ATM() {
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1");
var range = sheet.getRange("A1:D" + sheet.getLastRow());
var values = range.getValues().map(r => [r[0] == "ATM" ? "Cash withdrawal" : r[3]]);
range.offset(0, 3, values.length, 1).setValues(values);
sheet.getRange('A1').activate();
}
Modied script 2:
In this pattern, TextFinder is used.
function ATM() {
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1");
var ranges = sheet.getRange("A1:A" + sheet.getLastRow()).createTextFinder("ATM").matchEntireCell(true).findAll().map(r => r.offset(0, 3).getA1Notation());
sheet.getRangeList(ranges).setValue("Cash withdrawal");
sheet.getRange('A1').activate();
}
References:
map()
createTextFinder(findText) of Class Range
You can try looping but after importing the values with .getValues; and then set values with the whole array. If the spreadsheet has too many columns you could specify a smaller range (say .getRange("A:E").getValues())
function ATM() {
// changes the description of the ATM lines into Cash withdrawal
var spreadsheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1");
var rowCount = spreadsheet.getDataRange().getNumRows();
var data = spreadsheet.getDataRange().getValues()
//Logger.log(data)
for (i=1; i<=rowCount-1; i++) {
if(data[i][0] == "ATM"){
data[i][3] = "Cash withdrawal"
}
}
spreadsheet.getDataRange().setValues(data)
}
With data in 1000 rows and 5 columns it took 3 seconds:
I'm working on a sheet to build a list of products to import into Shopify.
For this, I have a pdf of some basic data (that is irrelevant here) out of which I build a string to crawl the product supplier's website and format the data in a way suitable for import in Shopify.
The products have a varying number of images (1 - 8), so I'm trying to build my script in a way that if a product has more than one image, I am trying to add additional rows under it and add every image past the first into a new row.
Here is my code:
function iterateThroughRows() {
// get spreadsheet
const sheet = SpreadsheetApp.getActive().getSheetByName("MySheet");
const data = sheet.getDataRange().getValues();
// Loop over rows
data.forEach( (row, rowIndex) => {
const imageSrcArray = [ /* list of image URLs, fetched from remote server */ ]
imageSrcArray.forEach( (img, arrayIndex) => {
if(arrayIndex == 0) { // for the first array item, just add it to the current row
const imageCell = sheet.getRange(rowIndex + 1, 24)
imageCell.setValue( imageSrcArray[arrayIndex] )
} else { // for each array item past the first one, add a new row and enter the value there
sheet.insertRows(rowIndex)
const imageCell = sheet.getRange(rowIndex + arrayIndex + 1, 24)
imageCell.setValue( imageSrcArray[arrayIndex] )
}
})
// adding some more values to other cells
});
}
As is this doesn't really work.
I worked on this all day yesterday and had a version using insertRowAfter() that did add additional rows, but added them all lumped together (i.e. there would be 15 rows after the first product, but none after any of the others). But since Google App Script doesn't have version control I lost that version.
I think the problem was that the forEach seems to move on to the newly created rows and keeps adding things from there rather than moving on to the initial next row.
So I'm more or less at the end of my wit with this. Any advise on how to properly do this would be highly appreciated.
I can understand your frustration, it is indeed because you care calculating the row based on the sheet in its version before you added new rows to it.
So my proposal would be to do this, as the currentRow allows you to track the current row you are working on. I also updated the insertRowAfter(), as I assume this is what you actually wanted to do.
let currentRow = 1;
data.forEach( (row, rowIndex) => {
const imageSrcArray = [ "img1URL", "img2URL"]
if( !imageSrcArray.length ) return
imageSrcArray.forEach( (img, arrayIndex) => {
if( arrayIndex == 0 ){
sheet.getRange(currentRow, 24).setValue( img )
} else {
sheet.insertRowAfter(currentRow)
sheet.getRange(currentRow+1, 24).setValue( img )
}
// New rows in between were created
currentRow++
})
});
Hi I want to make a function that generates available time slots. It should generate the time slots while keeping in mind that the time slot can't overlap with an already made appointment.Before the time slots are generated a user can specify which kind of appointment to schedule. Each appointment sort has a duration. So it should also check if the time slot added with the duration doesn't overlap.
I'm struggling to make this all working so far I get time slots but it seems to only checks the start of an already made appointment. I'm kind of running in circles here and would love for some advice or part solutions that I can implement to make my idea work
const GenerateAvailableTimeSlots = (start, serviceObject, allAppointments) => {
const moment = extendMoment(Moment);
var x = {
nextSlot: 15,
appointmentsOfThatDay: [],
startTime: '8:00',
endTime: '20:00'
};
// function to filter only the appointment that occur on specified day --> ( start )
let filterAppointments = (allAppointments, start) => {
let results = [];
let filterAppoinments = allAppointments.filter(appoinment => appoinment.date === start.format('MMMM Do YYYY'));
filterAppoinments.map(appoinment => results.push([appoinment.start.format('HH:mm'), appoinment.end.format('HH:mm')]))
console.log("results", results);
return results;
};
x.appointmentsOfThatDay = filterAppointments(allAppointments, start)
console.log("appointmentsOfThatDay", x.appointmentsOfThatDay)
var slotTime = moment(x.startTime, "HH:mm");
var endTime = moment(x.endTime, "HH:mm");
// function to check time slot overlaps with already made appointments
function OverlapsScheduledAppointment(slotTime, appointments) {
//added duration to timeslot so I could check if a suggested timeslot + the duration also doesn't overlap with already made appointment
var slotTimeWithDuration = slotTime.clone().add(serviceObject.hours, 'hours').add(serviceObject.minutes, 'minutes');
// I don't know where I also could check for slotTimeWithDuration overlap
return appointments.some((br) => {
console.log(slotTime >= moment(br[0], "HH:mm") && slotTime < moment(br[1], "HH:mm"));
return (slotTime >= moment(br[0], "HH:mm") && slotTime < moment(br[1], "HH:mm"));
});
}
let times = [];
while (slotTime < endTime) {
if (!OverlapsScheduledAppointment(slotTime, x.appointmentsOfThatDay)) {
times.push(slotTime.format("HH:mm"));
}
slotTime = slotTime.add(x.nextSlot, 'minutes');
}
return times;
};
I've found the answer to my question.
I was going in the right direction with the above code but in order for generating available time slots that keep in mind the duration of the service you want to schedule and the appointment that are already scheduled.
I had to change this line of code:
// this line just pushes the filtered appointment for a specific day
filterAppoinments.map(appoinment => results.push([appoinment.start.format('HH:mm'), appoinment.end.format('HH:mm')]))
To this
// this line filters the appointment for a specific day and also adds the duration of a service to the start time of an already scheduled appointment. This way when I check if a generated time slot for a service will overlap with an already scheduled appointment it filters out the ones that will overlap
filterAppoinments.map(appoinment => results.push([appoinment.start.clone().subtract(serviceObject.hours, 'hours').subtract(serviceObject.minutes, 'minutes').format('HH:mm'), appoinment.end.format('HH:mm')]))
I have a function that takes in two very large arrays. Essentially, I am matching up orders with items that are in a warehouse available to fulfill that order. The order is an object that contains a sub array of objects of order items.
Currently I am using a reduce function to loop through the orders, then another reduce function to loop through the items in each order. Inside this nested reduce, I am doing a filter on items a customer returned so as not to give the customer a replacement with the item they just send back. I am then filtering the large array of available items to match them to the order. The large array of items is mutable since I need to mark an item used and not assign it to another item.
Here's some psudocode of what I am doing.
orders.reduce(accum, currentOrder)
{
currentOrder.items.reduce(internalAccum, currentItem)
{
const prevItems = prevOrders.filter(po => po.customerId === currentOrder.customerId;
const availItems = staticItems.filter(si => si.itemId === currentItem.itemId && !prevItems.includes(currentItem.labelId)
// Logic to assign the item to the order
}
}
All of this is running in a MESOS cluster on my server. The issue I am having is that my MESOS system is doing a health check every 10 seconds. During this working of the code, the server will stop responding for a short period of time (up to 45 seconds or so). The health check will kill the container after 3 failed attempts.
I am needing to find some way to do this complex looping without blocking the response of the health check. I have tried moving everything to a eachSerial using the async library but it still locks up. I have to do the work in order or I would have done something like async.each or async.eachLimit, but if not processed in order, then items might be assigned the same thing simultaneously.
You can do batch processing here with a promisified setImmediate so that incoming events can have a chance to execute between batches. This solution requires async/await support.
async function batchReduce(list, limit, reduceFn, initial) {
let result = initial;
let offset = 0;
while (offset < list.length) {
const batchSize = Math.min(limit, list.length - offset);
for (let i = 0; i < batchSize; i++) {
result = reduceFn(result, list[offset + i]);
}
offset += batchSize;
await new Promise(setImmediate);
}
return result;
}
I'm setting up a fairly complex Google sheet and trying to automate some routine interpolation with a script. I now have a script that works, but I want to optimise it.
Let me briefly describe the set up with some (simple) example data:
A B C D E
1 Lookup date Result Recorded date Value
2 17/8/2018 - 31/12/2018 210
3 31/12/2018 210 31/3/2019 273
4 14/2/2019 241.5 12/6/2019 411
5 31/3/2019 273
6 12/6/2019 411
7 1/7/2019 411
In this example, I have a small number of recorded values (columns D and E) and I want to compute the value for any date (column A). Column B is the output of the script. The problem is that my script is very slow, taking quite a while on my laptop (sometimes I must refresh the page), and never fully executing on my iPad.
I think part of this may be the volume of requests: I run this script for about 200 cells in my sheet.
I will briefly explain the script (full javascript code below).
It creates a custom function getvalue(x, y, lookupdate) which, for a given x-range (col. D) y-range (col. E) and "lookup date" (eg A4) will return the correct result (eg B4). This result is either:
blank if the lookup date occurs before the first recorded date
the exact value if the lookup date equals a recorded date
an interpolated value if the lookup date is in between two recorded dates
the final recorded value if the lookup date is beyond the range of the recorded dates
Now I have optimised this somewhat. In my implementation, I actually run it as an array for 100 cells in column A (only some of which actually need to run the script). I have another simple system that basically auto-populates the date in column A as a binary flag to say the script needs to run. So using ISBLANK() as a switch, my array formula for cells B3:B103 is:
=ArrayFormula(IF(ISBLANK(A3:A103),"",getvalue(D:D,E:E,A3:A103)))
Even though the array covers 100 cells, only about 50 of them are "activated" with a date in the A column, so only about 50 of them actually need to run the getvalue function. However, as a final complication, I am actually doing this to calculate four different values for each "lookup date", running four different arrays in four columns, so that's what I say the script runs approx. 200 times.
Here is my actual script:
function getvalue(x, y, lookupdate) {
/// LOOKUP AN ARRAY
if (lookupdate.map) {
return lookupdate.map(function(v) {
return getvalue(x, y, v);
});
}
/// GET RID OF EMPTY CELLS IN COLUMN
var xf = x.filter(function(el) {
return el != "";
});
var yf = y.filter(function(el) {
return el != "";
});
/// GET RID OF HEADER ROW
xf.shift()
yf.shift()
/// SAVE THE FIRST AND LAST VALUES
var firstx = xf[0][0]
var firsty = yf[0][0]
var lastx = xf[xf.length - 1][0]
var lasty = yf[yf.length - 1][0]
/// FIGURE OUT WHAT TO RETURN
if (lookupdate < firstx) {
return "";
} else if (lookupdate.valueOf() == firstx.valueOf()) {
return firsty;
} else if (lookupdate > lastx) {
return lasty;
} else {
var check = 0, index;
for(var i = 0, iLen = xf.length; i < iLen; i++) {
if(xf[i][0] == lookupdate) {
return yf[i][0];
} else {
if(xf[i][0] < lookupdate && ((xf[i][0] - check) < (lookupdate - check))) {
check = xf[i][0];
index = i;
}
}
}
var xValue, yValue, xDiff, yDiff, xInt;
yValue = yf[index][0];
xDiff = xf[index+1][0] - check;
yDiff = yf[index+1][0] - yValue;
xInt = lookupdate - check;
return (xInt * (yDiff / xDiff)) + yValue;
}
}
The error message on the iPad is simply the cells never move past "Loading...", and on the laptop it takes much longer than expected.
The most confusing thing is that I think it has gotten worse since I set it up as an array. I previously had it where all 400 cells would run the ISBLANK() check, and then for the approx 200 triggered cells, they would individually run the script. This at least would load on the iPad.
I read on here and on general Google support that scripts will run a lot faster if you batch operations, but it seems to have gotten slower since moving from 200 single cells to 4 arrays.
Does this need to be optimised further, or is there some other reason it might be stalling on my iPad?
Is it even possible to optimise it down and do this in a single call, instead of in 4 arrays?
Accounting for the case
else if (lookupdate.valueOf() == firstx.valueOf()) return firsty;
is superfluous because it is covered already by if(xf[i][0] == lookupdate)
(xf[i][0] - check) < (lookupdate - check) can be simplified to xf[i][0] < lookupdate
You are using pure javascript code, but keep in mind that App Script has many additional functions which are handy when working with Spreadsheet.
https://developers.google.com/apps-script/reference/spreadsheet/
So, e.g. for running your function only for the populated range functions like getDataRange() or getRange() in combination with getNextDataCell() and getLastRow() will be very useful for you.
The main important point - the functionality of your script. Are you assuming that there is an approximately linear relationship between Recorded date and value, and thus interpolate the value for not recorded dates?
In this case the statistically most precise way (and the programmatically simplest one) would be to calculate your slope between the first and last x and y respectively. That is:
Result=first_y+((y_last-y_first)/(x_last-x_first)*(Lookup_Date-first_x))
If this approach is suitable for you, your code would simplify and would look in App Script something like:
function myFunction() {
var ss=SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
var Result_Range=ss.getRange("A2:B")
var limit=Result_Range.getNextDataCell(SpreadsheetApp.Direction.DOWN).getLastRow()
var Result_values=Result_Range.getValues();
var valueRange=ss.getRange("D1:E");
var values=valueRange.getValues();
var last_Index=valueRange.getNextDataCell(SpreadsheetApp.Direction.DOWN).getLastRow()
var last_y=values[last_Index-1][1];
var last_x=values[last_Index-1][0].valueOf();
var first_y=values[1][1];
var first_x=values[1][0].valueOf();
var slope=(last_y-first_y)/(last_x-first_x);
for(var i=1;i<limit;i++)
{
Result_Range.getCell(i,2).setValue(first_y+(slope*(Result_values[i-1][0].valueOf()-first_x)))
Logger.log(i)
Logger.log(Result_values[i][0].valueOf()-first_x)
}
}