I want to batch my records for uploading so I don't create server issues. I would like to be able to push 10 records at a time, every five seconds, just to prove the concept for now. I've put setInterval functions all over my code but can't get it to run at the right time. I've been at this for days but can't figure it out.
chunkData(data) {
const maxRecords = 10;
const loops = (data.length % maxRecords > 0) ? Math.floor(data.length / maxRecords) + 1 : data.length / maxRecords;
//console.log('data: ', data);
//console.log('loops: ', loops);
//setInterval(() => {
for (let loop = 0; loop < loops; loop++) {
console.log('loop: ', loop);
let start = loop * maxRecords;
//setInterval(() => {
for (let batch = start; batch < start + maxRecords; batch++) {
// the line below will become the upload function once I get this to work
if (data[batch] !== undefined) console.log('data[batch]: ', data[batch]);
}
//start = start + 10;
//}, 5000);
}
//}, 5000);
}
I'm certain it's a simple tweak I need but I'm clueless as to how to make it happen right now. Any advice would be greatly appreciated.
The bigger problem you'll have to figure out is that a client-side change will not help in this case. If you're trying to help your server this way, what happens when there is more than 1 concurrent user uploading? 2? 3? 100? 1000? This solution isn't scalable. You'll eventually (or very quickly) have to make sure your server is robust enough to handle upload traffic.
As for your specific code. Your problem is that you're using setInterval inside a for-loop but use the same value. Remember, uploading (or any XHR/fetch request) is an asynchronous action. Right now, you're setting the intervals to run at basically the same time.
To get actual intervals between uploads, you'd need something like this:
for (let loop = 0; loop < loops; loop++) {
console.log('loop: ', loop);
let start = loop * maxRecords;
for (let i=1, batch = start; batch < start + maxRecords; i++, batch++) {
// the line below will become the upload function once I get this to work
if (data[batch] !== undefined) {
setInterval(() => {
//make upload request here
}, (loop + 1) * i * 5000);
}
}
}
I'm not sure what your "start" variable is supposed to be.
In any case, this code is really error prone and fragile. I really advise reconsidering your approach and look into fixing your server side.
If you still wish to go with this client-side hack and even if not, and you're looking for a more stable client-side solution. I advise to go with react-uploady. It takes care of the uploads for you and all the edge cases that come with managing uploads in React.
You can even do your intervals easily:
import ChunkedUploady, { useChunkStartListener } from "#rpldy/chunked-uploady";
import UploadButton from "#rpldy/upload-button";
const CHUNK_SIZE = 1e+6;
const UploadButtonDelayedChunks = () => {
useChunkStartListener(() => new Promise((resolve) => {
//delays chunk upload by 5 seconds
setTimeout(resolve, 5000);
}));
return <UploadButton/>;
};
export const ChunkedExample = () => {
return (
<ChunkedUploady
destination={{ url: "https://my-server/upload" }}
chunkSize={CHUNK_SIZE}>
<UploadButtonDelayedChunks/>
</ChunkedUploady>
);
};
Related
This question already has answers here:
JavaScript closure inside loops – simple practical example
(44 answers)
Closed 4 years ago.
The community reviewed whether to reopen this question 3 months ago and left it closed:
Duplicate This question has been answered, is not unique, and doesn’t differentiate itself from another question.
I am running an event loop of the following form:
var i;
var j = 10;
for (i = 0; i < j; i++) {
asynchronousProcess(callbackFunction() {
alert(i);
});
}
I am trying to display a series of alerts showing the numbers 0 through 10. The problem is that by the time the callback function is triggered, the loop has already gone through a few iterations and it displays a higher value of i. Any recommendations on how to fix this?
The for loop runs immediately to completion while all your asynchronous operations are started. When they complete some time in the future and call their callbacks, the value of your loop index variable i will be at its last value for all the callbacks.
This is because the for loop does not wait for an asynchronous operation to complete before continuing on to the next iteration of the loop and because the async callbacks are called some time in the future. Thus, the loop completes its iterations and THEN the callbacks get called when those async operations finish. As such, the loop index is "done" and sitting at its final value for all the callbacks.
To work around this, you have to uniquely save the loop index separately for each callback. In Javascript, the way to do that is to capture it in a function closure. That can either be done be creating an inline function closure specifically for this purpose (first example shown below) or you can create an external function that you pass the index to and let it maintain the index uniquely for you (second example shown below).
As of 2016, if you have a fully up-to-spec ES6 implementation of Javascript, you can also use let to define the for loop variable and it will be uniquely defined for each iteration of the for loop (third implementation below). But, note this is a late implementation feature in ES6 implementations so you have to make sure your execution environment supports that option.
Use .forEach() to iterate since it creates its own function closure
someArray.forEach(function(item, i) {
asynchronousProcess(function(item) {
console.log(i);
});
});
Create Your Own Function Closure Using an IIFE
var j = 10;
for (var i = 0; i < j; i++) {
(function(cntr) {
// here the value of i was passed into as the argument cntr
// and will be captured in this function closure so each
// iteration of the loop can have it's own value
asynchronousProcess(function() {
console.log(cntr);
});
})(i);
}
Create or Modify External Function and Pass it the Variable
If you can modify the asynchronousProcess() function, then you could just pass the value in there and have the asynchronousProcess() function the cntr back to the callback like this:
var j = 10;
for (var i = 0; i < j; i++) {
asynchronousProcess(i, function(cntr) {
console.log(cntr);
});
}
Use ES6 let
If you have a Javascript execution environment that fully supports ES6, you can use let in your for loop like this:
const j = 10;
for (let i = 0; i < j; i++) {
asynchronousProcess(function() {
console.log(i);
});
}
let declared in a for loop declaration like this will create a unique value of i for each invocation of the loop (which is what you want).
Serializing with promises and async/await
If your async function returns a promise, and you want to serialize your async operations to run one after another instead of in parallel and you're running in a modern environment that supports async and await, then you have more options.
async function someFunction() {
const j = 10;
for (let i = 0; i < j; i++) {
// wait for the promise to resolve before advancing the for loop
await asynchronousProcess();
console.log(i);
}
}
This will make sure that only one call to asynchronousProcess() is in flight at a time and the for loop won't even advance until each one is done. This is different than the previous schemes that all ran your asynchronous operations in parallel so it depends entirely upon which design you want. Note: await works with a promise so your function has to return a promise that is resolved/rejected when the asynchronous operation is complete. Also, note that in order to use await, the containing function must be declared async.
Run asynchronous operations in parallel and use Promise.all() to collect results in order
function someFunction() {
let promises = [];
for (let i = 0; i < 10; i++) {
promises.push(asynchonousProcessThatReturnsPromise());
}
return Promise.all(promises);
}
someFunction().then(results => {
// array of results in order here
console.log(results);
}).catch(err => {
console.log(err);
});
async await is here
(ES7), so you can do this kind of things very easily now.
var i;
var j = 10;
for (i = 0; i < j; i++) {
await asycronouseProcess();
alert(i);
}
Remember, this works only if asycronouseProcess is returning a Promise
If asycronouseProcess is not in your control then you can make it return a Promise by yourself like this
function asyncProcess() {
return new Promise((resolve, reject) => {
asycronouseProcess(()=>{
resolve();
})
})
}
Then replace this line await asycronouseProcess(); by await asyncProcess();
Understanding Promises before even looking into async await is must
(Also read about support for async await)
Any recommendation on how to fix this?
Several. You can use bind:
for (i = 0; i < j; i++) {
asycronouseProcess(function (i) {
alert(i);
}.bind(null, i));
}
Or, if your browser supports let (it will be in the next ECMAScript version, however Firefox already supports it since a while) you could have:
for (i = 0; i < j; i++) {
let k = i;
asycronouseProcess(function() {
alert(k);
});
}
Or, you could do the job of bind manually (in case the browser doesn't support it, but I would say you can implement a shim in that case, it should be in the link above):
for (i = 0; i < j; i++) {
asycronouseProcess(function(i) {
return function () {
alert(i)
}
}(i));
}
I usually prefer let when I can use it (e.g. for Firefox add-on); otherwise bind or a custom currying function (that doesn't need a context object).
var i = 0;
var length = 10;
function for1() {
console.log(i);
for2();
}
function for2() {
if (i == length) {
return false;
}
setTimeout(function() {
i++;
for1();
}, 500);
}
for1();
Here is a sample functional approach to what is expected here.
ES2017: You can wrap the async code inside a function(say XHRPost) returning a promise( Async code inside the promise).
Then call the function(XHRPost) inside the for loop but with the magical Await keyword. :)
let http = new XMLHttpRequest();
let url = 'http://sumersin/forum.social.json';
function XHRpost(i) {
return new Promise(function(resolve) {
let params = 'id=nobot&%3Aoperation=social%3AcreateForumPost&subject=Demo' + i + '&message=Here%20is%20the%20Demo&_charset_=UTF-8';
http.open('POST', url, true);
http.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
http.onreadystatechange = function() {
console.log("Done " + i + "<<<<>>>>>" + http.readyState);
if(http.readyState == 4){
console.log('SUCCESS :',i);
resolve();
}
}
http.send(params);
});
}
(async () => {
for (let i = 1; i < 5; i++) {
await XHRpost(i);
}
})();
JavaScript code runs on a single thread, so you cannot principally block to wait for the first loop iteration to complete before beginning the next without seriously impacting page usability.
The solution depends on what you really need. If the example is close to exactly what you need, #Simon's suggestion to pass i to your async process is a good one.
i'm new to dart and i'm trying to run multiple algorithms using isolates to more efficiently run them multiple times, however, when calling more than one function with Isolate.spawn i start getting 'Malformed message' in the console, it still works, but i would like to know why am i getting this message, also any help in improving the code is welcome since i'm still learning the intricacies of using isolates
stressTest()
{
ReceivePort rpDouble = new ReceivePort();
ReceivePort rpString = new ReceivePort();
ReceivePort rpInteger = new ReceivePort();
int counter = 0;
int tempoTotal = 0;
rpDouble.listen((data) {
counter++;
tempoTotal += data; //data is a stopwatch.toMilliseconds
setState(() { //updating the "progress" and the time it took to run the algorithm in the ui
test = counter.toString() + '%';
_counter = tempoTotal.toString();
});
});
rpInteger.listen((data){
counter++;
tempoTotal += data;
setState(() {
test = counter.toString() + '%';
_counter = tempoTotal.toString();
});
});
rpString.listen((data){
counter++;
tempoTotal += data;
setState(() {
test = counter.toString() + '%';
_counter = tempoTotal.toString();
});
});
for(int i = 0; i < 5; i++) {
Isolate.spawn(DoubleTest, rpDouble.sendPort);
Isolate.spawn(StringStress, rpString.sendPort);
Isolate.spawn(integerTest, rpInteger.sendPort);
}
}
}
the DoubleTest, StringStress and IntegerTest are functions that return a stopwatch.toMilliseconds integer to the sendPort.
Thanks in advance, any help is appreciated
This happens when you want to see the Performance App. However, during the Reload app, resyncing the data results in flutter performance not getting an address or interrupting the progress reader leading to that message. This does not affect the application or your application error. If you need to read "Memory usage" or "frame rendering times" then just restart the IDE and start again.
As your code is running fine, so you don't need to worry much about the error. However,try reformatting your code to avoid this error message.
I had a similar problem in my program. Just saving the file before 'hot restart' did the job.
I am new to discord.js but learned that I can delete my messages using bulkDelete and it will delete them all, even if they are older than 2 weeks. I clear my messages in a server I moderate manually once a month and needless to say it takes forever. I was wondering if anyone would be able to help me make a command that will do this automatically whenever I call it?
Thanks,
K
I would set up a recursive function that checks if there are messages in the channel (100 max every time): if there are no messages it stops otherwise it deletes them and restarts.
function clean(channel, limit = 100) {
return channel.fetchMessages({limit}).then(async collected => {
let mine = collected.filter(m => m.author.id == 'your_id_here'); // this gets only your messages
if (mine.size > 0) {
await channel.bulkDelete(mine, true);
clean(channel);
} else channel.send("The channel is now empty!").delete(5000); // this message is deleted after 5 s
});
}
You can adapt this idea to your existing command parser or, if you don't know how to implement this, try:
client.on('message', msg => {
if (msg.author.bot || msg.author != YOU) return;
// with YOU i mean your User object, to check permissions
let command = 'clean', // the name of your command
args = msg.content.split(' ');
if (args[0].toLowerCase() == command)
clean(msg.channel, !isNaN(args[1]) ? args[1] : undefined); //<-- THIS is how to use the function
// used a ternary operator to check if the other arg is a number
});
This is just a very basic implementation, there are a lot of better ways to detect commands.
I've just found a way to filter messages.
You can fetch messages and then check for each message if its yours
await message.channel.fetchMessages({
limit: 100
}).then((msgCollection) => {
msgCollection.forEach((msg) => {
if(msg.author.id == message.author.id) {
msg.delete();
}
})
});
Background
Developing a Chrome extension (latest Chrome running on Mac OS Sierra) and I can't work out how to loop over an array which is also dynamically built at runtime.
Forgive me if I am missing something really obvious, but I cannot for the life of me work out why this for..of loop is not being entered.
I've also tried a for..in and the good old classic for loop structure i.e. for (let i = 0; i < array.length; i++) - no matter what style of loop this block is never entered, despite my array at runtime reportedly having a single item in it.
Problem Code and Statement
This code gets all files inside a directory and slices off the last 3 chars (to remove .js):
const getDirectoryContents = (path) => {
let fileNames = []
chrome.runtime.getPackageDirectoryEntry( (directoryEntry) => {
directoryEntry.getDirectory(path, {}, (subDirectoryEntry) => {
const subDirectoryReader = subDirectoryEntry.createReader()
subDirectoryReader.readEntries( (entries) => {
for (const entry of entries) {
fileNames.push(entry.name.slice(0, -3))
}
})
})
})
return fileNames
}
From inside the chrome.runtime.onStartup() callback function we want to add some context menus, which we do like so:
const addContextMenus = () => {
console.log(getDirectoryContents('commands'))
for (const command of getDirectoryContents('commands')) {
const properties = {
id: command,
title: command,
contexts: ['editable']
}
chrome.contextMenus.create(properties)
console.log(`Created context menu ${properties.title}`)
}
console.log('End addContextMenus')
}
Now, during runtime, the above code will output this inside the background page console:
However as we can see (due to the lack of the console logging "Created context menu ..." - the loop is never entered, despite the array having a length of 1.
I've found nothing online inside the Chrome developer docs that indicated that getDirectoryContents is asynchronous -- which would be one possible explanation -- but just to be sure I even tried adding a callback param to the getDirectoryContents function to ensure we are looping after the array has been populated.
EDIT: after closer inspection of the original function, it's clear that the array is in fact being returned before is has a chance to be populated by the directory reader. Answer below.
Same result!
Any help would be much appreciated. Thanks for your time.
How embarrassing! Passing in a callback function and executing it at the right time solved it. Comments were all correct - typical async issue - thanks for the support.
The problem was on line 15 of the original function: I was returning the array before it had a chance to be populated.
Working function:
const getDirectoryContents = (path, callback) => {
chrome.runtime.getPackageDirectoryEntry( (directoryEntry) => {
directoryEntry.getDirectory(path, {}, (subDirectoryEntry) => {
const subDirectoryReader = subDirectoryEntry.createReader()
let fileNames = []
subDirectoryReader.readEntries( (entries) => {
for (const entry of entries) {
fileNames.push(entry.name.slice(0, -3))
}
callback(fileNames)
})
})
})
}
Loop index (i) is not what I'm expecting when I use Protractor within a loop.
Symptoms:
Failed: Index out of bound. Trying to access element at index:'x', but there are only 'x' elements
or
Index is static and always equal to the last value
My code
for (var i = 0; i < MAX; ++i) {
getPromise().then(function() {
someArray[i] // 'i' always takes the value of 'MAX'
})
}
For example:
var expected = ['expect1', 'expect2', 'expect3'];
var els = element.all(by.css('selector'));
for (var i = 0; i < expected.length; ++i) {
els.get(i).getText().then(function(text) {
expect(text).toEqual(expected[i]); // Error: `i` is always 3.
})
}
or
var els = element.all(by.css('selector'));
for (var i = 0; i < 3; ++i) {
els.get(i).getText().then(function(text) {
if (text === 'should click') {
els.get(i).click(); // fails with "Failed: Index out of bound. Trying to access element at index:3, but there are only 3 elements"
}
})
}
or
var els = element.all(by.css('selector'));
els.then(function(rawelements) {
for (var i = 0; i < rawelements.length; ++i) {
rawelements[i].getText().then(function(text) {
if (text === 'should click') {
rawelements[i].click(); // fails with "Failed: Index out of bound. Trying to access element at index:'rawelements.length', but there are only 'rawelements.length' elements"
}
})
}
})
The reason this is happening is because protractor uses promises.
Read https://github.com/angular/protractor/blob/master/docs/control-flow.md
Promises (i.e. element(by...), element.all(by...)) execute their then functions when the underlying value becomes ready. What this means is that all the promises are first scheduled and then the then functions are run as the results become ready.
When you run something like this:
for (var i = 0; i < 3; ++i) {
console.log('1) i is: ', i);
getPromise().then(function() {
console.log('2) i is: ', i);
someArray[i] // 'i' always takes the value of 3
})
}
console.log('* finished looping. i is: ', i);
What happens is that getPromise().then(function() {...}) returns immediately, before the promise is ready and without executing the function inside the then. So first the loop runs through 3 times, scheduling all the getPromise() calls. Then, as the promises resolve, the corresponding thens are run.
The console would look something like this:
1) i is: 0 // schedules first `getPromise()`
1) i is: 1 // schedules second `getPromise()`
1) i is: 2 // schedules third `getPromise()`
* finished looping. i is: 3
2) i is: 3 // first `then` function runs, but i is already 3 now.
2) i is: 3 // second `then` function runs, but i is already 3 now.
2) i is: 3 // third `then` function runs, but i is already 3 now.
So, how do you run protractor in loops?
The general solution is closure. See JavaScript closure inside loops – simple practical example
for (var i = 0; i < 3; ++i) {
console.log('1) i is: ', i);
var func = (function() {
var j = i;
return function() {
console.log('2) j is: ', j);
someArray[j] // 'j' takes the values of 0..2
}
})();
getPromise().then(func);
}
console.log('* finished looping. i is: ', i);
But this is not that nice to read. Fortunately, you can also use protractor functions filter(fn), get(i), first(), last(), and the fact that expect is patched to take promises, to deal with this.
Going back to the examples provided earlier. The first example can be rewritten as:
var expected = ['expect1', 'expect2', 'expect3'];
var els = element.all(by.css('selector'));
for (var i = 0; i < expected.length; ++i) {
expect(els.get(i).getText()).toEqual(expected[i]); // note, the i is no longer in a `then` function and take the correct values.
}
The second and third example can be rewritten as:
var els = element.all(by.css('selector'));
els.filter(function(elem) {
return elem.getText().then(function(text) {
return text === 'should click';
});
}).click();
// note here we first used a 'filter' to select the appropriate elements, and used the fact that actions like `click` can act on an array to click all matching elements. The result is that we can stop using a for loop altogether.
In other words, protractor has many ways to iterate or access element i so that you don't need to use for loops and i. But if you must use for loops and i, you can use the closure solution.
Hank did a great job on answering this.
I wanted to also note another quick and dirty way to handle this. Just move the promise stuff to some external function and pass it the index.
For example if you want to log all the list items on the page at their respective index (from ElementArrayFinder) you could do something like this:
var log_at_index = function (matcher, index) {
return $$(matcher).get(index).getText().then(function (item_txt) {
return console.log('item[' + index + '] = ' + item_txt);
});
};
var css_match = 'li';
it('should log all items found with their index and displayed text', function () {
$$(css_match).count().then(function (total) {
for(var i = 0; i < total; i++)
log_at_index(css_match, i); // move promises to external function
});
});
This comes in handy when you need to do some fast debugging & easy to tweak for your own use.
I am NOT arguing with the logic or wisdom of the far more learned people discussing above. I write to point out that in the current version of Protractor within a function declared as async, a for loop like the below (which I was writing in typeScript, incorporating flowLog from #hetznercloud/protractor-test-helper, though I believe console.log would also work here) acts like what one might naively expect.
let inputFields = await element.all(by.tagName('input'));
let i: number;
flowLog('count = '+ inputFields.length);
for (i=0; i < inputFields.length; i++){
flowLog(i+' '+await inputFields[i].getAttribute('id')+' '+await inputFields[i].getAttribute('value'));
}
producing output like
count = 44
0 7f7ac149-749f-47fd-a871-e989a5bd378e 1
1 7f7ac149-749f-47fd-a871-e989a5bd3781 2
2 7f7ac149-749f-47fd-a871-e989a5bd3782 3
3 7f7ac149-749f-47fd-a871-e989a5bd3783 4
4 7f7ac149-749f-47fd-a871-e989a5bd3784 5
5 7f7ac149-749f-47fd-a871-e989a5bd3785 6
...
42 7f7ac149-749f-47fd-a871-e989a5bd376a 1
43 7f7ac149-749f-47fd-a871-e989a5bd376b 2
As I understand it, the await is key here, forcing the array to be resolved up front (so count is right) and the awaits within the loop cause each promise to be resolved before i is allowed to be incremented.
My intent here is to give readers options, not to question the above.
The easier way for doing this these days
it('test case', async () => {
let elems = element.all(selector)
for (let i=0; i < await elems.count(); i++) {
console.log(await elems.get(i).getText())
}
});