I am trying to accumulate API responses on a server and return them to the client as a single object. To do this I am looping through items in an array and mapping the responses back into the original object. This is working fine for an array of length 1, but logs blanks when looping through larger arrays.
When looping through the array does Node create a new instance of the function or does it keep passing data into the same function even if it hasn't returned a value yet?
loopThroughArray(req, res) {
for(let i=0; i<req.map.length; i++) {
stack[i] = (callback) => {
let data = getApi(req, res, req.map[i], callback)
}
}
async.parallel(stack, (result) => {
res.json(result)
})
}
....
function getApi(req, res, num, cb) {
request({
url: 'https://example.com/api/' + num
},
(error, response, body) => {
if(error) {
// Log error
} else {
let i = {
name: JSON.parse(body)['name'],
age: '100'
}
console.log(body) // Returns empty value array.length > 1 (req.map[i])
cb(i)
}
})
If Node is overloading the function, how can I ensure data has been received before running the function again?
The api calls are async.
When running the loop, the code is making many rest calls without waiting for the answer.
If the loop is not too big then you could synchronize the calls using recursion.
You could also synchronize the calls using nimble:
http://caolan.github.io/nimble/
a loop waits for the method inside it to finish before it loop back. so there's really no such thing as a loop that moves really fast unless you're using "threading" which your obviously not.
Related
Bellow is an example on code I am using to submit data to a server and the return I receive and save
this.apollo.mutate( { mutation: XXXXXXXX, variables: { instance_string: X,
accesstoken: X } })
.subscribe({
next: (data: any ) => {
console.log("data returned from Server", data);
// This data can be sent from the Server in a loop on occassion
this.SaveData(data);
},
error: (err) => {
this.presentToastFail(err);
}
});
On occasion the Server returns the data in a loop. I have no control of the Server and it appears this will be a reoccurring bug for quite a while. Is there a way that I can can insure the next: only runs once and ignores the Loop of data returns the Server can send.
It looks like you have to use two pipeable operators like take(), filter():
.pipe(
filter(resp => resp !== undefined && resp !== null),
take(1) // <--- It will only take one successful valid response
)
.subscribe(...)
It looks like u can also use takeuntil
Takeuntil reference document
const example = evenSource.pipe(
//also give me the current even number count for display
withLatestFrom(evenNumberCount),
map(([val, count]) => `Even number (${count}) : ${val}`),
//when five even numbers have been emitted, complete source observable
takeUntil(1)//this will take 1st response
);
The method call "Insert All sms data" run more than once I tried to prevent using counter variable but on client side it run once and on the server side it runs x time as shown in the image, thus adding data to db multiple timres which I do not want, this should add to db only once. I want also add another call whith in method call "Insert All sms data" which should run x as the loop. I stuck here.
Meteor 1.8 react 16.8
imports/api/kuser.js
'Find all numbers' (smsText) {
if(!this.userId) {
throw new Meteor.Error('not-authorized');
}
let counter = 0;
return userKisandb.find({
userId: this.userId
}).fetch().map((Mob) => {
// return Mob.Mnumber
if(Mob.Mnumber) {
Meteor.call('Send to all',Mob.Mnumber,smsText,(err,resp) => {
//sms.js
if(err){
console.log("send all numbers error2", err);
} else {
console.log("send all numbers ", resp);
if(counter === 0) {
Meteor.call('Insert All sms data',smsText,counter,(err,resp1) => {
//get inserted data id
//allSmsdb
if(err){
console.log("Insert All sms data error", err);
} else {
console.log("this should run only once ",counter);
//Another call to be added which should run x times
}
})
counter++
}
}
});
} //Mob.Mnumber
});
},
and the method 1 is
'Send to all'(mob,text) {
return "sucess";
},
method 2 is
'Insert All sms data' (smstext,counter) {
if(!this.userId) {
throw new Meteor.Error('not-authorized');
}
console.log("Inserted same data x times",counter);
if(counter === 0) {
return allSms.insert({
smstext,
userId: this.userId,
updatedAt: moment().valueOf(),
});
}
},
And output is
Meteor methods can be called from the front end, asking the back end to do something. It doesn't make sense for a back end method to do a Meteor.call, because it's already on the back end. It will work, kind of, but then you get into trouble because your counter isn't passed around as you expect.
Make your method only work on the server, and just loop around the data, doing all of the work, rather than breaking it up into pieces like you have.
I am trying to access an api and I will have to run the api calls several times based on the page numbers I need to iterate, the following is the code which I am using and how can I get the all the response pushed into an array.
as nodeJs is single threaded It is not waiting for the responses from the api.
How can I can tackle this and ensure all the response values are being pushed into an array
Inside the for loop I want the final array which has all the values of the api response. So, I check the total page value and response page Number if that matches which means that will be the last page and I push the array to another function but when I do that it does not have all the values because nodejs does not wait for the api response.
const fs = require('fs');
var pepKey = 'asdfasdfasd';
var pepResponse;
var pepTimecards = [];
pep();
function pep(){
var options = {
headers: {
"content-type": "application/json",
},
agentOptions: {
pfx: fs.readFileSync('./certificate/asdfsdaf.p12'),
passphrase: 'asdasdsda'
}
};
request.get('https://source.asdfasdf.io/api/organisations/asdfasdf/timecard_keys?timecard_type=Flex',options, (err, res, body) => {
if (err) { return console.log(err); }
pepResponse = JSON.parse(body)
pepTimecards = pepResponse.data;
if(pepResponse.pages > 1){
for(let i=2;i<=pepResponse.pages;i++){
var url = 'https://source.`pepme`.io/api/organisations/sdfsadf/timecard_keys?timecard_type=Flex&page='+pageNo;
request.get(url,options, (err, res, body) => {
if (err) { return console.log(err); }
body = JSON.parse(body)
pepTimecards = pepTimecards.concat(body.data)
if(pepResponse.pages == body.page){
console.log(pepResponse.pages)
console.log(body.page +"body page")
console.log(pepTimecards)
}
});
}
}else{
}
});
}
Use the request-promise library which supplies promisified versions of the request library. Then, you can use async/await in your for loop to serialize your operations:
Newer answer to go with the edited code in the OP's question
const fs = require('fs');
const rp = require('request-promise');
const pepKey = 'asdfasdfasd';
pep().then(pepTimecards => {
// the timecard data is valid in here
console.log(pepTimecards);
}).catch(err => {
console.log(err);
});
async function pep() {
let timecards = [];
const options = {
headers: {
"content-type": "application/json",
},
agentOptions: {
pfx: fs.readFileSync('./certificate/asdfsdaf.p12'),
passphrase: 'asdasdsda'
},
json: true,
uri: 'https://source.asdfasdf.io/api/organisations/asdfasdf/timecard_keys?timecard_type=Flex'
};
let pepResponse = await rp(options);
timecards = pepResponse.data;
if (pepResponse.pages > 1) {
for (let i = 2; i <= pepResponse.pages; i++) {
options.uri = 'https://source.`pepme`.io/api/organisations/sdfsadf/timecard_keys?timecard_type=Flex&page='+pageNo;
let body = await rp(url, options);
// add body.data onto the existing array
timecards.push(...body.data);
}
} else {
}
console.log(pepResponse.pages)
console.log(timecards)
return timecards;
}
Prior Answer before OP edited the code in their question:
const rp = require('request-promise');
// I'm assuming this is some sort of method definition on a class, otherwise it needs the function keyword
async pageno(pageNo) {
for (let i=2;i<=pepResponse.pages;i++){
try {
options.uri = 'https://test/timecard_keys?timecard_type=asdas&page='+pageNo;
// let request-promise parse the json for you automatically
options.json = true;
let body = await rp(options);
pepTimecards = pepTimecards.concat(body.data)
if (pepResponse.pages == body.page){
console.log(pepResponse.pages)
console.log(body.page +"body page")
console.log(pepTimecards)
}
} catch(e) {
// decide what to do for error handling
// this will log and rethrow so the caller will get a rejected promise
console.log(e);
throw e;
}
}
// return some value here to be the resolved value of the returned promise
return pepTimecards;
}
In your code, it is not clear where the options, pepTimecards, pepResponse variables are declared. They should probably be declared as local variables here or passed in to the function and/or returned from your function.
Summary of modifications:
Add async to method declaration so we can use await.
Load request-promise library into rp variable
Add options.json = true to the let the request-promise library parse the JSON result for us automatically
Change rp() to just use the options structure (add URL to that)
Add try/catch to catch any errors from the await, log them, then rethrow so pageno() will return a promise that rejects if there is an error (you can customize the behavior when there's an error if desired)
Add a return value so there is meaningful resolved value to the promise (you should not be using side-effect programming as it is now (modifying variables that are not passed in, declared locally or returned).
Things for you still to fix:
Stop using side-effect programming where you modify free variables that aren't passed in, aren't declared locally and aren't returned. This is a bad way to design code. You don't show enough overall context from the calling code or where these other variables are defined to make a concrete recommendation on how it should be done.
Decide what your error handling strategy is if there's an error on one of the requests and implement that strategy and proper handling.
I'm doing an Ionic project and I'm getting a little bit frustrated whit promises and '.then()' although I've read a lot of documentation everywhere.
The case is that I have one provider with the functions loadClients and getWaybills.
The first one gets all the clients that have waybills and the second one gets all the waybills from one concrete client.
loadClients() {
return new Promise(resolve => {
this.http.get('http://localhost/waybills?fields=descr1_sped&idUser='+ this.id)
.map(res => res)
.subscribe(data => {
this.data = data.json();
resolve(this.data);
});
});
}
// GET WAYBILLS
getWaybills(client) {
return new Promise(resolve => {
this.http.get('http://localhost/waybills/?stato=0&idUser='+ this.id +'&descr1_sped='+ client)
.map(res => res)
.subscribe(data => {
this.data = data.json();
resolve(this.data);
});
});
}
On the other hand, on the component welcome.ts I have a function loadWaybills which is called on the view load and is executing the following code, my idea is to get all the clients and then get the respective waybills of each one. Then I'll take just of the ones that are defined.
The problem is that on the second .then() instead of getting the variable data I'm getting just undefined... I've understood that if you put a synchronous code inside .then() can be properly executed and work with the "data" which is the result of the promise. Why am I getting this undefined?
loadWaybills() {
//We first load the clients
this.waybills.loadClients()
.then(data => {
this.waybill = data;
var preClients = this.waybill;
this.clients = [];
//Here we're deleting duplicated clients and getWaybills of them)
for (let i = 0; i < preClients.length; i++) {
if (this.clients.indexOf(preClients[i].descr1_sped) == -1) {
this.waybills.getWaybills(preClients[i].descr1_sped)
.then(data => {
**//Here we'll check if the clients has waybills or not**
this.clientWaybills[i] = data;
this.clients.push(preClients[i].descr1_sped)
});
}
}
});
}
It is hard to say because we don't know what the API is meant to return. For example, there may be a missing field somewhere from the first GET and now for the second one, it returns as undefined sometimes. If it only returns undefined sometimes, a simple solution to this, would be to check that the value is defined before assigning it to the variable.
If it always returns as undefined and shouldn't, try to debug the code and make sure that the values are present before the second .then.
I'm looking to create an array of functions to call dynamically, which will be later used in the Q.all([]) promise call.
For example;
//data is previously generated
var promiseArray = [];
for (var i = 0; i < data.length; i++){
promiseArray.push(functionCall(data[i]))
}
Q.all(promiseArray).then(function(){
//Do something
})
How would I push to the array without calling the function until the Q.all statement? I don't want to call it in the for loop as it will not catch any errors and I can't process the response further.
EDIT:
So to clarify my problem (as I don't think I was as clear as I should have been), here is a solution for a static data length of say 3;
//data is previously generated
var data = [12432432,4324322392,433324323];
//Each function call can happen in parallel or series as its an external POST to an API
//I'm not bothered about speed for this application (as its low throughput) and can wait a few seconds for each
// response
//FunctionCall returns a promise
functionCall(data[0]).then(function(){
//Log success / failure to mongo
});
functionCall(data[1]).then(function(){
//Log success / failure to mongo
});
functionCall(data[2]).then(function(){
//Log success / failure to mongo
});
//OR
functionCall(data[0]).then(function(){
//Log success/failure to mongo
functionCall(data[1]).then(function(){
//Log success/failure to mongo
functionCall(data[2]).then(function(){
//Log success/failure to mongo
});
});
});
But I wont know the length of data until runtime
If I understand correctly, you want to call functionCall for an array of items, and have Q.all resolve once all the promises returned by functionCall have completed regardless if they resolve or reject - if you don't care about the results (as you don't seem to in your code) simply handle the rejection in the promise you push - i.e.
var promiseArray = [];
for (var i = 0; i < data.length; i++) {
promiseArray.push(functionCall(data[i]).then(function(result) {
// log success
return logToMongoFunction(result);
}, function(error) {
// log failure
return logToMongoFunction(error);
}).catch(function(error) {
// catch and ignore any error thrown in either logToMongoFunction above
return;
}));
}
Q.all(promiseArray).then(function () {
//Do something
});
Note: the above can be simplified to
Q.all(data.map(function (item) {
return functionCall(item).then(function(result) {
// log success
return logToMongoFunction(result);
}, function(error) {
// log failure
return logToMongoFunction(error);
}).catch(function(error) {
// catch and ignore any error thrown in either logToMongoFunction above
return;
});
})).then(function() {
//Do something
});
the edited question suggests you can perform the actions in series also - in series it would be
data.reduce(function(promise, item) {
return promise.then(function() {
return functionCall(item).then(function(result) {
// log success
return logToMongoFunction(result);
}, function(error) {
// log failure
return logToMongoFunction(error);
}).catch(function(error) {
// catch and ignore any error thrown in either logToMongoFunction above
return;
});
});
}, Promise.resolve()).then(function() {
// all done
});
instead of Promise.resolve() you could use whatever Q has as an equivalent that creates a resolved promise
logToMongoFunction would log to mongo and needs to return a promise if you need to wait for that to finish before processing the next data item. If you do not need to wait for the mongo logging to complete then there's no need for that function to return a promise
i will recommend using Promise.mapSeries or async library for this because its very easy to catch errors. One more thing looping using a for loop doesnt seems to be good approach if you have database calls in the callback because that might flush the calls to the database and node.js can have memory issues or node.js wont be able to entertain any other request because it will be busy entertaining the request in the for loop. so its always good to run loop serially or limit the numer of parallel executions at a time.
please see example below
This will run Array serially one at a time when 1st one completes execution next will be called
async.eachOfSeries(data, function(dataInstance, key, next) {
functionCall(dataInstance).then(function(){
next();
}).catch(funtion(err){
next(err);
})
}, function() {
//iteration completed
});
OR
async.eachOfSeries(data, function(dataInstance, key, next) {
functionCall(dataInstance, function(err , result){
if(err)
{
console.log(err);
next(err);
}
else
next();
});
}, function() {
//iteration completed
});