so I have this working fetch I wanted to know if I could improve it more so it will be faster, so the api only allows a limit of 100 items to return per call, and has a skipQ to skip a number so we could skip the previous responses and not get duplication - so what I did was do a while loop until the response from one of the fields (dataCount) returns 0. Thoughts on improving and other alternatives?
const fetchUsers = async () => {
let d = 1
let skip = 0
while (d !== 0) {
const response = await axios.post(`${URI}, {limit: '100', skipQ: skip}`)
skip += 100
if(response.data.dataCount === 0) {
d = response.data.dataCount
}
setUsers(prev => [...prev, response.data].flat())
} else {
navigate('/')
}
}
fetchUsers()
this function is inside a useEffect hhok
Related
I want to store 100000 record in my application so i can use it in offline mod first I try to use Asyncstorage
try{
await AsyncStorage.setItem(#Devices, JSON.stringify(data));
}
but I have problem when I try to get the saved data
let Devices = await AsyncStorage.getItem('#Devices');
let newList = await JSON.parse( Devices );
AsyncStorage throw error :
AsyncStorage Couldn't read row 0, col 0 from CursorWindow
then I try to use react-native-sqlite-storage I face problem where
I have a problem that the data must be added one by one (for loop to insert 100000 record) and this led to the application being stuck
// Didmount
useEffect(() => {
createTable();
});
const createTable = () => {
db.transaction(tx => {
tx.executeSql(
'CREATE TABLE IF NOT EXISTS' +
'USERS' +
'(ID INTEGER PRIMARY KEY AUTOINCREMENT,item_code TEXT,item_desc TEXT,ser_id TEXT,inv_no TEXT,status TEXT,location TEXT,)',
);
});
};
const insertIntoTable = () => {
deviceList.map(item=>{
db.transaction(tx => {
tx.executeSql(
'INSERT INTO USERS (item_code,item_desc,ser_id,inv_no,status,location) VALUES (?,?,?,?,?,?,)',
[item.item_code, item.item_desc, item.ser_id, iitem.nv_no, item.status, item.location],
);
});
})
};
you have to call transaction at first ,and define loop in , it's something like this :
try {
db.transaction(
tx => {
deviceList.forEach(async item => {
await tx.executeSql(
'INSERT INTO USERS (item_code,item_desc,ser_id,inv_no,status,location) VALUES (?,?,?,?,?,?,)',
[item.item_code, item.item_desc, item.ser_id, iitem.nv_no, item.status, item.location],
);
});
},
null,
null,
);
} catch (error) {
result = false;
}
maybe it's Helpful :)
I solved my problem by dividing the data into 10,000 for each one, so the 100,000 divided into 10 AsyncStorage
let loop = (await res.items.length) / 10000;
let loop2 = (await loop) % 1 === 0 ? loop : parseInt(loop) + 1;
let first = -10000;
let second = 0;
let last = res.items.length;
await AsyncStorage.setItem('loop', loop2.toString());
for (var i = 0; i < loop2; i++) {
first = first + 10000;
second = i + 1 == loop2 ? last : second + 10000;
let data = await res.items.slice(first, second);
try {
let Device = await ('#Device' + i + 1);
await AsyncStorage.setItem(Device, JSON.stringify(data));
if (i + 1 == loop2) {
setIsLoading(false);
}
I am trying to implement a filter which allows me to filter users based on the distance between a location and their address. Data is provided to the table using useMemo, basically like this:
const data = useMemo(
() =>
contacts.filter(contact => {
var shouldReturn = true;
clientFilter.map((filter, i) => {
if (filter.condition === 'max_10km') {
const originAddress = `${contact['street']} ${contact['number']}, ${contact['zip']} ${contact['city']}, ${contact['country']}`;
calculateDistance(originAddress, filter.value, function(distance) {
console.log('distance is calculated: ', distance);
if (distance > 10000) {
console.log('distance is > for', contact['name']);
shouldReturn = false;
}
});
}
});
return shouldReturn;
}),
[clientFilter]
);
This works fine, in console the results return as I expect them to be. However, my table doesn't update. I suspect it is because the result of the API calls are async, and thus the table is re-rendered before the results are in.
I have tried updating the data using useEffect, but this brings me in a loop which constant re-renders, and thus exceeding the maximum (Maximum update depth exceeded.).
How should I go about this? Should I try async functions? If so, how can I wait to update my data until all promises are resolved?
EDIT 14 NOV
So, I have been looking further into this today. I have managed to switch the filtering to useEffect() instead of useMemo(), so currently, it looks like this:
const [filteredContacts, setFilteredContacts] = useState(contacts);
const data = useMemo(() => filteredContacts, [filteredContacts]);
const [isLoading, setIsLoading] = useState(false);
useEffect(() => {
console.log('Log: In useEffect()');
if (!isLoading) {
setIsLoading(true);
(async () => {
console.log('Log: State has changed, filtering will start');
filterContacts().then(() => setIsLoading(false));
})();
}
}, [clientFilter]);
async function filterContacts() {
setFilteredContacts(
contacts.filter(contact => {
var shouldReturn = true;
clientFilter.map((filter, i) => {
if (
filter.condition === 'equal' &&
contact[filter.field] != filter.value &&
shouldReturn
) {
shouldReturn = false;
}
if (filter.condition === 'max_10km' && shouldReturn) {
const originAddress = `${contact['street']} ${contact['number']}, ${contact['zip']} ${contact['city']}, ${contact['country']}`;
calculateDistance(originAddress, filter.value, async function(
distance
) {
console.log('Log: Distance is calculated: ', distance);
if (distance > 10000) {
console.log(
'Log: Distance is further away for',
contact['title']
);
shouldReturn = false;
}
});
}
});
console.log('Log: About to return shouldReturn value');
return shouldReturn;
})
);
}
Now, this works for my other filters, but the async distance calculation still runs after the return of shouldReturn has been done. So my logs look like this (I have 16 contacts/users currently):
Log: In useEffect()
Log: State has changed, filtering will start
(16) Log: About to return shouldReturn value
Log: Distance is calculated: 1324
Log: Distance is calculated: 4326
...
So basically, it still ignores the async state of my function calculateDistance. Any ideas?
EDIT 15/11
Might be useful as well, this is my calculateDistance() function:
function calculateDistance(origin, destination, callback) {
const google = window.google;
const directionsService = new google.maps.DirectionsService();
directionsService.route(
{
origin: origin,
destination: destination,
travelMode: google.maps.TravelMode.DRIVING
},
(result, status) => {
if (status === google.maps.DirectionsStatus.OK) {
callback(result.routes[0].legs[0].distance.value);
} else {
console.error('Google Maps API error: ', status);
callback(null);
}
}
);
}
EDIT 18/11
After #Secret's suggestion, I changed the code to:
const shouldRemoveContact = await(filters, contact) = () => {
for (const filter of filters) {
if (
filter.condition === 'equal' &&
contact[filter.field] != filter.value
) {
return true
}
if (filter.condition === 'max_10km') {
const originAddress = `${contact['street']} ${contact['number']}, ${contact['zip']} ${contact['city']}, ${contact['country']}`
// please update calculateDistance to return a promise
const distance = await calculateDistance(originAddress, filter.value)
return distance > 10000
}
return false
}
}
async function filterContacts (filters, contact) {
// for every contact, run them to shouldRemoveContact
// since shouldRemoveContact is async, we use Promise.all
// to wait for the array of removeables to be ready
const shouldRemove = await Promise.all(
contact.map(c => shouldRemoveContact(filters, c))
)
// use shouldRemove to check if contact should be removed
// and voila!
return contacts.filter((c, i) => !shouldRemove[i])
}
This results in:
Syntax error: Unexpected reserved word 'await'.
on the line:
const shouldRemoveContact = await(filters, contact) = () => {
You are correct in your mistake - your calculateDistance is async and therefore you can't filter your data properly. Essentially, your code can be boiled down to:
setFilteredContacts(
contacts.filter(contact => {
var shouldReturn = true;
// THIS IS WHERE CALCULATE IS
// but it doesn't do anything because it is async
// so effectively, it does nothing like a comment
console.log('Log: About to return shouldReturn value');
return shouldReturn;
})
);
As you can see, your contacts will never be filtered out as shouldReturn will always return true - it never changes in the middle of that function!
What you can do is calculate the list of filterables beforehand, and then use that to run your filter. Something like this (psuedocode):
// given a contact and a list of filters
// asynchronously return if it should or should not be filtered
const shouldRemoveContact = async () => {}
// THEN, in the filterContacts part:
// let's generate an array of calls to shouldRemoveContact
// i.e. [true, true, false, true], where `true` means it should be removed:
// note that we use await Promise.all here, waiting for all the data to be finished.
const shouldRemove = await Promise.all(
contact.map(c => shouldRemoveContact(filters, c))
)
// we then simply shouldRemove to filter it all out
return contacts.filter((c, i) => !shouldRemove[i])
All together:
// given a list of filters and one contact
// asynchronously returns true or false depending on wheter or not
// the contact should be removed
const shouldRemoveContact = async (filters, contact) => {
for (const filter of filters) {
if (
filter.condition === 'equal' &&
contact[filter.field] != filter.value
) {
return true
}
if (filter.condition === 'max_10km') {
const originAddress = `${contact['street']} ${contact['number']}, ${contact['zip']} ${contact['city']}, ${contact['country']}`
// please update calculateDistance to return a promise
const distance = await calculateDistance(originAddress, filter.value)
return distance > 10000
, async function(
distance
) {
console.log('Log: Distance is calculated: ', distance);
if (distance > 10000) {
console.log(
'Log: Distance is further away for',
contact['title']
);
shouldReturn = false;
}
});
}
}
return false
}
async function filterContacts (filters, contact) {
// for every contact, run them to shouldRemoveContact
// since shouldRemoveContact is async, we use Promise.all
// to wait for the array of removeables to be ready
const shouldRemove = await Promise.all(
contact.map(c => shouldRemoveContact(filters, c))
)
// use shouldRemove to check if contact should be removed
// and voila!
return contacts.filter((c, i) => !shouldRemove[i])
}
I am using React and I do not understand why in the useEffect when running a map function the second part of the code runs before the first part (which is a promise resolve).
Shouldn't the parser wait for the promise to resolve and then run the second part of the code?
useEffect(() => {
const pools = mainnet.Exchanges.Pancakeswap.LpTokens.map((lpPool) => {
// part 1
const [tokenZeroSymbol, tokenOneSymbol] = lpPool.name.replace(' LP', '').split('-');
const prices = fetchTokenPrice(tokenZeroSymbol.toLowerCase(), tokenOneSymbol.toLowerCase());
Promise.resolve(prices).then((values) => {
const [priceTokenZero, priceTokenOne] = values;
filteredFarmPools.find((pool) => {
if (lpPool.name.replace(' LP', '') === pool.name) {
pool.priceTokenZero = values[0].usd;
pool.priceTokenOne = values[1].usd;
}
console.log('inside the fethcprice promise');
});
});
// part 2
filteredFarmPools.find((pool) => {
if (lpPool.name.replace(' LP', '') === pool.name) {
const tvl0 = (pool.reserveTokenZero / 10 ** 18) * pool.priceTokenZero;
const tvl1 = (pool.reserveTokenOne / 10 ** 18) * pool.priceTokenOne;
pool.tvl = tvl0 + tvl1;
}
console.log('inside the tvl calc');
});
});
No.
Promises give you an object that you can pass around and call then on.
They do not turn asynchronous code into blocking code.
The second part of the code isn't inside the then callback so it runs while the asynchronous code (that will trigger the first promise to resolve) is running in the background.
That said, see the await keyword for asyntax that can give the illusion that a promise is blocking.
useEffect(() => {
const processPools = async () => {
for (let lpPool of mainnet.Exchanges.Pancakeswap.LpTokens) {
const [tokenZeroSymbol, tokenOneSymbol] = lpPool.name.replace(' LP', '').split('-');
const values = await fetchTokenPrice(tokenZeroSymbol.toLowerCase(), tokenOneSymbol.toLowerCase());
// Promise.resolve(prices).then((values) => {
const [priceTokenZero, priceTokenOne] = values;
filteredFarmPools.find((pool) => {
if (lpPool.name.replace(' LP', '') === pool.name) {
pool.priceTokenZero = values[0].usd;
pool.priceTokenOne = values[1].usd;
}
console.log('inside the fethcprice promise');
// });
});
}
}
processPools();
});
Original Array.map does not support async
Promise.resolve return immediately, no difference with Promise.then
Despite looking and following numerous answers here at stackoverflow,I have still failed to refactor this code to abide by the ESLint no-loop-func.
I keep getting the following warning, despite my efforts to refactor the code:
Compiled with warnings.
Function declared in a loop contains unsafe references to variable(s) 'lastResult', 'biologyBooks', 'page' no-loop-func
Here's the code:
import React from 'react';
import { apiFullCall } from '../../apiHelper';
const MyComponent = props => {
const [state, setState] = React.useState({ total: 0, biologyBooksByAuthor: [] });
let isLoaded = React.useRef(true);
const token = sessionStorage.getItem('token');
const authorID = sessionStorage.getItem('author_id');
const getBooks = async() => { // fetch items
let page = 1;
let scienceBooks, biologyBooks;
// create empty arrays to store book objects for each loop
let scienceBooks = biologyBooks = [];
// create a lastResult object to help check if there is a next page
let lastResult = { next: null };
do { // the looping - this is what I have failed to refactor
try {
await apiFullCall( // Make API calls over paginated records
'',
token,
'get',
`books/?author_id=1&page=${page}`
).then(res => {
if (res) {
const { status, body } = res;
if (status === 200 || status === 201) {
lastResult = body; // assign lastResult to pick "next"
body &&
body.results &&
body.results.map(eachBook => { // we map() over the returned "results" array
// the author with queried "author_id" writes science books;
// so we add each book (an object) into the science category
scienceBooks.push(eachBook);
// We then filter the author's biology books (from other science books)
biologyBooks = scienceBooks.filter(
({ is_biology }) =>
typeof(is_biology) === "boolean" && is_biology === true
);
return null;
}
);
// increment the page with 1 on each loop
page++;
}
}
}).catch(error => console.error('Error while fetching data:', error));
} catch (err) { console.error(`Oops, something went wrong ${err}`); }
// keep running until there's no next page
} while (lastResult.next !== null);
// update the state
setState(prevState => ({
...prevState, total: scienceBooks.length, biologyBooksByAuthor: biologyBooks,
}));
};
React.useEffect(() => { // fetch science books by author (logged in)
if (isLoaded && authorID) {
getBooks();
};
return function cleanup() {...}; // clean up API call, on unmount
}, [isLoaded, authorID]);
return (
// render the JSX code
);
}
Please note that I actually declared the said variables lastResult, biologyBooks and page outside the "do-while".
Any help or clues will be greatly appreciated.
The function the warning is referring to is the .then callback, if you're using async/await stick to it, try removing the .then part by assigning the result to a variable instead and remove the unnecessary .map, you can concatenate previous results with spread operator or .concat.
import React from 'react';
import { apiFullCall } from '../../apiHelper';
const MyComponent = props => {
const [state, setState] = React.useState({
total: 0,
scienceBooksByAuthor: [],
});
const isLoaded = React.useRef(true);
const token = sessionStorage.getItem('token');
const authorID = sessionStorage.getItem('author_id');
const getBooks = async () => {
// fetch items
let page = 1;
let scienceBooks = [];
// create a lastResult object to help check if there is a next page
let lastResult = { next: null };
do {
// the looping - this is what I have failed to refactor
try {
const res = await apiFullCall(
// Make API calls over paginated records
'',
token,
'get',
`books/?author_id=1&page=${page}`,
);
if (res) {
const { status, body } = res;
if (status === 200 || status === 201) {
lastResult = body; // assign lastResult to pick "next"
// concatenate new results
scienceBooks = [
...scienceBooks,
...((lastResult && lastResult.results) || []),
];
// increment the page with 1 on each loop
page += 1;
}
}
} catch (err) {
console.error(`Oops, something went wrong ${err}`);
}
// keep running until there's no next page
} while (lastResult.next !== null);
const biologyBooks = scienceBooks.filter(
({ is_biology }) =>
typeof is_biology === 'boolean' && is_biology === true,
);
// update the state
setState(prevState => ({
...prevState,
total: scienceBooks.length,
scienceBooksByAuthor: scienceBooks,
}));
};
React.useEffect(() => {
// fetch science books by author (logged in)
if (isLoaded && authorID) {
getBooks();
}
return function cleanup() {...}; // clean up API call, on unmount
}, [isLoaded, authorID]);
return (
// render the JSX code
);
};
I have an action which will take quite some time to complete. It will be looping through an array with over 250k entries. With each entry it will be doing some logic. I want to inform the user of the actions progress, which would be:
let progress = (i / array.length) * 100
How can I add this to redux store or components state so that I can communicate the current progress in the UI?
I tried dispatching an action (as you can see below) which updates the stores progress with the current iterator but the reducer only executes when the for loop is created.
export const generateKeyMap = (mapFrom, mapTo) => {
return async (dispatch) => {
await dispatch(startGeneration()); <-- This only runs when I use a setTimeout for the following action.
setTimeout(() => {
dispatch(buildKeyMap(mapFrom, mapTo));
}, 100);
};
};
export const buildKeyMap = (mapFrom, mapTo) => {
return async (dispatch) => {
let keyMap = {};
for (let i = 0; i < mapFrom.length; i++) {
await dispatch(reportProgress(i)); // <-- This does not update in store until for loop is done.
let randomIndex = randomIntFromInterval(0, mapTo.length - 1);
let key = mapFrom[i].toString();
let value = mapTo[randomIndex].toString();
Object.assign(keyMap, { [key]: value });
mapTo.splice(randomIndex, 1);
}
await dispatch(stopGeneration());
return { type: actionTypes.DELIVERKEYMAP, payload: keyMap };
}
};