Caching in React - reactjs

In my react App I have a input element. The search query should be memoized, which means that if the user has previously searched for 'John' and the API has provided me valid results for that query, then next time when the user types 'Joh', there should be suggestion for the user with the previously memoized values(in this case 'John' would be suggested).
I am new to react and am trying caching for the first time.I read a few articles but couldn't implement the desired functionality.

You don't clarify which API you're using nor which stack; the solution would vary somewhat depending on if you are using XHR requests or something over GraphQL.
For an asynchronous XHR request to some backend API, I would do something like the example below.
Query the API for the search term
_queryUserXHR = (searchTxt) => {
jQuery.ajax({
type: "GET",
url: url,
data: searchTxt,
success: (data) => {
this.setState({previousQueries: this.state.previousQueries.concat([searchTxt])
}
});
}
You would run this function whenever you want to do the check against your API. If the API can find the search string you query, then insert that data into a local state array variable (previousQueries in my example).
You can either return the data to be inserted from the database if there are unknowns to your view (e.g database id). Above I just insert the searchTxt which is what we send in to the function based on what the user typed in the input-field. The choice is yours here.
Get suggestions for previously searched terms
I would start by adding an input field that runs a function on the onKeyPress event:
<input type="text" onKeyPress={this._getSuggestions} />
then the function would be something like:
_getSuggestions = (e) => {
let inputValue = e.target.value;
let {previousQueries} = this.state;
let results = [];
previousQueries.forEach((q) => {
if (q.toString().indexOf(inputValue)>-1) {
result.push(a);
}
}
this.setState({suggestions: results});
}
Then you can output this.state.suggestions somewhere and add behavior there. Perhaps some keyboard navigation or something. There are many different ways to implement how the results are displayed and how you would select one.
Note: I haven't tested the code above

I guess you have somewhere a function that queries the server, such as
const queryServer = function(queryString) {
/* access the server */
}
The trick would be to memorize this core function only, so that your UI thinks its actually accessing the server.
In javascript it is very easy to implement your own memorization decorator, but you could use existing ones. For example, lru-memoize looks popular on npm. You use it this way:
const memoize = require('lru-memoize')
const queryServer_memoized = memoize(100)(queryServer)
This code keeps in memory the last 100 request results. Next, in your code, you call queryServer_memoized instead of queryServer.

You can create a memoization function:
const memo = (callback) => {
// We will save the key-value pairs in the following variable. It will be our cache storage
const cache = new Map();
return (...args) => {
// The key will be used to identify the different arguments combination. Same arguments means same key
const key = JSON.stringify(args);
// If the cache storage has the key we are looking for, return the previously stored value
if (cache.has(key)) return cache.get(key);
// If the key is new, call the function (in this case fetch)
const value = callback(...args);
// And save the new key-value pair to the cache
cache.set(key, value);
return value;
};
};
const memoizedFetch = memo(fetch);
This memo function will act like a key-value cache. If the params (in our case the URL) of the function (fetch) are the same, the function will not be executed. Instead, the previous result will be returned.
So you can just use this memoized version memoizedFetch in your useEffect to make sure network request are not repeated for that particular petition.
For example you can do:
// Place this outside your react element
const memoizedFetchJson = memo((...args) => fetch(...args).then(res => res.json()));
useEffect(() => {
memoizedFetchJson(`https://pokeapi.co/api/v2/pokemon/${pokemon}/`)
.then(response => {
setPokemonData(response);
})
.catch(error => {
console.error(error);
});
}, [pokemon]);
Demo integrated in React

Related

Using ApolloClient pagination API results in requests, even if all page content is already in cache

I am using the ApolloClient core pagination API approach to accumulate paginated requests in a merge function and the repaginate them with a read function: https://www.apollographql.com/docs/react/pagination/core-api
This all works, but now there is a request for each page, even the ones that are already in the cache.
Which defeats the whole purpose when I'm repaginating!
I'm using the default fetchStrategy, cache-first.
If all requested data is present in the cache, that data is returned. Otherwise, Apollo Client executes the query against your GraphQL server and returns that data after caching it.
I wonder how ApolloClient checks that all requested data is in the cache with the pagination implementation.
Because right now (and the docs seems to rely on this) it always does the request, even when the keyArgs match and the data is in the cache.
Does someone know what causes this and how I can customize this cache-first strategy to check if all the items of the requested page are already in the cache?
Here is my code, in case that helps for context or if I'm just doing something wrong:
typePolicies: {
Query: {
fields: {
paginatedProductTracking: {
// Include everything except 'skip' and 'take' to be able to use `fetchMore`
// and repaginate when reading cache
// (essential for switching between desktop pagination and mobile lazy loading
// without having to refetch)
keyArgs: (args) => JSON.stringify(omit(args, ['query.skip', 'query.take'])),
merge: (existing, incoming, { args }) => {
if (!existing) {
return incoming;
}
if (!incoming) {
return existing;
}
const data = existing.paginatedData;
const newData = incoming.paginatedData;
return {
...existing,
// conservative merge that is robust against pages being requested out of order
paginatedData: [
...data.slice(0, args?.query.skip || 0),
...newData,
...data.slice((args?.query.skip || 0) + newData.length),
],
};
},
},
},
},
},
const [pageSize, setPageSize] = useState(100);
const [page, setPage] = useState(0);
const skip = page * pageSize;
const query = {
filter,
aggregationInterval,
order,
skip,
take: pageSize,
search: search ? values : null,
locations: currentLocations.length > 0 ? currentLocations.map((location) => location.id) : undefined,
};
const { data, loading, fetchMore } = useProductTrackingAggregatedDataQuery({
variables: {
query,
},
});
onPageChange={async (newPage) => {
await fetchMore({
variables: {
query: {
...query,
skip: newPage * pageSize,
},
},
});
setPage(newPage);
}}
I was recently faced with the exact same issue and had everything implemented in the way the official documentation illustrates until I stumbled upon this issue which is still open so I'm guessing this is still how the fetchMore function actually behaves to date. So #benjamn says that:
The fetchMore method sends a separate request that always has a fetch policy of no-cache, which is why it doesn't try to read from the cache first.
This being the case, fetchMore is only useful if you are implementing an endless scroll sort of pagination where you know beforehand that the new data is not in the cache.
In the pagination documentation it also states that:
If you are not using React and useQuery, the ObservableQuery object returned by client.watchQuery has a method called setVariables that you can call to update the original variables.
If you change the variables to your query it will trigger your read function implementation. And if the read function finds the data within existing it can return them or return undefined which will in turn trigger a network request to your graphql server to fetch the missing data, which will trigger your merge function to merge the data in the desired way, which will again trigger the read function which will now be able to slice the data you requested according to your { args } out of your existing and return them, which will finally trigger your watched ObservableQuery to fire and your UI to be updated.
Now, this approach is counter intuitive and goes against the "recommended" way of implementing pagination, but contrary to the recommended way this approach actually works.
I was unable to find anything that would prove my conclusions about fetchMore to be wrong, so if any Apollo client guru happens to stumble upon this please do shed some light into this. Until then the only solution I can offer is working with setVariables instead of fetchMore.
Keep in mind that you will need to implement a read function along with your merge. It will be responsible for slicing your cached data and triggering a network request by returning undefined if it was unable to find a full slice.

Mapping State in React with Server Requests

I'm new to React as we are trying to migrate our app from AngularJS. One thing I'm struggling to wrap my head around is what's the best way to make and cache state mapping requests.
Basically, I would do a search, that returns a list of objects and one of the field is a status code (e.g. 100, 200, 300, etc.), some number. To display the result, I need to map that number to a string and we do that with a http request to the server, something like this:
GET /lookup/:stateId
So my problem now is:
I have a list of results but not many different states, how can I make that async call (useEffect?) to make that lookup only once for different stateId? Right now, I can get it to work, but the request is made on every single mapping. I'm putting the Axio call in a utility function to try and reuse this across multiple pages doing similar things, but is that the "React" way? In AngularJS, we use the "|" filter to map the code to text.
Once I have that mapping id => string, I want to store it in cache so next one that needs to map it no longer make the http request. Right now, I put the "cache" in the application level context and use dispatch to update/add values to the cache. Is that more efficient? It appears if I do a language change, where I keep the language in the same application context state, the cache would be re-initialized, and I'm not sure what other things would reset that. In AngularJS, we used the $rootState to 'cache'.
Thanks for any pointers!
In a lookupUtil.js
const DoLookupEntry = async (entryId) => {
const lookupUrl = `/lookup/${entryId}`;
try {
const response = await Axios.get(looupUrl,);
return response.data;
} catch (expt) {
console.log('error [DoLookupEntry]:',expt);
}
}
In a formatUtils.js
const formatLookupValue = (entryId) => {
const appState = useContext(AppContext);
const appDispatch = useContext(DispatchContext);
const language = appState.language;
if (appState.lookupCache
&& appState.lookupCache[entryId]
&& appState.lookupCache[entryId][language]) {
// return cached value
const entry = appState.lookupCache[entryId][language];
return entry.translatedValue;
}
// DoLookup is async, but we are not, so we want to wait...
DoLookupEntry(entryId)
.then((entry) => { // try to save to cache when value returns
appDispatch({type: States.APP_UPDATE_LOOKUP_CACHE,
value:{language, entry}})
return entry.translatedValue;
});
}
And finally the results.js displaying the result along the line (trying formatLookupValue to map the id):
{searchState.pageResults.map((item) => {
return (
<tr>
<td><Link to={/getItem/item.id}>{item.title}</Link></td>
<td>{item.detail}</td>
<td>{formatLookupValue(item.stateId)}</td>
</tr>
)
})}

RTK Query hooks - preventing polling to auto refetch when arg change

I'm trying to refresh a token in a Query hook, with the polling feature every 9 seconds:
"/App.tsx"
..
...
const [storedToken, setStoredToken] = useState(getStoredToken());
const { data, error, refetch } = useRefreshUserQuery(storedToken, {
pollingInterval: 9000,
// refetchOnMountOrArgChange: false // -> This has no effect
});
...
..
The problem is, it re-fetches instantly when the token is set with setStoredToken(token). The new token is passed as argument to the Query hook storedToken and refetch immediately (like an infinite loop).
That would be pretty neat to be able to do this. Is there any better way to refresh a token with polling?
I believe that issue is nothing to solve on RTK-Q level - it's a pretty common and expected "limitation" of hooks and rendering lifecycle architecture. And I feel that RTK-Q polling just won't fit your requirements here, of course, that you are trying to achieve - it's not actually polling in common sense. At least - it's conditional polling, which needs some more logic)
So I would solve this just by debouncing and useEffect:
const [storedToken, setStoredToken] = useState<string>(getStoredToken());
const [tokenDebounced] = useDebounce(storedToken, 9000);
const { data } = useRefreshUserQuery(tokenDebounced);
useEffect(() => {
if (data) {
setStoredToken(data);
// console.log(newToken);
}
}, [data]);
The useEffect content and data content may differ, but the overall idea should be clear.
useDebounce is from https://www.npmjs.com/package/use-debounce,
but your own implementations should work the same if you have some defined already.
Another idea, touching you AUTH setup a bit - is just avoid
const [storedToken, setStoredToken] = useState<string>(getStoredToken());
the part at all, and keep useRefreshUserQuery() without params.
Most likely and common is to store the token in localStorage or redux\other store, and define new baseQuery, based on fetchBaseQuery that will set header and\or to include cookies with credentials: "include" with a token from localStorage or redux\other store. Definitely, you will need to store it during the first AUTH then.
I think RTK-Q auth example reveals this case in some way also:
https://redux-toolkit.js.org/rtk-query/usage/examples#authentication
After you'll avoid that useState and query hook param - you'll be able to use polling with no issues:
const { data, error, refetch } = useRefreshUserQuery(undefined ,{
pollingInterval: 9000,
});
"Polling" here means "fetch X seconds after I have data", but of course you have to get the first data itself - and that is that first fetch. If you prevent that, polling will also never start.
Tbh., this is kind of a weird requirement and doing it like this will fill your cache with dozens of state entries.
I'd do something a little differently - solve it in the endpoint lifecycle.
This is untested pseudocode and you'll need to adjust it a bit:
function waitFor(ms) {
return new Promise(resolve => setTimeout(() => resolve("waited"), ms))
}
currentToken: build.query({
query() {
// whatever you need for the first token here
},
async onCacheEntryAdded(
arg,
{ updateCachedData, cacheDataLoaded, cacheEntryRemoved }
) {
try {
// wait for the initial query to resolve before proceeding
await cacheDataLoaded
while (true) {
const result = await Promise.race(waitFor(9000), cacheEntryRemoved)
if (result !== "waited") {
// cache entry was removed, stop the loop
break
}
// make a fetch call to get a new token here
const newToken = fetch(...)
updateCachedData((oldData) => newToken)
}
},
})
and then just
const result = useCurrentTokenQuery()
in your component

React Apollo Client - modify query data before it goes to cache

Is there a way to modify query response data before it is saved in the internal cache?
I'm using apollo hooks, but this question is relevant to any of front-end approaches using apollo client (HOC & Components as well).
const { data, updateQuery } = useQuery(QUERY, {
onBeforeDataGoesToCache: originalResponseData => {
// modify data before it is cached? Can I have something like this?
return modifiedData;
}
});
Obviously onBeforeDataGoesToCache does not exist, but that's exactly the behavior I'm looking for. There's an updateQuery function in the result, which basically does what is needed, but in the wrong time. I'm looking for something to work as a hook or a middleware inside the query mutation.
It sounds like you want Afterware which, much like Middleware that allows operations before the request is made, allows you to manipulate data in the response e.g.
const modifyDataLink = new ApolloLink((operation, forward) => {
return forward(operation).map(response => {
// Modify response.data...
return response;
});
});
// use with apollo-client
const link = modifyDataLink.concat(httpLink);

How to get a count of number of documents in a collection with Cloud Firestore [duplicate]

This question already has answers here:
Cloud Firestore collection count
(29 answers)
Closed 10 months ago.
In Firestore, how can I get the total number of documents in a collection?
For instance if I have
/people
/123456
/name - 'John'
/456789
/name - 'Jane'
I want to query how many people I have and get 2.
I could do a query on /people and then get the length of the returned results, but that seems a waste, especially because I will be doing this on larger datasets.
You currently have 3 options:
Option 1: Client side
This is basically the approach you mentioned. Select all from collection and count on the client side. This works well enough for small datasets but obviously doesn't work if the dataset is larger.
Option 2: Write-time best-effort
With this approach, you can use Cloud Functions to update a counter for each addition and deletion from the collection.
This works well for any dataset size, as long as additions/deletions only occur at the rate less than or equal to 1 per second. This gives you a single document to read to give you the almost current count immediately.
If need need to exceed 1 per second, you need to implement distributed counters per our documentation.
Option 3: Write-time exact
Rather than using Cloud Functions, in your client you can update the counter at the same time as you add or delete a document. This means the counter will also be current, but you'll need to make sure to include this logic anywhere you add or delete documents.
Like option 2, you'll need to implement distributed counters if you want to exceed per second
Aggregations are the way to go (firebase functions looks like the recommended way to update these aggregations as client side exposes info to the user you may not want exposed) https://firebase.google.com/docs/firestore/solutions/aggregation
Another way (NOT recommended) which is not good for large lists and involves downloading the whole list: res.size like this example:
db.collection("logs")
.get()
.then((res) => console.log(res.size));
If you use AngulareFire2, you can do (assuming private afs: AngularFirestore is injected in your constructor):
this.afs.collection(myCollection).valueChanges().subscribe( values => console.log(values.length));
Here, values is an array of all items in myCollection. You don't need metadata so you can use valueChanges() method directly.
Be careful counting number of documents for large collections with a cloud function. It is a little bit complex with firestore database if you want to have a precalculated counter for every collection.
Code like this doesn't work in this case:
export const customerCounterListener =
functions.firestore.document('customers/{customerId}')
.onWrite((change, context) => {
// on create
if (!change.before.exists && change.after.exists) {
return firestore
.collection('metadatas')
.doc('customers')
.get()
.then(docSnap =>
docSnap.ref.set({
count: docSnap.data().count + 1
}))
// on delete
} else if (change.before.exists && !change.after.exists) {
return firestore
.collection('metadatas')
.doc('customers')
.get()
.then(docSnap =>
docSnap.ref.set({
count: docSnap.data().count - 1
}))
}
return null;
});
The reason is because every cloud firestore trigger has to be idempotent, as firestore documentation say: https://firebase.google.com/docs/functions/firestore-events#limitations_and_guarantees
Solution
So, in order to prevent multiple executions of your code, you need to manage with events and transactions. This is my particular way to handle large collection counters:
const executeOnce = (change, context, task) => {
const eventRef = firestore.collection('events').doc(context.eventId);
return firestore.runTransaction(t =>
t
.get(eventRef)
.then(docSnap => (docSnap.exists ? null : task(t)))
.then(() => t.set(eventRef, { processed: true }))
);
};
const documentCounter = collectionName => (change, context) =>
executeOnce(change, context, t => {
// on create
if (!change.before.exists && change.after.exists) {
return t
.get(firestore.collection('metadatas')
.doc(collectionName))
.then(docSnap =>
t.set(docSnap.ref, {
count: ((docSnap.data() && docSnap.data().count) || 0) + 1
}));
// on delete
} else if (change.before.exists && !change.after.exists) {
return t
.get(firestore.collection('metadatas')
.doc(collectionName))
.then(docSnap =>
t.set(docSnap.ref, {
count: docSnap.data().count - 1
}));
}
return null;
});
Use cases here:
/**
* Count documents in articles collection.
*/
exports.articlesCounter = functions.firestore
.document('articles/{id}')
.onWrite(documentCounter('articles'));
/**
* Count documents in customers collection.
*/
exports.customersCounter = functions.firestore
.document('customers/{id}')
.onWrite(documentCounter('customers'));
As you can see, the key to prevent multiple execution is the property called eventId in the context object. If the function has been handled many times for the same event, the event id will be the same in all cases. Unfortunately, you must have "events" collection in your database.
Please check below answer I found on another thread. Your count should be atomic. Its required to use FieldValue.increment() function in such case.
https://stackoverflow.com/a/49407570/3337028
firebase-admin offers select(fields) which allows you to only fetch specific fields for documents within your collection. Using select is more performant than fetching all fields. However, it is only available for firebase-admin and firebase-admin is typically only used server side.
select can be used as follows:
select('age', 'name') // fetch the age and name fields
select() // select no fields, which is perfect if you just want a count
select is available for Node.js servers but I am not sure about other languages:
https://googleapis.dev/nodejs/firestore/latest/Query.html#select
https://googleapis.dev/nodejs/firestore/latest/CollectionReference.html#select
Here's a server side cloud function written in Node.js which uses select to count a filtered collection and to get the IDs of all resulting documents. Its written in TS but easily converted to JS.
import admin from 'firebase-admin'
// https://stackoverflow.com/questions/46554091/cloud-firestore-collection-count
// we need to use admin SDK here as select() is only available for admin
export const videoIds = async (req: any): Promise<any> => {
const id: string = req.query.id || null
const group: string = req.query.group || null
let processed: boolean = null
if (req.query.processed === 'true') processed = true
if (req.query.processed === 'false') processed = false
let q: admin.firestore.Query<admin.firestore.DocumentData> = admin.firestore().collection('videos')
if (group != null) q = q.where('group', '==', group)
if (processed != null) q = q.where('flowPlayerProcessed', '==', processed)
// select restricts returned fields such as ... select('id', 'name')
const query: admin.firestore.QuerySnapshot<admin.firestore.DocumentData> = await q.orderBy('timeCreated').select().get()
const ids: string[] = query.docs.map((doc: admin.firestore.QueryDocumentSnapshot<admin.firestore.DocumentData>) => doc.id) // ({ id: doc.id, ...doc.data() })
return {
id,
group,
processed,
idx: id == null ? null : ids.indexOf(id),
count: ids.length,
ids
}
}
The cloud function HTTP request completes within 1 second for a collection of 500 docs where each doc contains a lot of data. Not amazingly performant but much better than not using select. Performance could be improved by introducing client side caching (or even server side caching).
The cloud function entry point looks like this:
exports.videoIds = functions.https.onRequest(async (req, res) => {
const response: any = await videoIds(req)
res.json(response)
})
The HTTP request URL would be:
https://SERVER/videoIds?group=my-group&processed=true
Firebase functions detail where the server is located on deployment.
Following Dan Answer: You can have a separated counter in your database and use Cloud Functions to maintain it. (Write-time best-effort)
// Example of performing an increment when item is added
module.exports.incrementIncomesCounter = collectionRef.onCreate(event => {
const counterRef = event.data.ref.firestore.doc('counters/incomes')
counterRef.get()
.then(documentSnapshot => {
const currentCount = documentSnapshot.exists ? documentSnapshot.data().count : 0
counterRef.set({
count: Number(currentCount) + 1
})
.then(() => {
console.log('counter has increased!')
})
})
})
This code shows you the complete example of how to do it:
https://gist.github.com/saintplay/3f965e0aea933a1129cc2c9a823e74d7
Get a new write batch
WriteBatch batch = db.batch();
Add a New Value to Collection "NYC"
DocumentReference nycRef = db.collection("cities").document();
batch.set(nycRef, new City());
Maintain a Document with Id as Count and initial Value as total=0
During Add Operation perform like below
DocumentReference countRef= db.collection("cities").document("count");
batch.update(countRef, "total", FieldValue.increment(1));
During Delete Operation perform like below
DocumentReference countRef= db.collection("cities").document("count");
batch.update(countRef, "total", FieldValue.increment(-1));
Always get Document count from
DocumentReference nycRef = db.collection("cities").document("count");
I created an NPM package to handle all counters:
First install the module in your functions directory:
npm i adv-firestore-functions
then use it like so:
import { eventExists, colCounter } from 'adv-firestore-functions';
functions.firestore
.document('posts/{docId}')
.onWrite(async (change: any, context: any) => {
// don't run if repeated function
if (await eventExists(context)) {
return null;
}
await colCounter(change, context);
}
It handles events, and everything else.
If you want to make it a universal counter for all functions:
import { eventExists, colCounter } from 'adv-firestore-functions';
functions.firestore
.document('{colId}/{docId}')
.onWrite(async (change: any, context: any) => {
const colId = context.params.colId;
// don't run if repeated function
if (await eventExists(context) || colId.startsWith('_')) {
return null;
}
await colCounter(change, context);
}
And don't forget your rules:
match /_counters/{document} {
allow read;
allow write: if false;
}
And of course access it this way:
const collectionPath = 'path/to/collection';
const colSnap = await db.doc('_counters/' + collectionPath).get();
const count = colSnap.get('count');
Read more: https://code.build/p/9DicAmrnRoK4uk62Hw1bEV/firestore-counters
GitHub: https://github.com/jdgamble555/adv-firestore-functions
Use Transaction to update the count inside the success listener of your database write.
FirebaseFirestore.getInstance().runTransaction(new Transaction.Function<Long>() {
#Nullable
#Override
public Long apply(#NonNull Transaction transaction) throws FirebaseFirestoreException {
DocumentSnapshot snapshot = transaction
.get(pRefs.postRef(forumHelper.getPost_id()));
long newCount;
if (b) {
newCount = snapshot.getLong(kMap.like_count) + 1;
} else {
newCount = snapshot.getLong(kMap.like_count) - 1;
}
transaction.update(pRefs.postRef(forumHelper.getPost_id()),
kMap.like_count, newCount);
return newCount;
}
});

Resources