I currently have a web socket which pushes data in a JsonAPI format to a react-native app, which then places the items in a redux store at intervals.
Works perfectly for 5,000 - 10,000 items. However I need to support around 30,000 items. Each item(object) has on average 12-ish keys.
Currently this crashes the app, either when I normalize parts of the data or merge new chunks of data with existing data already in my store.
Any advice would be really appreciated.
import debounce from 'lodash/debounce'
import merge from 'lodash/merge
let data = {}
const migrateData = () => {
InteractionManager.runAfterInteractions(() => {
dispatch(createSocketData(data))
data = {}
})
}
const debounced = debounce(migrateData, 3000)
cable.subscriptions.create('SyncChannel', {
received: (payload) => {
if(payload.serialized){
data = merge(data, normalize(payload.serialized))
debounced()
}
}
})
I would not recommend you to store big data in redux store. For now you need to support 30,000 items. Tomorrow you need to support 50,000 items, 100,000 items...
You can try to use storage specific components to solve your problem.
For example i use SQLite3.
SQLite3 component for react-native support both for IOS and Android
You might want to use Realm.
There sholud be no issues storing 1000 objects in Realm. It has been optimized for storing huge data and only loads data into memory when needed.
Related
I'm developing an app using React Native that allows you to create your own checklists and add items to them.
For example you'd have "Create Checklist", and inside that you'll have the option to "Add Item", "Delete Item" "Edit Item", basic CRUD methods etc.
It's going to be completely offline but I'm wondering what the best approach to storing this data locally would be.
Should I be using a DB such as firebase? I have read that it is overkill and to use something like Redux but I'm not sure if the latter will accomplish everything I need. As long as it's storing data which can be edited, and will save on the user's device (with minimal effort) it sounds good to me.
Would appreciate some input on this, thanks!
You could use AsyncStorage for persisting data locally on the user's phone. It is a simple persistent key-value-storage.
Each checklist is most likely an array of JS objects. The documentation provides an example on how to store objects.
const storeData = async (value) => {
try {
const jsonValue = JSON.stringify(value)
await AsyncStorage.setItem('#storage_Key', jsonValue)
} catch (e) {
// saving error
}
}
The value parameter is any JS object. We use JSON.stringify to create a JSON string. We use AsyncStorage.setItem in order to persist the data. The string #storage_Key is the key for the object. This could be any string.
We retrieve a persisted object as follows.
const getData = async () => {
try {
const jsonValue = await AsyncStorage.getItem('#storage_Key')
return jsonValue != null ? JSON.parse(jsonValue) : null;
} catch(e) {
// error reading value
}
}
Both examples are taken from the official documentation.
Keep in mind that this functionality should be used for persistence only. If the application is running, you should load the complete list, or parts of the list if it is very large, in some sort of application cache. The implementation for this functionality now heavily depends on how your current code looks like. If you have a plain view, then you could access the local storage in an effect and just store it in a local state.
function MySuperList() {
const [list, setList] = useState([]);
React.useEffect(() => {
// retrieve data using the above functionality and set the state
}, [])
// render list
return (...)
}
I would implement some sort of save button for this list. If it is pressed, then we persist the data in the local storage of the phone.
I am developing a chrome extension that reads data from firestore, in the last few days, i'd finish a day's work with a maximum of 50 reads, today for some reason it is showing a whopping 34K reads, how can I fix this error?
This is my code
useEffect(() => {
const getNotes = () => {
db.collection("Users")
.doc(user.email)
.collection("Notes")
.get()
.then((snapshot) => {
const loadedNotes = snapshot.docs.map((docs) => {
return {
note: docs.data().note,
id: docs.id,
};
});
setNotes(Object.values(loadedNotes) ?? []);
});
};
getNotes();
});
This function is written 3 times in 3 different components, for notes, todos and upcoming Events. the todos have an extra document field which is the state of completion and upcoming Events have a field for event date.
I also have firebase authentication in my project, I don't know is this matters on not.
Where are your useEffect dependencies? As written, this will run on EVERY render of the component... is that what you intended?
If you are using get() on collection references, this loads every document in that collection every time it is requested which is highly excessive on reads. You should be using using limit(n) and pagination where needed, and if possible: source:cache on documents.
Ideally, you should be using some sort of redux/redis local storage which you add the data too once on app load and read from before making random scraping to Firebase Collections
https://firebase.google.com/docs/reference/android/com/google/firebase/firestore/Source
https://firebase.google.com/docs/firestore/query-data/order-limit-data
So in React, I'm reading a firebase real-time database using the "react-firebase-hooks/database" package.
import { useList } from "react-firebase-hooks/database";
import { db, auth } from "../../firebase";
function GameHistory() {
var dbRef = db.ref("/" + user.uid);
const [snapshots, loading, error] = useList(dbRef);
So basically snapshots variable contains all the firebase realtime database data.
Then later in my code I simply map each element of the snapshot array into a component.
Here is the problem, I want to sort my snapshots data in order of the .timestamp firebase property in my data. I'm not sure how to do this.
I tried to sort the snapshot data when I map it:
snapshots
.sort((a, b) => {
return a.val().timestamp > b.val().timestamp;
})
.map((game, index) => (MORE CODE
But that doesn't work because timestamp is a firebase object, and JavaScript doesn't know what to do with it.
Just to establish more context on the timestamp variable, I defined it as such:
timestamp: firebase.firestore.FieldValue.serverTimestamp()
So is there any way to sort my snapshot data? Or should I use another package? If I should use something else please show code for how you read and sort the realtime db.
You're mixing up two different databases here:
The import { useList } from "react-firebase-hooks/database" and other code are for the Realtime Database.
The timestamp: firebase.firestore.FieldValue.serverTimestamp() is for Cloud Firestore
While both databases are part of Firebase, they are completely separate and each has their own API.
To write a server-side timestamp to Realtime Database, use:
timestamp: firebase.database.ServerValue.TIMESTAMP
I'd actually also let the database server handle the sorting, instead of doing this in your code, which is done with:
var dbRef = db.ref("/" + user.uid);
const dbQuery = dbRef.orderByChild("timestamp");
const [snapshots, loading, error] = useList(dbQuery);
...
Actually am new in react and am trying to create an event app in which a user can join an event
here is code for joining an event
export const JoinEvent = (id) => {
return async dispatch => {
let data = await firebase.firestore().collection('Events').doc(id).get()
let tmpArray = data.data()
let currentUser = firebase.auth().currentUser
let newArray = tmpArray.PeopleAttending
await firebase.firestore().collection('Events').doc(id).update({
PeopleAttending : {...newArray, [currentUser.uid]: {displayName : currentUser.displayName}}
})
}
}
actually i have created an action bascailly in JoinEvent an id is passed of the particular event which is clicked.
here is my firestore structure look like this..
so basically i have to download the whole data and store in local array and then add new user and then finally update
So here am basically download the whole data is there any way to just simply add new Object without downloading whole data??
thankyou
You are doing it wrong. Firestore document size limit is Maximum size for a document 1 MiB (1,048,576 bytes), so sooner or later you're going to reach that limit if you keep adding data like this. It may seems that you're not going to reach that limit, but it's very unsafe to store data that way. You can check Firestore query using an object element as parameter how to query objects in firestore documents, but I suggest you don't do it that way.
The proper way to do it, is to create a subcollection PeopleAttending on each document inside the Events collection and then use that collection to store the data.
Also you can try document set with merge or mergeFields like documented here https://googleapis.dev/nodejs/firestore/latest/DocumentReference.html#set and here https://stackoverflow.com/a/46600599/1889685.
Okay. I'm kinda new to react and I'm having a #1 mayor issue. Can't really find any solution out there.
I've built an app that renders a list of objects. The list comes from my mock API for now. The list of objects is stored inside a store. The store action to fetch the objects is done by the components.
My issue is when showing these objects. When a user clicks show, it renders a page with details on the object. Store-wise this means firing a getSpecific function that retrieves the object, from the store, based on an ID.
This is all fine, the store still has the objects. Until I reload the page. That is when the store gets wiped, a new instance is created (this is my guess). The store is now empty, and getting that specific object is now impossible (in my current implementation).
So, I read somewhere that this is by design. Is the solutions to:
Save the store in local storage, to keep the data?
Make the API call again and get all the objects once again?
And in case 2, when/where is this supposed to happen?
How should a store make sure it always has the expected data?
Any hints?
Some if the implementation:
//List.js
componentDidMount() {
//The fetch offers function will trigger a change event
//which will trigger the listener in componentWillMount
OfferActions.fetchOffers();
}
componentWillMount() {
//Listen for changes in the store
offerStore.addChangeListener(this.retriveOffers);
}
retrieveOffers() {
this.setState({
offers: offerStore.getAll()
});
}
.
//OfferActions.js
fetchOffers(){
let url = 'http://localhost:3001/offers';
axios.get(url).then(function (data) {
dispatch({
actionType: OfferConstants.RECIVE_OFFERS,
payload: data.data
});
});
}
.
//OfferStore.js
var _offers = [];
receiveOffers(payload) {
_offers = payload || [];
this.emitChange();
}
handleActions(action) {
switch (action.actionType) {
case OfferConstants.RECIVE_OFFERS:
{
this.receiveOffers(action.payload);
}
}
}
getAll() {
return _offers;
}
getOffer(requested_id) {
var result = this.getAll().filter(function (offer) {
return offer.id == requested_id;
});
}
.
//Show.js
componentWillMount() {
this.state = {
offer: offerStore.getOffer(this.props.params.id)
};
}
That is correct, redux stores, like any other javascript objects, do not survive a refresh. During a refresh you are resetting the memory of the browser window.
Both of your approaches would work, however I would suggest the following:
Save to local storage only information that is semi persistent such as authentication token, user first name/last name, ui settings, etc.
During app start (or component load), load any auxiliary information such as sales figures, message feeds, and offers. This information generally changes quickly and it makes little sense to cache it in local storage.
For 1. you can utilize the redux-persist middleware. It let's you save to and retrieve from your browser's local storage during app start. (This is just one of many ways to accomplish this).
For 2. your approach makes sense. Load the required data on componentWillMount asynchronously.
Furthermore, regarding being "up-to-date" with data: this entirely depends on your application needs. A few ideas to help you get started exploring your problem domain:
With each request to get offers, also send or save a time stamp. Have the application decide when a time stamp is "too old" and request again.
Implement real time communication, for example socket.io which pushes the data to the client instead of the client requesting it.
Request the data at an interval suitable to your application. You could pass along the last time you requested the information and the server could decide if there is new data available or return an empty response in which case you display the existing data.