I am building a chat app in react using Firebase Firestore as backend database.
I get recent 25 messages in useEffect hook as
useEffect(() => {
const q = query(
collection(db, 'messages'),
orderBy('createdAt', 'desc'),
limit(25)
);
return onSnapshot(q, (snapshot) => {
setData(
snapshot.docs.map((doc) => {
console.log('document read');
return { ...doc.data(), id: doc.id };
})
);
});
}, []);
But this operation results in 25 document reads on page load and 50 additional on sending a message.
If more users are connected, 25 request per user happen on single message send by any user.
Is there any way to reduce the reads?
Complete code:- https://github.com/Puneet56/Converse
You don't get 25 reads in a running query with a limit of 25 resulst if you get a new one inside. As the documentation says:
When you listen to the results of a query, you are charged for a read
each time a document in the result set is added or updated. You are
also charged for a read when a document is removed from the result set
because the document has changed. (In contrast, when a document is
deleted, you are not charged for a read.)
Also, if the listener is disconnected for more than 30 minutes (for
example, if the user goes offline), you will be charged for reads as
if you had issued a brand-new query.
As stated in the docs you will be charged only for the one that is added and the one that get's out of the query limit because a new one got in. So you get probably only 2 reads per new message. I think that is a reasonable amount. I can't any way to reduce this amount in a chat App. Even if you would increase the query limit only your inital reads (if it's a fresh read with no old or empty cache) will increas to but the reads while listening would stay the same.
Related
I want to create a member count that excludes all bots.
I've talked to a friend that told me the code looks to be correct. But it only gives me a count of 0. Perhaps you have better eyes than him!
module.exports = async(client) => {
const guild = client.guilds.cache.get('912706237806829598');
setInterval(() => {
const memberCount = guild.members.cache.filter(member => !member.user.bot).size;
const channel = guild.channels.cache.get('959556115383861298');
channel.setName(`╭👥・Members: ${memberCount.toLocaleString()}`);
console.log('Updating Member Count');
console.log(memberCount);
}, 5000);
}
Thanks.
Tried my code and expect a value representing the amount of humans on my server, but only got 0.
There are two potential problems with your code. First, you're not fetching the member collection and are instead interacting with the cache; if I remember correctly the cache usually only shows you online members and not offline ones. Second, the dreaded issue of rate limits.
The first issue is an easy fix. You just need to fetch the members instead of using the cache before all of your setInterval code:
setInterval(async () => {
const memberCount = (await guild.members.fetch()).filter(member => !member.user.bot).size;
const channel = guild.channels.cache.get('959556115383861298');
channel.setName(`╭👥・Members: ${memberCount.toLocaleString()}`);
console.log('Updating Member Count');
console.log(memberCount);
}, 5000);
That was probably causing the issue you were having. However, there's an additional problem here, one you will run into if you use this code. Rate limits. The normal rate limit for most actions in the Discord API is, if I remember correctly, 10,000 requests per 10 mins. However, the rate limit for changing the name of a channel is just 2 requests per 10 mins (last I checked; I don't know if they've increased this limit since last year). Your code is changing the name of a channel every 5 seconds. In just 10-15 seconds, you'll surpass the rate limit.
I have created channels that keep track of server stats for my bots in the past as well, and ran into this issue when I was doing so. Luckily, I brewed up a way to bypass the rate limit issue back when I was creating my own stats system. Though changing channel names has an incredibly low rate limit, fully creating channels has the normal rate limit of 10,000 requests per 10 mins. Therefore, instead of changing the channel name directly, you could instead: a) clone the channel; b) change the clone's name while creating it; c) set the clone's position in the channel list to the same position as the actual channel; d) delete the original channel. The cloned channel will keep all of the perms, settings, etc of the original. This all assumes, of course, that the original channel isn't a text channel with important messages or such inside, as those messages would all disappear with this method. I used voice channels for my system.
Here's how this bypass could look:
setInterval(async () => {
const memberCount = (await guild.members.fetch()).filter(member => !member.user.bot).size;
const channel = guild.channels.cache.get('959556115383861298');
const pos = channel.position;
const clone = await channel.clone({
name: `╭👥・Members: ${memberCount.toLocaleString()}`
});
await clone.setPosition(pos);
console.log('Updating Member Count');
console.log(memberCount);
await channel.delete();
}, 5000);
With this approach, rate limits are no longer an issue. Note that this can still be further improved, however. For example, inside the interval you could check if the member count has changed from the previous count, and only modify the channel name if and only if the count has changed. The entire approach potentially could be changed as well; perhaps you could use guildMemberAdd and guildMemberRemove events to track when the member count increases or decreases, and only modify the name when one of those happens (and only if the member being added/removed is not a bot). But whether to add either of those optional potential improvements is up to you.
Note: Please try this code out on a test channel before using it on the actual channel you are using, just to be safe.
If you wanted to get to all members including the bot you can use memberCount
const totalCount = guild.memberCount
If you wanted to get only the members size
const memberCount = guild.members.cache.filter(member => !member.user.bot).size;
And if you only wanted to get the bot size
const botCount = guild.members.cache.filter(member => member.user.bot).size;
Full code:
const guild = client.guilds.cache.get('912706237806829598')
const memberCount = guild.members.cache.filter(member => !member.user.bot).size;
const botCount = guild.members.cache.filter(member => member.user.bot).size;
const totalCount = guild.memberCount
console.log(memberCount, botCount, totalCount)
I was trying to debug a problem related to refunding Paypal orders (in a sandbox environment) using order IDs (which were stored previously). Every time I tried to perform a refund, the Paypal API would return an INVALID_RESOURCE_ID error, meaning that no such order existed. After much debugging, I have made a revelation with the initial process when I stored said order ID. The following method is how I am retrieving and storing said order id:
const onApprove = (data, actions) => {
// Redux method of saving checkout in backend with order ID via using data.orderID
dispatch(saveCheckout(data.orderID);
return actions.order.capture();
}
<PayPalButton
amount={totalPrice}
currency= "AUD"
createOrder={(data, actions) => createOrder(data, actions)}
onApprove={(data, actions) => onApprove(data, actions)}
options={{
clientId: "<placeholder>",
currency: "AUD"
}}
/>
I am using the recommended data.orderID from the docs and yet, upon inspecting the network tab, the following is shown:
{"id":"5RJ421191B663801G","intent":"CAPTURE","status":"COMPLETED","purchase_units":[{"reference_id":"default","amount":{"currency_code":"AUD","value":"24.00"},"payee":{"email_address":"sb-sg4zd7438633#business.example.com","merchant_id":"EJ7NSJGC6SRXQ"},"shipping":{"name":{"full_name":"John Doe"},"address":{"address_line_1":"1 Cheeseman Ave Brighton East","admin_area_2":"Melbourne","admin_area_1":"Victoria","postal_code":"3001","country_code":"AU"}},"payments":{"captures":[{"id":"7A2856455D561633D","status":"COMPLETED","amount":{"currency_code":"AUD","value":"24.00"},"final_capture":true,"seller_protection":{"status":"ELIGIBLE","dispute_categories":["ITEM_NOT_RECEIVED","UNAUTHORIZED_TRANSACTION"]},"create_time":"2021-10-11T00:40:58Z","update_time":"2021-10-11T00:40:58Z"}]}}],"payer":{"name":{"given_name":"John","surname":"Doe"},"email_address":"sb-432azn7439880#personal.example.com","payer_id":"KMEQSKCLCLUZ4","address":{"country_code":"AU"}},"create_time":"2021-10-11T00:40:48Z","update_time":"2021-10-11T00:40:58Z","links":[{"href":"https://api.sandbox.paypal.com/v2/checkout/orders/5RJ421191B663801G","rel":"self","method":"GET"}]}
The id saved by onApprove is 5RJ421191B663801G but there is another ID under captures and id which is 7A2856455D561633D. This is the actual order id I need to save in order to make the refund later on. However, I am struggling as to how I can retrieve this value as that id value seems to be only visible via the network. The objects returned via the onApprove and action.order.get() methods only return the first "false" id. Any advice would be greatly appreciated.
These are two separate types of IDs, the order ID (used only during buyer checkout approval), and the payment/transaction ID (which only exists after an order is captured, and is the one needed for any later refund or accounting purposes)
Since you are capturing on the client side with actions.order.capture(), this is where you would need to add a .then(function(data){ ... }) to do something with the capture data (particularly data.purchase_units[0].payments.captures[0].id). That is the id you would use for a refund.
In actual best practice, if anything important needs to be done with the capture id -- such as storing it in a database for reference -- you should not be creating and capturing orders on the client side, and instead calling a server-side integration where that database write will be performed.
Follow the Set up standard payments guide and make 2 routes on your server, one for 'Create Order' and one for 'Capture Order', documented here. Both routes should return only JSON data (no HTML or text). Inside the 2nd route, when the capture API is successful you should store its resulting payment details in your database (particularly the aforementioned purchase_units[0].payments.captures[0].id, which is the PayPal transaction ID) and perform any necessary business logic (such as sending confirmation emails or reserving product) immediately before forwarding your return JSON to the frontend caller.
Pair those 2 routes with the frontend approval flow: https://developer.paypal.com/demo/checkout/#/pattern/server
Or for react, use the official react-paypal-js
I am a beginner in React native and firestore, and using these to build a kind of social media app, and I have a weird problem(I think I structured the db the wrong way). I want to have a feed, with all posts, no following-based, no nothing. The first time I structured my posts in db like this: users(collection)->user(doc)->thisUserPosts(collection inside doc) - but I couldn't find a way to fetch through all the thisUserPosts from all user(doc) and display them properly.
So I re-structured the db like this:
2 main collection, posts and users. Completely different. In users collection, only docs of users and their data(name, age, etc). In the other, their posts(name, media, desc, AND userId - where userId == the person who created it. userId field from posts collection docs should exist in users collection).
This second approach works just fine. In feed, I only fetch posts. But the problem arrises when I try to open the post(need to have this feature). I need to be able to display on react-navigation header the name of the user, yet I only have details of the post and only userId, which is to no good use.
So I came up with a solution : add a userName field in the posts collection doc, next to userId and simply display that. Now here's the catch: I need to figure a way(in firestore I think) to listen to updates from users collection docs, in case a user updates his name/username(I don't want to showcase the old name). And I don't know if that's possible inside firestore or how. Or is it better to find a different db structure?
TLDR: Need a function in firestore to listen to updates from other collection OR restructuring the db.
If you are fetching posts of a single user then you can just set a listener for his document.
Make sure that document has no sensitive information that must not be shared with others and is limited to the owner only.
If you are fetching posts from multiple users then you can use in operator:
db.collection("users").where("userID", "in", ["user_id1", "user_id2"])
.onSnapshot((snapshot) => {
console.log(snapshot.docs.map(user => user.data()))
});
If I assume you will be updating the new name in all the user's posts then you can set the listener on the posts document itself but that won't be nice in case all 30 posts fetched are from same user. That'll end up costing 30 reads just to update the same name.
Edit:
A simple example of reading a user's posts and listening updates on the user name:
const userID = "my_user_id"
// fetching user's 30 posts
const postsRef = firebase.firebase().collection("posts").where("userID", "==", userID).limit(30)
const postsSnapshot = await postsRef.get()
const postsData = postsSnapshot.docs.map(post => post.data())
// Array of posts data objects
// listening to change in user's name
firebase.firestore().collection("users").doc("user_id")
.onSnapshot((doc) => {
console.log("data: ", doc.data());
const newUsername = doc.data().username
const updatedPostsData = postsData.map(post => {
return ({...post, username: newUsername})
})
});
In my (greatly simplified) model I have users, accounts and account_types. Each user can have multiple accounts of each account_type. When an account of type TT is created I'm updating the "users" field of that object so it keeps the users which have accounts of that types, and the number of such accounts they have.
users: {
some fields
},
accounts: {
userID: UU,
type: TT
},
account_type:
users: { UU: 31 }
}
I use the onCreate and onDelete cloud triggers for accounts to update the account_type object. Since multiple accounts can be created simultaneously I have to use transactions:
exports.onCreateAccount = functions.firestore
.document('accounts/{accountID}')
.onCreate((account, context) => {
const acc_user = account.data().userID;
const acc_type = account.data().type;
return admin.firestore().runTransaction(transaction => {
// This code may get re-run multiple times if there are conflicts.
const accountTypeRef = admin.firestore().doc("account_types/"+acc_type);
return transaction.get(accountTypeRef).then(accTypeDoc => {
var users = accTypeDoc.data().users;
if (users === undefined) {
users = {};
}
if (users[acc_user] === undefined) {
users[acc_user] = 1;
} else {
users[acc_user]++;
}
transaction.update(accountTypeRef, {users: users});
return;
})
})
.catch(error => {
console.log("AccountType create transaction failed. Error: "+error);
});
});
In my tests I'm first populating the database with some data so I'm also adding a user and 30 accounts of the same type. With the local emulator this works just fine and at the end of the addition I see that the account_type object contains the user with the counter at 30. But when deployed to Firebase and running the same functions the counter gets to less than 30. My suspicion is that since Firebase is much slower and transactions take longer, more of them are conflicted and fail and eventually don't execute at all. The transaction failure documentation (https://firebase.google.com/docs/firestore/manage-data/transactions) says:
"The transaction read a document that was modified outside of the transaction. In this case, the transaction automatically runs again. The transaction is retried a finite number of times."
So my questions:
What does "finite" mean?
Any way to control this number?
How can I make sure my transactions are executed at some point and don't get dropped like that so my data is consistent?
Any other idea as to why I'm not getting the correct results when deployed to the cloud?
What does "finite" mean?
It's the opposite of "unlimited". It will retry no more than a set number of times.
Any way to control this number?
Other than modifying the source code of the SDK, no. The SDK itself advertise a specific number, as it might change.
How can I make sure my transactions are executed at some point and don't get dropped like that so my data is consistent?
Detect the error and retry in your app. If you aren't seeing the transaction fail with an error, then nothing went wrong.
Any other idea as to why I'm not getting the correct results when deployed to the cloud?
Since we can't see what exactly you're doing to trigger the function, and have no specific expected results to compare to, it's not really possible to say.
I have a server-side-rendered reactjs app using firebase firestore.
I have an area of my site that server-side-renders content that needs to be retrieved from firestore.
Currently, I am using firestore rules to allow anyone to read data from these particular docs
What worries me is that some bad person could setup a script to just continuously hit my database with reads and rack up my bills (since we are charged on a per-read basis, it seems that it's never wise to allow anyone to perform reads.)
Current Rule
// Allow anonymous users to read feeds
match /landingPageFeeds/{pageId}/feeds/newsFeed {
allow read: if true;
}
Best Way Forward?
How do I allow my server-side script to read from firestore, but not allow anyone else to do so?
Keep in mind, this is an initial action that runs server-side before hydrating the client-side with the pre-loaded state. This function / action is also shared with client-side for page-to-page navigation.
I considered anonymous login - which worked, however, this generated a new anonymous user with every page load - and Firebase does throttle new email/password and anonymous user accounts. It did not seem practical.
Solution
Per Doug's comment, I thought about the admin SDK more. I ended up creating a separate API in firebase functions for anonymous requests requiring secure firestore reads that can be cached.
Goals
Continue to deny public reads of my firestore database
Allow anonymous users to trigger firestore reads for server-side-rendered reactjs pages that require data from Firestore database (like first-time visitors, search engines).
Prevent "read spam" where a third party could hit my database with millions of reads to drive up my cloud costs by using server-side CDN cache for the responses. (by invoking unnessary reads in a loop, I once racked up a huge bill on accident - I want to make sure strangers can't do this maliciously)
Admin SDK & Firebase Function Caching
The admin SDK allows me to securely read from firestore. My firestore security rules can deny access to non-authenticated users.
Firebase functions that are handling GET requests support server caching the response. This means that subsequent hits from identical queries will not re-run all of my functions (firebase reads, other function invocations) - it will just instantly respond with the same data again.
Process
Anonymous client visits a server-side rendered reactjs page
Initial load rendering on server triggers a firebase function (https trigger)
Firebase function uses Admin SDK to read from secured firestore database
Function caches the response for 3 hours res.set('Cache-Control', 'public, max-age=600, s-maxage=10800');
Subsequent requests from any client anywhere for the next 3 hours are served from the cache - avoiding unnecessary reads or additional computation / resource usage
Note - caching does not work on local - must deploy to firebase to test caching effect.
Example Function
const functions = require("firebase-functions");
const cors = require('cors')({origin: true});
const { sendResponse } = require("./includes/sendResponse");
const { getFirestoreDataWithAdminSDK } = require("./includes/getFirestoreDataWithAdminSDK");
const cachedApi = functions.https.onRequest((req, res) => {
cors(req, res, async () => {
// Set a cache for the response to limit the impact of identical request on expensive resources
res.set('Cache-Control', 'public, max-age=600, s-maxage=10800');
// If POST - response with bad request code - POST requests are not cached
if(req.method === "POST") {
return sendResponse(res, 400);
} else {
// Get GET request action from query
let action = (req.query.action) ? req.query.action : null;
console.log("Action: ", action);
try {
// Handle Actions Appropriately
switch(true) {
// Get Feed Data
case(action === "feed"): {
console.log("Getting feed...");
// Get feed id
let feedId = (req.query.feedId) ? req.query.feedId : null;
// Get feed data
let feedData = await getFirestoreDataWithAdminSDK(feedId);
return sendResponse(res, 200, feedData);
}
// No valid action specified
default: {
return sendResponse(res, 400);
}
}
} catch(err) {
console.log("Cached API Error: ", err);
return sendResponse(res, 500);
}
}
});
});
module.exports = {
cachedApi
}