The logs tell me how much a certain request costs
api_cpu_ms=278 cpm_usd=0.009244
Is this cost still accurate with the new billing model?
Can I deduce anything about the number of datastore operations the request used?
Is there any way to know the exact number of read, write and small datastore operations used during a single request?
I was told about 2 months ago that they do not reflect the new billing model, and I haven't seen any indications that they've been fixed to reflect the new billing model.
You can get the number of operations per request by enabling Appstats in your application.
Related
I'd like to make a GAE app multi-tenant to cater to different clients (companies), database namespaces seems like a GAE endorsed solution. Is there a meaningful way to split GAE fees among client/namespaces? GAE costs for app are mainly depends on user activities - backend instances up time, because new instances are created or (after 15 min delay) terminated proportionally to the server load, not total volume of data user has or created. Ideally the way the fees are split should be meaningful and could be explained to the clients.
I guess the most fair fee splitting solution is just create a new app for a new client, so all costs reported separately, yet total cost will grow up, I expect few apps running on same instances will use server resources more economically.
Every app engine request is logged with a rough estimated cost measurement. It is possible to log the namespace/client associated with every request and query the logs to add up the estimated instance costs for that namespace. Note that the estimated cost field is deprecated and may be inaccurate. It is mostly useful as a rough guide to the proportion of instance cost associated with each client.
As far as datastore pricing goes, the cloud console will tell you how much data has been stored in each namespace, and you can calculate costs from that. For reads/writes, we have set up a logging system to help us track reads and writes per namespace (i.e. every request tracks the number of datastore reads and writes it does in each namespace and logs these numbers at the end of the request).
The bottom line is that with some investments into infrastructure and logging, it is possible to roughly track costs per namespace. But no, App Engine does not make this easy, and it may be impossible to calculate very accurate cost estimates.
I have launched a new web app this month. I'm trying to understand why I'm getting such a high datastore read value, even though all my entities are cached.
So, my main point of misunderstanding is this. in the total quota overview for this month I have 1.12 M read operations in the datastore.
But when I go to the memcache section in the console, it tells me the hit ratio is 96.35% and the numbers are: 1,457,499 hit / 55,177 miss
First of all, is it true that these numbers are per month or are they per day?
Second, how is this possible?
I know that reads in transactions don't user the cache. But I do not make heavy use of transactions. Is there anything other than transactions that can cause this?
If you want more insight into your Objectify memcache hit rates, mount the MemcacheStatsServlet (or look at its code and do something similar). This will provide your cache hit ratio broken down by Kind.
Keep in mind that since it is reporting for just one instance (whichever you happen to hit with your request for stats), this is only a representative sample of what is going on in your cluster.
Does anyone have any number on Google App Engine (free quota) in terms of total number of request and unique visitors it allows per day?
Maybe someone who has live production code can tell us this?
Rough number is enough, just to get the idea.
I can not get this information from the pricing model.
Thanks
I had this question when I first started using App Engine, but it's impossible to answer with the information in your question.
You must have an estimate on the individual API quota usages, then calculate based on that.
You might be able to simplify it by trying to figure out which API quota you're likely to hit first, and then figuring out the number of requests you can serve before that quota runs out. ie:
Storing photos or other large data for users? You'll probably hit the blobstore quota first. Daily/unique visitor counts probably won't matter.
Serving lots of photos or large data? You'll probably hit the bandwidth quota first.
Need to start a channel for every view? You'll probably hit the channel quota first and get 100 views per day.
Need to send an email for every view? You'll probably hit the mail quota first.
Need to query the datastore a lot? You'll probably hit the datastore limit first.
The datastore limit is the hardest to calculate. You get 50k read and 50k write ops. Most likely you'd read more than write.
If you need 2 read ops per page, you might could do 25k views per day.
If you need 2 read ops per page, but you're smart and you memcache them, and memcache is effective 80% of the time, you could get 125k views per day.
If you need 500 read ops per page and you can't cache it, you can do 100 views per day. That's provided you don't run out of one of the other quotas.
Do your own math.
The quotas and rates (for free and paid apps) are listed on https://developers.google.com/appengine/docs/quotas.
I'm having some trouble with the google app engine datastore. Ever since the new pricing model was introduced, the cost of running my app has increased massively.
The culprit appears to be "Datastore small operations", which come in at more than 20 Million ops per day!
Has anyone had this problem, I don't think I'm doing an excessive amount of key lookups, and I only have 5000 users, with roughly 10 - 20 requests per minute.
Thanks in advance!
Edit
Ok got some stats, these are after abut 3 hours. Here is what I am seeing in my dashboard, in the billing section:
And here are some of the stats:
Obviously there are quite a lot of calls to datastore.get. I am starting to think that it is my design that is causing the problem. Those gets correspond to accounts. Every user has an account, but an account can be one of two types, for this I use composition. So each account entity has a link to its sub account entity.
As a result when I do a search for nearby users it involves fetching the accounts using the query, and then doing a get on each account to get its sub account. The top request in the stats picture is a call that gets 100 accounts, and then has to do a get on each one. I would have thought that this was a very light query, but I guess not. And I am still confused by the number of datastore small ops being recorded in my dashboard.
Definitely use appstats as Drew suggests; regardless of what library you're using, it will tell you what operations your handlers are doing. The most likely culprits are keys-only queries and count operations.
My advice would be to use AppStats (Python / Java) to profile your traffic and figure out which handler is generating the most datastore ops. If you post the code here we can potentially suggest optimizations.
Don't scan your datastore, use get(key) or get_by_id(id) or get_by_key_name(keyname) as much as you can.
Do you have lots of ReferenceProperty properties in your models? Accessing them will trigger db.get for each property unless you prefetch them. This would trigger 101 db.get requests.
class Foo(db.Model):
user = db.ReferenceProperty(User)
foos = Foo.all().fetch(100)
for f in foos:
print f.user.name # this triggers db.get(parent=f, key=f.user)
I've chosen google-app-engine because of its scalability, and now I try to understand how much I will have to pay once I release the product.
I've looked back and forth in the google app engine documentation to find an answer for question and couldn't find. I found few details in the "Quotas" page, I found how much I can get for free and how much is the Billing Default Quota.
In Billing Page there are number for CPU, etc with Resource and Unit and how much it cost. But no where could I find how much will it cost me per channel calls/created, etc.
I can't even try to make calculations with what's in the Admin console, because the current numbers there now are 0 (since there are 2 users which are the programmers).
How can I be ready for the releasing of the product that (hopefully) will have a huge number of channels created daily?
Is there a page I missed, or is there a tool for calculating?
Thanks!
EDIT:
Moishe, thanks for the quick and readable answer. So here are some more questions:
1. Do you think - if needed - that I will be able to get even more quota for the number of channels? I saw there's a special form to ask for more quotas, but I'm not sure that includes the Channel-API feature...
2. Are there any posts you've made for "how to use channel-API efficiently"? I saw some stuff about reusing the tokens per user. Is there more?
Thanks again.
Creating a channel costs about 2.7 CPU-seconds. A CPU-hour costs $0.10. So, each channel created costs
(2.7 / 3 600) * $0.10 = $7.5 × 10^-5
So creating 1000 channels will cost $0.075, or 7.5 cents.
You'll also get charged the normal outgoing bandwidth costs for any data sent over a channel.
The CPU cost probably isn't the biggest concern; you're more likely to run into quota caps then running out of money. Paid apps are limited to 86400 channel creations/day (1/second).