Google's documentation states that there is a 25 quota unit rate limit per user per second:
https://developers.google.com/gmail/api/v1/reference/quota
My use case for the Gmail API requires downloading a user's entire mailbox and it appears that use case may not be feasible given these quotas. Downloading 10,000 messages would take roughly 33 minutes.
However, that link also states that the limit is a moving average and suggests that it can be exceeded in small bursts. Can that limit be exceeded significantly enough to meet my use case?
You can raise the per-user limit to 50 units/user/second for your project in the developers console. The limit is only applied over a longer moving window, so you can definitely exceed it for a little bit (O(seconds)). If you need more for longer period, you can request more quota.
While not a full answer, here is my experience on migrating from Gmail to Google Apps for work. My 120000+ conversations/ 200000+ messages mailbox ran close to the 15 GB limit, so I decided to professionalize my account.
After almost 2 days I am now at 22 percent of that migration both in terms of numbers as well as Gigabytes. That is about 60 megabytes/hour and 1000 messages/hour.
Fortunately this runs in reverse chronological order.
Related
I've searched for this topic, but I haven't found a good explanation of the moving average part.
https://developers.google.com/gmail/api/v1/reference/quota explains how Gmail quota works. It says the following
250 quota units per user per second, moving average (allows short bursts)
Gmail API message.get costs 5 quotas. Does that mean that I can only make 50 (250/5) message.get requests per second?
How does moving average work with the rate limit and when does it allow short bursts?
I just enabled billing just because my daily bandwidth seemed to exceed 1 GB per day.
But I realized from the Billing History section, days before the day the billing is enabled, the daily bandwidth has exceeded 1 GB (even it ever reached 2.5 GB, and the last column says $0.18), and I was not charged anything.
How come the free version of app engine allows more than 1 GB of bandwidth? If it is so, then there is no purpose of enabling billing.
If you want your application to work reliably and serve all requests, you should enable billing when about to get over the limits. Sometimes (for example, if you are at 0.8GB and someone starts 500MB download) you may get a little bit over the quota, but this is usually rare and I would not try to build my business on this.
AppEngine features a member in its logs called cpm_usd. As far as I understand this is the approximate cost of 1000 of such requests in US Dollars.
Since 08/16/2012 these numbers are SIGNIFICANTLY smaller (factor 500) for my app (I did not change my app). I was wondering what this is about?
Did Google change the way they calculate those costs?
Are frontend hours included or does this only include calls to services like the datastore?
The only answer I have is that they stopped including the frontend hours in the calculation (I am currently still in dev mode and am thus accumulating a lot of idle time that could have distorted the original / old result).
I am not sure why your numbers have changed but my understanding is that, as of the pricing changes last year, this number is not relevant anymore.
Around the time your numbers changed Google was adding cost tracking functionality to the AppStats tool. What you can do now is turn on pricing metrics in AppStats and get an accurate picture of the RPC costs of your request (which covers pretty much all costs except instance hours).
A quick test of a few requests on one of my apps shows that the cpm_usd and the cost reported by AppStats are not in line at all. Based on the number reported by cpm_usd for the requests I was just testing there is no way that number could contain datastore costs which means it is basically useless to me.
Check out the cost tracking that AppStats can provide and see how your own numbers line up:
https://developers.google.com/appengine/docs/java/tools/appstats#cost
Update Sept 5, 2012:
I asked about the current relevance of cpm_usd in a recent App Engine office hours hangout and while they could not give an exact answer they indicated they think it is still a relevant number. It would be nice to have more insight into what cpm_usd currently represents. Here is a recording of Amy answering the question:
https://www.youtube.com/watch?v=W-YBnWdllfI&feature=player_detailpage#t=3127s
Does anyone have any number on Google App Engine (free quota) in terms of total number of request and unique visitors it allows per day?
Maybe someone who has live production code can tell us this?
Rough number is enough, just to get the idea.
I can not get this information from the pricing model.
Thanks
I had this question when I first started using App Engine, but it's impossible to answer with the information in your question.
You must have an estimate on the individual API quota usages, then calculate based on that.
You might be able to simplify it by trying to figure out which API quota you're likely to hit first, and then figuring out the number of requests you can serve before that quota runs out. ie:
Storing photos or other large data for users? You'll probably hit the blobstore quota first. Daily/unique visitor counts probably won't matter.
Serving lots of photos or large data? You'll probably hit the bandwidth quota first.
Need to start a channel for every view? You'll probably hit the channel quota first and get 100 views per day.
Need to send an email for every view? You'll probably hit the mail quota first.
Need to query the datastore a lot? You'll probably hit the datastore limit first.
The datastore limit is the hardest to calculate. You get 50k read and 50k write ops. Most likely you'd read more than write.
If you need 2 read ops per page, you might could do 25k views per day.
If you need 2 read ops per page, but you're smart and you memcache them, and memcache is effective 80% of the time, you could get 125k views per day.
If you need 500 read ops per page and you can't cache it, you can do 100 views per day. That's provided you don't run out of one of the other quotas.
Do your own math.
The quotas and rates (for free and paid apps) are listed on https://developers.google.com/appengine/docs/quotas.
I've chosen google-app-engine because of its scalability, and now I try to understand how much I will have to pay once I release the product.
I've looked back and forth in the google app engine documentation to find an answer for question and couldn't find. I found few details in the "Quotas" page, I found how much I can get for free and how much is the Billing Default Quota.
In Billing Page there are number for CPU, etc with Resource and Unit and how much it cost. But no where could I find how much will it cost me per channel calls/created, etc.
I can't even try to make calculations with what's in the Admin console, because the current numbers there now are 0 (since there are 2 users which are the programmers).
How can I be ready for the releasing of the product that (hopefully) will have a huge number of channels created daily?
Is there a page I missed, or is there a tool for calculating?
Thanks!
EDIT:
Moishe, thanks for the quick and readable answer. So here are some more questions:
1. Do you think - if needed - that I will be able to get even more quota for the number of channels? I saw there's a special form to ask for more quotas, but I'm not sure that includes the Channel-API feature...
2. Are there any posts you've made for "how to use channel-API efficiently"? I saw some stuff about reusing the tokens per user. Is there more?
Thanks again.
Creating a channel costs about 2.7 CPU-seconds. A CPU-hour costs $0.10. So, each channel created costs
(2.7 / 3 600) * $0.10 = $7.5 × 10^-5
So creating 1000 channels will cost $0.075, or 7.5 cents.
You'll also get charged the normal outgoing bandwidth costs for any data sent over a channel.
The CPU cost probably isn't the biggest concern; you're more likely to run into quota caps then running out of money. Paid apps are limited to 86400 channel creations/day (1/second).