Using Gmail API, I keep getting hundreds of error 429 "User-rate limit exceeded".
My script sends 100 emails at a time, every 10 minutes.
Shouldn't this be within the API limits?
The error pops up after only 5 or 6 successful sends.
Thanks
Have a look at the Gmail Usage Limits:
So, if you send 100 emails during a very short time, this corresponds to 10 000 units - thats is 40 times more than the allowed quota per second.
So while short bursts are allowed, if you exceed the quota significantly this might be beyond the scope of the allowed burst.
In this case you should implement exponential backoff as recommended by Google.
Related
If I use the gmail api to batch fetch 100 mails, does that count 500 quota units or 5? If its 500, then what should the queue setting on my App Engine queue.yaml be so I don't hit the 250 quota units/sec rate limit?
It will count as n items and not one. Also, 250 Quota units per sec is for per user. So, if in your batch request, you have different users, each user will have a 250 Unit Limit.
I am using the api sending messages in batches.
I'm getting many messages with code 400 and 500.
I need to control the time between requests when sending multiple batches?
example:
messages.get = 5 per second
If I send 100 messages in a batch request, have to wait 20 seconds to send the next batch?
or
need to send 20 requests with 5 messages each?
At this point probably batches of 10 messages each is best. You can change the per-user limit to 50/sec using the developers console. Even then, you can exceed the limit for a little bit before you start getting quota errors. Either way, if you get quota errors for a user you'll want to do some type of backoff for the user.
I send emails with cron job and task queue usage. The job is executed every 15 minutes and the queue used has the following setup:
- name: send-emails
rate: 1/m
max_concurrent_requests: 1
retry_parameters:
task_retry_limit: 0
But quite often apiproxy_errors.OverQuotaError exception happens. I am checking Quota Details and see that I am still within the daily quota (Recipients Emailed, Attachment Data Sent etc.), and I believe I couldn't be over maximum per minute limit, since the the rate I use is just 1 task per minute (i.e. send no more than 1 mail per minute).
Where am I wrong and what should I check?
How many emails are you sending? You have not set a bucket-size, so it defaults to 5. Your rate sets how often the bucket is replenished. So, with your current configuration, you can send 5 emails every minute. That means if you are sending more than 75 emails to the queue every 15 minutes, the queue will fill up, and eventually go over quota.
I have not tried this myself, but when you catch the apiproxy_errors.OverQuotaError exception, does the message contain any detail as to why it is over quota/which quota has been exceeded?
try:
send_mail_here
except apiproxy_errors.OverQuotaError, message:
logging.error(message)
I see some enigmatic time fields in the standard Google app-engine logs which make me curious:
2011-05-09 21:56:00.577 /whatever 200 211ms 775cpu_ms 589api_cpu_ms
0.1.0.1 - - [09/May/2011:21:56:00 -0700] "GET /whatever HTTP/1.1"
200 34 - "AppEngine-Google; (+http://code.google.com/appengine)"
"****.appspot.com" ms=212 cpu_ms=776 api_cpu_ms=589 cpm_usd=0.021713
queue_name=__cron task_name=dc4d411120bc75ea8bbea773d23eaecc
Particularly: ms, cpu_ms, api_cpu_ms, each of them two times with slightly different values.
Additionally, when I log timing information myself with a simple structure below for the GET request, it prints a somewhat lower value. In this case, particulary 182 msecs, against the official 775.
protected void doGet(HttpServletRequest req, HttpServletResponse resp) {
long t0 = System.currentTimeMillis();
//Do the stuff
long t1 = System.currentTimeMillis();
log.info("Completed in " + (t1-t0) + " msecs.\n");
}
So, my questions are: Why the difference between my measured time result and the cpu_ms value and how could I lower it? What are the exact meaning of time values in GAE log fields?
I want to optimize my code and I realized based on the aforementioned facts, that most time (nearly 600 msecs!) doesn't spent directly during doGet request. (I use JPA, URLFetch and this is a cron task.)
211ms: It's the response's time, as it will be perceived by the user who requested the page. You will try to decrease it, in order to improve the speed of your website.
775cpu_ms: According to the App Engine documentation, "CPU time is reported in "seconds," which is equivalent to the number of CPU cycles that can be performed by a 1.2 GHz Intel x86 processor in that amount of time. The actual number of CPU cycles spent varies greatly depending on conditions internal to App Engine, so this number is adjusted for reporting purposes using this processor as a reference measurement."
Then, it's normal not to have the "real" time: it should be different from what you measured with System.currentTimeMillis() because it's adjusted. Instead, you should use the Quota API to monitor the CPU usage: see documentation here. CPU time is billable (the free quota is 6.5 CPU-hours per day, and you can pay for more CPU time). Then, you will try to decrease it, in order to pay less.
589api_cpu_ms: It's the adjuested CPU time spent by the API usage (Datastore, User API, etc.)
We recently got some data back on a benchmarking test from a software vendor, and I think I'm missing something obvious.
If there were 17 transactions (I assume they mean successfully completed requests) per second, and 1500 of these requests could be served in 5 minutes, then how do I get the response time for a single user? Is this sort of thing even possible with benchmarking? I have a lot of other data from them, including apache config settings, but I'm not sure how to do all the math.
Given the server setup they sent, I want to know how I can deduce the user response time. I have looked at other similar benchmarking tests, but I'm having trouble measuring requests to response time. What other data do I need to provide here to get that?
If only 1500 of these can be served per 5 minutes then:
1500 / 5 = 300 transactions per min can be served
300 / 60 = 5 transactions per second can be served
so how are they getting 17 completed transactions per second? Last time I checked 5 < 17 !
This doesn't seem to fit. Or am I looking at it wrongly?
I presume be user response time, you mean the time it takes to serve a single transaction:
If they can serve 5 per second than it takes 200ms (1/5) per transaction
If they can serve 17 per second than it takes 59ms (1/17) per transaction
That is all we can tell from the given data. Perhaps clarify how many transactions are being done per second.