I am getting the DeadlineExceededError with the Google App Engine.
Read this https://developers.google.com/appengine/articles/deadlineexceedederrors I believe that it is due to:
google.appengine.runtime.apiproxy_errors.DeadlineExceededError: raised if an RPC exceeded its deadline. This is typically 5 seconds, but it is settable for some APIs using the 'deadline' option.
However, I can't find how to set the deadline option. How do I do it? I can only find examples of how to set the deadline for url_fetch, not the apiproxy_errors.
Adding more details:
I am using Python. An example of my usage is:
taskqueue.add( url='/hl', deadline=60)
So after I add to the taskqueue, the page at xyz.appspot.com/hl will be called, but it gave me the deadline exceeded error after 5 seconds.
Related
I am using Pub/Sub push subscription, ack deadline is set to 10 minutes, the push endpoint is hosted within AppEngine using basic scaling.
In my logs, I see that some of the Pub/Sub (supposedly delivered to starting instances) push requests are failed with 503 error status and Request was aborted after waiting too long to attempt to service your request. log message. The execution time for this request varies from 10 seconds (for most of the requests) up to 30 seconds for some of them.
According to this article https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#instance_scaling Deadlines for HTTP request is 24 hours and request should not be aborted in 10 seconds.
Is there a way to avoid such exceptions?
These failed requests are most likely timing out in the Pending Request Queue, meaning that no instances are available to serve them. This usually happens during spikes of PubSub messages that are delivered in burst and App Engine can't scale up quickly enough to cope with them.
One option to mitigate this would be to switch the scaling option to automatic scaling in your app.yaml file. You can tweak the min_pending_latency and max_pending_latency to better fit your scenario. You can also specify min_idle_instances to get idle instances that would be ready to handle extra load (make sure to also enable and handle warmup requests)
Take into account though that PubSub will automatically retry to deliver failed messages. It will adjust the delivery rate according to your system's behavior, as documented here. So you may experience some errors during spikes of messages, while new instances are being spawned, but your messages will eventually be processed (as long as you have setup max_instances high enough to handle the load).
I have a small hobby project using Google Cloud's Natural Language Processing API. I also made sure to set up a daily budget for the project of just $2.00 USD.
My questions is: what happens when/if the spending limit is reached? Does the API cease working to prevent any further queries to the API? Basically, does having a spending limit prevent me from having to worry about additional charges to the project in question?
Thanks!
Yes, if your daily spending limit is hit, services that cost money will cease to function until the limit resets.
See When a resource is depleted for details:
For resources that are required to initiate a request, when the resource is depleted, App Engine by default returns an HTTP 403 or 503 error code for the request instead of calling a request handler.
For all other resources, when the resource is depleted, an attempt in the application to consume the resource results in an exception. This exception can be caught by the application and handled, such as by displaying a friendly error message to the user.
In the Python API, this exception is apiproxy_errors.OverQuotaError.
In the API for Java, this exception is com.google.apphosting.api.ApiProxy.OverQuotaException.
In the Go API, the appengine.IsOverQuota function reports whether an error represents an API call failure due to insufficient available quota.
What is the maximum timeout deadline for a URL Fetch on Google App Engine?
I understand that for normal requests, it can be no more than 60 seconds (which is the maximum length of the request). But what about backend requests or Taskqueues which can run up to ten minutes? There are a number of questions on this topic with conflicting information. The official documentation is silent on the issue. Any ideas?
60 seconds for both of them. It's specified in Java docs here.
You may use sockets if you want longer deadline, but as I remember you cannot use HTTPS there.
I am using a cron job to download a web page and save it, using Google App Engine.
After 5 seconds I get a Dealine exceeded error.
How can I avoid the error. Searching around this site, I can extend the time limit for urlfetch. But it isn't urlfetch that is causing issues. It is the fact that I can't run the task for over 5 seconds.
For example, I have tried this fix, but it only works if I am running the page myself, not via a cron job:
HTTPException: Deadline exceeded while waiting for HTTP response from URL: #deadline
Cron jobs can run for a maximum of 10 minutes, but that doesn't mean the URLFetch can't timeout before then. The default timeout for URLFetch is exactly 5 seconds but you can raise this.
"You can set a deadline for a request, the most amount of time the service will wait for a response. By default, the deadline for a fetch is 5 seconds. The maximum deadline is 60 seconds for HTTP requests and 10 minutes for task queue and cron job requests. When using the URLConnection interface, the service uses the connection timeout (setConnectTimeout()) plus the read timeout (setReadTimeout()) as the deadline."
see: https://developers.google.com/appengine/docs/java/urlfetch/?csw=1#Requests
"A cron job will invoke a URL, using an HTTP GET request, at a given time of day. An HTTP request invoked by cron can run for up to 10 minutes, but is subject to the same limits as other HTTP requests."
see: https://developers.google.com/appengine/docs/python/config/cron
Furthermore check out this older stackoverflow question with very similar issue.
If Google App Engine cron jobs have a 10 minute limit, then why do I get a DeadlineExceededError after the normal 30 seconds?
I have configured Google App Engine to record exception with ereporter.
The cron job is configured to run every 59 minutes. The cron.yaml is as follows
cron:
- description: Daily exception report
url: /_ereporter?sender=xxx.xxx#gmail.com # The sender must be an app admin.
schedule: every 59 minutes
How to do I run this immediately.
What I am trying to do here is simulate a 500 HTTP error and see the stack trace delivered immediately via the cron job.
Just go to the URL from your browser.
You can't using cron. Cron is a scheduling system, you could get it to run every minute.
Alternately you could wrap your entire handler in a try/except block and try to catch everything. (You can do this for some DeadlineExceededErrors for instance) then fire off a task which invokes ereporter handler, and then re-raise the Exception.
However in many cases Google infrastructure can be the cause of the Error 500 and you won't be able to catch the error. To be honest you are only likely to be able to cause an email sent for a subset of all possible Error 500's. The most reliable way probably be to have a process continuously monitor the logs, and email from there.
Mind you email isn't consider reliable or fast so a 1 min cron cycle is probably fast enough.
I came across this thread as I was trying to do this as well. A (hacky) solution I found was to add a curl command at the end of my cloudbuild.yaml file that triggers the file immediately per this thread. Hope this helps!
Make a curl request in Cloud Build CI/CD pipeline