I am using a cron job to download a web page and save it, using Google App Engine.
After 5 seconds I get a Dealine exceeded error.
How can I avoid the error. Searching around this site, I can extend the time limit for urlfetch. But it isn't urlfetch that is causing issues. It is the fact that I can't run the task for over 5 seconds.
For example, I have tried this fix, but it only works if I am running the page myself, not via a cron job:
HTTPException: Deadline exceeded while waiting for HTTP response from URL: #deadline
Cron jobs can run for a maximum of 10 minutes, but that doesn't mean the URLFetch can't timeout before then. The default timeout for URLFetch is exactly 5 seconds but you can raise this.
"You can set a deadline for a request, the most amount of time the service will wait for a response. By default, the deadline for a fetch is 5 seconds. The maximum deadline is 60 seconds for HTTP requests and 10 minutes for task queue and cron job requests. When using the URLConnection interface, the service uses the connection timeout (setConnectTimeout()) plus the read timeout (setReadTimeout()) as the deadline."
see: https://developers.google.com/appengine/docs/java/urlfetch/?csw=1#Requests
"A cron job will invoke a URL, using an HTTP GET request, at a given time of day. An HTTP request invoked by cron can run for up to 10 minutes, but is subject to the same limits as other HTTP requests."
see: https://developers.google.com/appengine/docs/python/config/cron
Furthermore check out this older stackoverflow question with very similar issue.
If Google App Engine cron jobs have a 10 minute limit, then why do I get a DeadlineExceededError after the normal 30 seconds?
Related
I want to migrate from App Engine Cron jobs to Cloud Scheduler, but in Cloud Scheduler the request deadline timeout is 60 seconds, not the 10 minutes that has the requests from Cron jobs.
Is there a way to configure Cloud Scheduler App Engine request's to have a deadline timeout of 10 minutes?
This will set the deadline for a job to 30mins. Which is the max for HTTP targets.
gcloud beta scheduler jobs update http <job> --attempt-deadline=1800s --project <project>
The allowed duration for this deadline is: For HTTP targets, between 15 seconds and 30 minutes. For App Engine HTTP targets, between 15 seconds and 24 hours.
A duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s".
Source: https://cloud.google.com/scheduler/docs/reference/rest/v1/projects.locations.jobs#Job
According to their scheduler.v1beta1, it is possible to set that Deadline using the attemptDeadline.
The deadline for job attempts. If the request handler does not respond by this deadline then the request is cancelled and the attempt is marked as a DEADLINE_EXCEEDED failure. The failed attempt can be viewed in execution logs. Cloud Scheduler will retry the job according to the RetryConfig.
The allowed duration for this deadline is:
For HTTP targets, between 15 seconds and 30 minutes.
For App Engine HTTP targets, between 15 seconds and 24 hours.
For PubSub targets, this field is ignored.
https://cloud.google.com/nodejs/docs/reference/scheduler/0.3.x/google.cloud.scheduler.v1beta1#.Job
When we look at Cloud Scheduler, we see that when the time is reached to fire a job the request to fire that job may fail. At this point, the request will be retried based on the configuration of that job ... see:
https://cloud.google.com/sdk/gcloud/reference/beta/scheduler/jobs/create/http
Among these settings we find:
--max-backoff
--max-doublings
--max-retry-attempts
--max-retry-duration
--min-backoff
It seems that if we want to keep trying for a solid 10 minutes we might be able to specify:
--max-backoff: 0s
--max-doublings: 0
--max-retry-attempts: 0
--max-retry-duration: 10m
--min-backoff: 0s
What is the maximum timeout deadline for a URL Fetch on Google App Engine?
I understand that for normal requests, it can be no more than 60 seconds (which is the maximum length of the request). But what about backend requests or Taskqueues which can run up to ten minutes? There are a number of questions on this topic with conflicting information. The official documentation is silent on the issue. Any ideas?
60 seconds for both of them. It's specified in Java docs here.
You may use sockets if you want longer deadline, but as I remember you cannot use HTTPS there.
I am trying to fetch a large amount of data from an remote server,for that i am sending a request from GAE ,As a response getting a large amount of data in an SOAP Format XML ,i want to know How much data we can receive with in 60 secs In Google App engine ?
The response size should be less than 32MB and within 60 sec. Don't forget that if you're going to hit any of these limits, your application will raise a timeout or something, which is not good.
Most likely you should use the Task Queue API or the deferred library to run these tasks in the background and to make sure that the user won't wait 60 seconds before it will time out.
I am connecting to a 3rd party server on the following wsdl
http://webservices.ticketvala.com/axis2/services/WSTicketvala?wsdl
I am using JAX-WS to generate client code and call relevent method on 3rd party server. 3rd party server may take time between 15-25 seconds to send response.
It works fine on tomcat.
Now when i deploy this to GAE 1.5.3, often i get ScocketTimeoutException in less than 10 seconds. Sometimes it is succesfull taking even 20 seconds. I want to know why it fails many times. And any workaround to increase this response deadline time / to avoid this ScoketTimeOutException forever.
Similarly,
I have another RESTfull service at http://ticketgoose.com/bookbustickets/TGWSStationNameResponseAction.do?" +
"event=getStationDetails&password=123456&userId=ctshubws
I am connecting it through java.net.URL and many times i get TimeoutException. How can i raise this timeout limit to more than 30 seconds?
Thanks
Deepak
No user-initiated request can take more than 30 seconds to complete in Google App Engine:
http://code.google.com/intl/en/appengine/docs/java/runtime.html#The_Request_Timer
And a HTTP request to a external URL in a user-initiated request can't take more than 10 seconds to complete:
http://code.google.com/intl/en/appengine/docs/java/urlfetch/overview.html#Requests
If you need to do the overall work in more than 30 seconds and can do it in background (not needing to return the response directly via HTTP), use Task Queues. Note that the simplest way to do background work with tasks is to use the DeferredTask. And the best part: HTTP requests to external URLs on tasks can take up to 10 minutes to complete.
Google App Engine imposes a 30 seconds timeout for every request. Is this applicable when uploading to the blobstore?
Thanks!
No. Even with regular requests to your app, the 30 second limit only applies to time your application spends processing requests. The timer does not include time the server takes to receive the request nor time spent sending the response back to the user.