I want to migrate from App Engine Cron jobs to Cloud Scheduler, but in Cloud Scheduler the request deadline timeout is 60 seconds, not the 10 minutes that has the requests from Cron jobs.
Is there a way to configure Cloud Scheduler App Engine request's to have a deadline timeout of 10 minutes?
This will set the deadline for a job to 30mins. Which is the max for HTTP targets.
gcloud beta scheduler jobs update http <job> --attempt-deadline=1800s --project <project>
The allowed duration for this deadline is: For HTTP targets, between 15 seconds and 30 minutes. For App Engine HTTP targets, between 15 seconds and 24 hours.
A duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s".
Source: https://cloud.google.com/scheduler/docs/reference/rest/v1/projects.locations.jobs#Job
According to their scheduler.v1beta1, it is possible to set that Deadline using the attemptDeadline.
The deadline for job attempts. If the request handler does not respond by this deadline then the request is cancelled and the attempt is marked as a DEADLINE_EXCEEDED failure. The failed attempt can be viewed in execution logs. Cloud Scheduler will retry the job according to the RetryConfig.
The allowed duration for this deadline is:
For HTTP targets, between 15 seconds and 30 minutes.
For App Engine HTTP targets, between 15 seconds and 24 hours.
For PubSub targets, this field is ignored.
https://cloud.google.com/nodejs/docs/reference/scheduler/0.3.x/google.cloud.scheduler.v1beta1#.Job
When we look at Cloud Scheduler, we see that when the time is reached to fire a job the request to fire that job may fail. At this point, the request will be retried based on the configuration of that job ... see:
https://cloud.google.com/sdk/gcloud/reference/beta/scheduler/jobs/create/http
Among these settings we find:
--max-backoff
--max-doublings
--max-retry-attempts
--max-retry-duration
--min-backoff
It seems that if we want to keep trying for a solid 10 minutes we might be able to specify:
--max-backoff: 0s
--max-doublings: 0
--max-retry-attempts: 0
--max-retry-duration: 10m
--min-backoff: 0s
Related
Google describes basic scaling like this:
I don't really have any other choice, as I'm using a B1 instance, so automatic scaling is not allowed.
That raises the question though, if I have an endpoint that takes a variable amount of time (could be minutes, could be hours), and I basically have to set an idle_timeout, will App Engine calculate idle_timeout from the time the request was made in the first place or when the app is done handling the request?
If the former is right, then it feels a bit unfair to have to guess how long requests will take when thread activity is a usable indicator of whether or not to initiate shutdown of an app.
Here you are mixing up two different terms.
idle_timeout is the time that the instance will wait before shutting down after receiving its last request
Request timeout is the amount of time that App Engine will wait for the return of a request from your app
As per the documentation:
Requests can run for up to 24 hours. A manually-scaled instance can
choose to handle /_ah/start and execute a program or script for many
hours without returning an HTTP response code. Task queue tasks can
run up to 24 hours.
I am using Pub/Sub push subscription, ack deadline is set to 10 minutes, the push endpoint is hosted within AppEngine using basic scaling.
In my logs, I see that some of the Pub/Sub (supposedly delivered to starting instances) push requests are failed with 503 error status and Request was aborted after waiting too long to attempt to service your request. log message. The execution time for this request varies from 10 seconds (for most of the requests) up to 30 seconds for some of them.
According to this article https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#instance_scaling Deadlines for HTTP request is 24 hours and request should not be aborted in 10 seconds.
Is there a way to avoid such exceptions?
These failed requests are most likely timing out in the Pending Request Queue, meaning that no instances are available to serve them. This usually happens during spikes of PubSub messages that are delivered in burst and App Engine can't scale up quickly enough to cope with them.
One option to mitigate this would be to switch the scaling option to automatic scaling in your app.yaml file. You can tweak the min_pending_latency and max_pending_latency to better fit your scenario. You can also specify min_idle_instances to get idle instances that would be ready to handle extra load (make sure to also enable and handle warmup requests)
Take into account though that PubSub will automatically retry to deliver failed messages. It will adjust the delivery rate according to your system's behavior, as documented here. So you may experience some errors during spikes of messages, while new instances are being spawned, but your messages will eventually be processed (as long as you have setup max_instances high enough to handle the load).
When setting up a Google App Engine instance you can configure a cron.yaml to set up Cron jobs.
There does not seem to be any documentation on how to configure jobs that run say every 30 seconds.
I tried
schedule: every 30 seconds
and
schedule: 0/30 0 0 ? * * *
But no good. Google Cloud tells me the format is incorrect when I deploy. Can you schedule in frequencies less then 1 minute with Google App Engine Cron jobs?
You cannot configure GAE cron services with resolutions below 1 minute. FWIW, you can't do that on unix/linux systems either.
But it is possible to use an every 1 minutes cron job from which you can further trigger delayed execution of deferred/push/pull queue tasks with down to 1 second resolution, see High frequency data refresh with Google App Engine
Had the same problem, solved with setTimeout().
Using setTimeout for 30 seconds inside the appengine one-minute cron job did the trick, in this case we'll be fetching data 2 times per minute that is every 30 seconds.
saveData();
setTimeout(function () { saveData(); }, 30000);
Tested and worked fine,
I am using a cron job to download a web page and save it, using Google App Engine.
After 5 seconds I get a Dealine exceeded error.
How can I avoid the error. Searching around this site, I can extend the time limit for urlfetch. But it isn't urlfetch that is causing issues. It is the fact that I can't run the task for over 5 seconds.
For example, I have tried this fix, but it only works if I am running the page myself, not via a cron job:
HTTPException: Deadline exceeded while waiting for HTTP response from URL: #deadline
Cron jobs can run for a maximum of 10 minutes, but that doesn't mean the URLFetch can't timeout before then. The default timeout for URLFetch is exactly 5 seconds but you can raise this.
"You can set a deadline for a request, the most amount of time the service will wait for a response. By default, the deadline for a fetch is 5 seconds. The maximum deadline is 60 seconds for HTTP requests and 10 minutes for task queue and cron job requests. When using the URLConnection interface, the service uses the connection timeout (setConnectTimeout()) plus the read timeout (setReadTimeout()) as the deadline."
see: https://developers.google.com/appengine/docs/java/urlfetch/?csw=1#Requests
"A cron job will invoke a URL, using an HTTP GET request, at a given time of day. An HTTP request invoked by cron can run for up to 10 minutes, but is subject to the same limits as other HTTP requests."
see: https://developers.google.com/appengine/docs/python/config/cron
Furthermore check out this older stackoverflow question with very similar issue.
If Google App Engine cron jobs have a 10 minute limit, then why do I get a DeadlineExceededError after the normal 30 seconds?
Google App Engine imposes a 30 seconds timeout for every request. Is this applicable when uploading to the blobstore?
Thanks!
No. Even with regular requests to your app, the 30 second limit only applies to time your application spends processing requests. The timer does not include time the server takes to receive the request nor time spent sending the response back to the user.