How can one run multiple instances of a Script Using Google App Engine's Cron system?
By default, it will run, then wait the specified interval before running again, which means that only one instance runs. What i am looking for is how one can get a script that takes 2+ minutes to run start a new instance every 30-60 seconds regardless of if it is running already or not, which does assume the script does not interfere with itself if multiple instances are running. this would effectively allow the script to deal with several times more information in the same period of time.
Edit, Completely reworded the question.
You only get resolution to the minute. To get finer-grained, you'll need instances that know whether they should handle the request from chron immediately, of if they'll have to sleep 30 seconds first. A 30 second sleep uses up half of the 60 second request deadline. Depending on the workload you expect to handle, this might require that you use Modules.
By the way, I'm not aware of any guarantee that a job scheduled for 01:00 will fire at exactly 01:00:00 (and not at, say, 01:00:03).
Since the cron service doesn't allow intervals below 1 min you'd need to achieve staggering script launching in a different manner.
One possibility would be to have a cron entry handler running every 2 mins which internally sleeps for 30 seconds (or as low as your "few seconds of each-other" requirements are) between triggering the respective script instance launches.
Note: the sleeps would probably burn into your Instance Hours usage. You might be able to incorporate the staggered triggering logic into some other long-living task you may have instead of simply sleeping.
To decouple the actual script execution from the cron handler (or the other long-living task) execution you could use dedicated task queues for each script instance, with queue handlers sharing the actual script code if needed. The actual triggering would be done by enqueueing tasks in the respective script instance queue. As a bonus you may further control each script instance executions by customizing the respective queue configuration.
Note: if your script execution time exceeds the 2 minutes cron period you may need to take extra precautions in the queue configurations as there can be extra delays (due to queueing) which could push lauching of the respective script instance closer to the next instance launch.
Working off Dave W. Smith's answer, The Line would be
every 1 minute from 00:00 to 23:59
Which means that it would create a new instance every minute, even if the script takes longer than a minute to run. It does seem that specifying seconds is not possible.
Related
I have an script in App engine that gets called every 10min. I am the only user.
The script pulls data from a web source does light processing and returns an image. It takes several minutes to run the first time. The source gets updated every 10min, so the next time my script runs, (10min later), it returns in a few seconds.
I'm using over 30 instance hours a day which is over the 28 free hours.
I read somewhere that every time an instance starts, it uses a minimum of 15min. (so 144x15=36hrs)
Therefore, am I better off trying to keep the instance running 24hrs (using up 24hrs) and limiting to one instance max? Perhaps setting idle_timeout to 10min. Another potential way to save would be to somehow pause my script during late night/early morning hours.
This is where you likely found the 15 minutes minimum statement and it refers to when the accrual of instance hours ends. If you look at that documentation you will see that it depends on what type of scaling you are using.
Here are my tought in all options you mentioned on your question for staying in the free tier:
Use only one instance. The problem with this approach is that giving a higher load to a single instance may increase other cost such as vCPU and memory since all the processing will be done there, also, it's likely that it's going to take longer to run the script.
Pause the script on low activity hours, this will be the ideal if the acumulated load when turning it back on it not too big.
Setting idle_timeout to 10min. This won't work if it runs every 10 min on all instances, since your app runs every 10min and app engine will only stop charging you every 15min idle, which will never happen, unless it does not run every 10min for every instance, if that is the case, it could be worth trying.
So, summing it up, all 3 options have their pros and cons, I would suggest that you test all of them and see what options suits your needs better.
Hope this helped.
I am trying to figure out how to run a service(1) when it does not receive any calls.
I want to use Microservices Architecture.
Basically i want to run this service (1) when the other service(2) is receiving calls and all data.
As the service(1) i mentioned is not receiving it would not have to spawn new instances and i would want only the service(2) to scale.
I have noticed scheduling jobs with cron yaml but the number of calls is limited.
I need to get this service(1) to be active every 1 min when service(2) is active.
It's hard to give a good answer without knowing more about what service (1) has to do when it is 'active'. It sounds you want cron to launch a task every minute.
You can use cron in conjunction with push queues: https://cloud.google.com/appengine/docs/standard/go/taskqueue/push/
When creating a push queue task, you can set the property delay before adding it to the queue: https://cloud.google.com/appengine/docs/standard/go/taskqueue/reference#Task
(For me in Python they called it countdown https://cloud.google.com/appengine/docs/standard/python/refdocs/google.appengine.api.taskqueue.taskqueue#google.appengine.api.taskqueue.taskqueue.add)
You could have a cron job that fires every 24 hrs. That cron job would load up your push queue with tasks who's delays are staggered. The delay of the first one is 1 min, the delay of the second one is 2 min, etc.
Is it possible to start and then stop an agent by schedule, say run at 12 PM and then stop at 1 PM everyday. Thanks.
You have several options.
You can do as Craig suggests and start and stop it with a cron job. I don't like this idea as it goes against the purpose of an agent which should be always on.
Your agent can watch for device publishes to start appearing with timestamps that fall within the desired block of time (this is a common method).
Your agent can schedule devices for the desired block of time (if you are going to be sending control signals) and then subscribe to the schedule announce topic and watch for your schedule start.
You can schedule a periodic function call to be triggered at the start of the block.
Like 4 you can use schedule instead of periodic. You have to schedule the next event as schedule will only call the callback once, but it is useful for events that change dynamically during run time. The actuator agent uses this for publishing schedule states.
In 2-5 the agent will remain dormant for most of the time and only be become active during the allotted time.
One way is to set up a cron job that executes volttron-ctl start agent_uuid to start it...then another one that stops it.
The other way would involve just leaving the agent running and adding a periodic that can run every 24 hours, which would execute 24 hours from when you started the agent.
I want to develop an app on which a user can register for alerts( multiple) so that whenever the fare hits below some threshold, he gets a notification. Fares are fetched from a third party website.I want to do this on google app-engine.
Now from what i understand , i need a process running 24/7 which checks the fares at say intervals of 30 mins and send out a notification whenever it hits below the threshold. Probably the cron job of app-engine can be used for this task ? But at max 100 cron jobs can be scheduled, what would be the better way to this. Also having a process for each user would be wastage of resources, what would be better scheduling algorithms for higher efficiency ?
You want to schedule a single cron that runs every 30 minutes and throws an item onto a task queue. That single item on the task queue would then be able to go through all your users, and generate tasks to fetch whatever you need in the background again. Two important things:
You want the initial cron call to return as quickly as possible, as URLs have a 60 second deadline.
Split up any work into separate task queues to achieve above and also iterate through data sources and/or users.
Based on what you're explaining, you can use push task queues: https://cloud.google.com/appengine/docs/python/taskqueue/overview-push
I run an application that usually triggers a heavy update every 2 hours, by queuing heavy tasks, using the cron mecanism. This has been working well for months.
However, the december 16th of 2012, this url has been called (by user-agent AppEngine-Google) every few seconds between 15:17 and 15:51, launching hundreds of heavy tasks. This resulted in an explosion of my quotas and forced me to switch to the paid version of the application, in order for my website to stay alive.
Anybody having the same issue? Any idea of what happened and how I could avoid this problem in the future?
I had the same issue.
I don't have an answer, but I think this is a task queue problem.
I have 4 cron jobs and some task queue tasks piled up. Everything was normal until 14:05Z (16th 6:05 PST?).
At 14:06Z and 14:07Z, two of my cron jobs were called (at their scheduled times) and finished with 200. After that, AppEngine-Google started to call the same jobs few times per every minutes. The disorder vanished after 14:50Z and no issues right now.
During the period, one of my task queue tasks was called at 14:11Z and finished with 503 (this was an expected failure). The task was scheduled to retry some hours later, but was called hundreds of times in an hour. The task's retry count was not incremented.
My guess is that something went wrong in task queues ("__cron" and, for me, "default") and the tasks were not removed until 14:50Z.
My app's App ID is vidssage.