Consider a web service, for instance, where user can make an API request to start a task at certain scheduled time. Task definition and scheduled time are persisted in a database.
First approach I came up with is to start a Go timer and wait for the timer to expire in a Goroutine (not blocking the request). This goroutine, after time expiration, will also fire another API request to start executing the task.
Now the problem arises when this service is redeployed. For zero downtime deployment I am using Einhorn with goji. After code reload, obviously both timer goroutine and timer-expiration-handler goroutine dies. Is there any way to recover Go timer after code reload?
Another problem I am struggling with is to allow the user to interrupt the timer (once its started). Go timer has Stop to facilitate this. But since this is a stateless API, when the \interrupt request comes in service doesn't have context of timer channel. And it seems its not possible to marshal the channel (returned from NewTimer) to disk/db.
Its also very well possible that I am not looking at the problem from correct perspective. Any suggestions would be highly appreciated.
One approach that's commonly used is to schedule the task outside your app, for example using crontab or systemd timers.
For example using crontab:
# run every 30 minutes
*/30 * * * * /usr/bin/curl --head http://localhost/cron?key=something-to-verify-local-job >/dev/null 2>&1
Using an external task queue is also a valid option like #Not_a_Golfer mentioned but more complicated.
Related
I'm working with Go on App Engine and I'm trying to build an API that needs to perform a long-running background task - in this case it needs to parse and chunk a big file out to task queues. I'd like it to return a 200 and close the user connection immediately and let the process keep running in the background until it's complete (this could take 5-10 minutes). Task queues alone don't really work for my use case because parsing the initial file can take more than the time limit for an API request.
At first I tried a Go routine as a solution for this problem. This failed because my app engine context expired as soon as the parent function closed the user connection. (I suppose I could try writing a go routine that doesn't require a context, but then I'd lose logging and I'd need to fetch the entire remote file and pass it to the go routine.)
Looking through the docs, it looks like App Engine used to have functionality to support exactly what I want to do: [runtime.RunInBackground], but that functionality is now deprecated and the replacement isn't obvious.
Is there a "right" or recommended way to do background processing now?
I suppose I could put a link to my big file into a task queue, but if I understand correctly, even functions called through task queues have to complete execution within a specified amount of time (is it 90 seconds?) I need to be able to run longer than that.
Thanks for any help.
try using:
appengine.BackgroundContext()
it should be long-lived but will only work on GAE Flex
I created a custom review and approve activiti workflow. I need to start this workflow automatically in every certain period , say every 30 minutes. For this i used timer start event as below:
<startEvent id="timerStart" name="Timer start" activiti:formKey="scheduledtask:submitParallelReviewTask">
<timerEventDefinition>
<timeCycle>R5/PT30M</timeCycle>
</timerEventDefinition>
</startEvent>
This created new process instance in every 30 minutes and repetition occured 5 times as required. But in the new timer started processes' tasks , initiator and other process variables were null. Also if i set the process variables as mandatory, timer executor job failed.
How can i set the initiator and other mandatory process variables in the newly created i.e. timer auto started process instances and its respective tasks?
Please suggest how to fix these bugs.
Thank you in advance !
Well I guess for the solution you should use cron job of alfresco.
For using cron job you can find below link usefull.
https://wiki.alfresco.com/wiki/Scheduled_Actions
Use workflowService for setting parameters.
My goal is to receive updates for some service (using http request-response) all the time, and when I get a specific information, I want to push it to the users. This code must run all the time (let's say every 5 seconds).
How can I run some code doing this when the server is up (means, not by an http request that initiates the execution of this code) ?
I'm using Java.
Thanks
You need to use
Scheduled Tasks With Cron for Java
You can set your own schedule (e.g. every minute), and it will call a specified handler for you.
You may also want to look at
App Engine Modules in Java
before you implement your code. You may separate your user-facing and backend code into different modules with different scaling options.
UPDATE:
A great comment from #tx802:
I think 1 minute is the highest frequency you can achieve on App Engine, but you can use cron to put 12 tasks on a Push Queue, either with delays of 5s, 10s, ... 55s using TaskOptions.countdownMillis() or with a processing rate of 1/5 sec.
I have installed PostgreSQL 8.4. What I want to do is call a web service through a C function, enabled by an insert/update trigger and pass the NEW values in this webservice. How can I do that, I searched the web and couldn't find an example.
Thanks in advance.
Please don't do this. If you do you will have wonderful questions like how you handle the web service being down. Also you will have to address what happens when your application rolls back. You can't uncall the web service. Also if the connection times out, your procedure will hang for quite a bit of time (retaining all locks etc) while waiting for the response which never comes.
A better approach is to use a queuing solution like pgq or pg_message_queue and queue up the data at trigger time, only to run it against the web service asynchronously.
I want to run a script every few seconds (4 or less) in google app engine to process user input and generate output. What is the best way to do this?
Run a cron job.
http://code.google.com/appengine/docs/python/config/cron.html
http://code.google.com/appengine/docs/java/config/cron.html
A cron job will invoke a URL at a
given time of day. A URL invoked by
cron is subject to the same limits and
quotas as a normal HTTP request,
including the request time limit.
.
Also consider the Task Queue - http://code.google.com/appengine/docs/python/taskqueue/overview.html
Reconsider what you're doing. As Ash Kim says, you can do it with the task queue, but first take a close look if you really need to run a process like this. Is it possible to rewrite things so the task runs only when needed, or immediately, or lazily (that is, only when the results are needed)?