I'm trying to build a timer queue in C to save the jobs that need to be processed based on time, is priority queue the best thing here or is there something else I need to look at?
The following is what I have,
job 1, start_time, time_to_exec, execute after 5 minutes
job 2, start_time, time_to_exec, execute after 5 minutes
Then new job came in
job 3, start_time, time_to_exec, execute after 1 minutes
...
and so on
and I'll be checking the first element every few microseconds.
start_time is the time before entering the queue.
time_to_exec is the time that will be used to insert the job into the queue
Related
How to set schedule as "every 1 minutes from every hours from HH:10 to HH:15"?
I want the task to be executed at a specific time every hour, but gcp doesn't seem to support it.
In a nutshell,I want to execute the task 5 times per hour.
https://cloud.google.com/appengine/docs/standard/python/config/cronref
"every 1 minutes from every hours from HH:10 to HH:15" not working.
"5 times per hour" would essentially mean every 12 minutes. Try:
schedule: every 12 minutes
If you are using Cloud Scheduler, you can specify a cron:
*/12 * * * *
For running its every 11-15th every hour being 10-15 hours, try using the cron:
11-15 10-15 * * *
I have a flink job with a global window and custom process.
The Process is failed after ~10 minutes on the next error:
java.io.InterruptedIOException
This is my job:
SingleOutputStreamOperator<CustomEntry> result = stream
.keyBy(r -> r.getId())
.window(GlobalWindows.create())
.trigger(new CustomTriggeringFunction())
.process(new CustomProcessingFunction());
The CustomProcessingFunction is run for a long time (more then 10 minutes), but after 10 minutes, the process is stoped and failed on InterruptedIOException
Is it possible t increase the timeout of flink job?
From Flink's point of view, that's an unreasonably long period of time for a user function to run. What is this window process function doing that takes more than 10 minutes? Perhaps you can restructure this to use the async i/o operator instead, so you aren't completely blocking the pipeline.
That said, 10 minutes is the default checkpoint timeout interval, and you're preventing checkpoints from being able to complete while this function is running. So you could experiment with increasing execution.checkpointing.timeout.
If the job is failing because checkpoints are timing out, that will help. Or you could increase execution.checkpointing.tolerable-failed-checkpoints from its default (0).
Consider I have a cron job running in app engine as below,
every 10 minutes from 00:00 to 02:00
My doubt is, If the task takes 5 mins to complete, Will the next task gets executed on the 15th minute or in the 10th minute?
My requirement is as below.
Task #1 starts at 00:00, runs for 4 mins.
Task #2 starts at 00:10, runs for 1 min.
Task #3 starts at 00:20, runs for 3 mins and so on!
Thanks,
Karthick.
You can find an answer in the documentation:
By default, an interval schedule starts the next interval after the last job has completed. If a from...to clause is specified, however, the jobs are scheduled at regular intervals independent of when the last job completed. For example:
every 2 hours from 10:00 to 14:00
This schedule runs the job three times per day at 10:00, 12:00, and 14:00, regardless of how long it takes to complete.
In Task Queues code is executed to connect to the server side
through URL Fetch.
My file queue.yaml.
queue:
- Name: default
rate: 10 / m
bucket_size: 1
In such settings, Tusk performed all at once, simultaneously.
Specificity is that between the requests should be delayed at least 5
sec. Task must perform on stage with a difference> 5 sec. (but
does not parallel).
What are the values set in queue.yaml?
You can't specify minimum delays between tasks in queue.yaml, currently; you should do it (partly) in your own code. For example, if you specify a bucket size of 1 (so that more than one task should never be executing at once) and make sure the tasks runs for at least 5 seconds (get a start=time.time() at the start, time.sleep(time.time()-(5+start)) at the end) this should work. If it doesn't, have each task record in the store the timestamp it finished, and when it start check if the last task ended less than 5 seconds ago, and in that case terminate immediately.
The other way could be store the task data in table. In your task-queue add a id parameter. Fetch 1st task from table and pass its id to task queue processing servlet. In servlet at the end delay for 5 second and feth next task, pass its id and.... so on.
I love the new TaskQueue API.
I have a question about the ETA/Countdown, if I set it a new Task to execute 10 minutes in the future and it is the only item in the queue - will it execute in roughly 10 minutes or will it execute straight away?
It will execute no sooner than 10 minutes from now (it may execute later if the queue is full, naturally).