I love the new TaskQueue API.
I have a question about the ETA/Countdown, if I set it a new Task to execute 10 minutes in the future and it is the only item in the queue - will it execute in roughly 10 minutes or will it execute straight away?
It will execute no sooner than 10 minutes from now (it may execute later if the queue is full, naturally).
Related
I have a flink job with a global window and custom process.
The Process is failed after ~10 minutes on the next error:
java.io.InterruptedIOException
This is my job:
SingleOutputStreamOperator<CustomEntry> result = stream
.keyBy(r -> r.getId())
.window(GlobalWindows.create())
.trigger(new CustomTriggeringFunction())
.process(new CustomProcessingFunction());
The CustomProcessingFunction is run for a long time (more then 10 minutes), but after 10 minutes, the process is stoped and failed on InterruptedIOException
Is it possible t increase the timeout of flink job?
From Flink's point of view, that's an unreasonably long period of time for a user function to run. What is this window process function doing that takes more than 10 minutes? Perhaps you can restructure this to use the async i/o operator instead, so you aren't completely blocking the pipeline.
That said, 10 minutes is the default checkpoint timeout interval, and you're preventing checkpoints from being able to complete while this function is running. So you could experiment with increasing execution.checkpointing.timeout.
If the job is failing because checkpoints are timing out, that will help. Or you could increase execution.checkpointing.tolerable-failed-checkpoints from its default (0).
I would like to run a http request in thread group for jmeter. But it should be run per 40 minute. Therefore it should be run with frequency.
Also Im using jmeter 2.12 and jdk 1.7. So how can I configure this proceed and should I use any timer or anything?
Thread group
- Get token request
It should be run automatically every 40 minute.
Please advice me,
Your expectation is correct:
Thread Group (1 thread, loop count: forever or enough loops)
Get Token Request
Constant Timer (Thread delay: 2400000)
In this case Get Token Request method will be fired every 40 minutes (40 * 60 * 1000) where 40 is minutes, 60 is seconds per minute and 1000 is milliseconds per second
Take a look at How to Use Variables in Different Thread Groups guide to learn how to use fresh token in another Thread Group if needed.
I'm trying to build a timer queue in C to save the jobs that need to be processed based on time, is priority queue the best thing here or is there something else I need to look at?
The following is what I have,
job 1, start_time, time_to_exec, execute after 5 minutes
job 2, start_time, time_to_exec, execute after 5 minutes
Then new job came in
job 3, start_time, time_to_exec, execute after 1 minutes
...
and so on
and I'll be checking the first element every few microseconds.
start_time is the time before entering the queue.
time_to_exec is the time that will be used to insert the job into the queue
Google app engine task queues have configuration as (example)
<queue>
<name>mail-queue</name>
<rate>5/m</rate>
<bucket-size>10</bucket-size>
</queue>
Here, what does the 'bucket-size' mean? I could not find a comprehensive documentation about this in google app engine documentation.
Does specifying this as 10 means that if 100 tasks are queued at an instant only 10 of those will be put in the queue and rest will be ignored?
bucket-size is perfectly described here:
Limits the burstiness of the queue's processing, i.e. a higher bucket size allows bigger spikes in the queue's execution rate. For example, consider a queue with a rate of 5/s and a bucket size of 10. If that queue has been inactive for some time (allowing its "token bucket" to fill up), and 20 tasks are suddenly enqueued, it will be allowed to execute 10 tasks immediately. But in the following second, only 5 more tasks will be able to be executed because the token bucket has been depleted and is refilling at the specified rate of 5/s.
If no bucket_size is specified for a queue, the default value is 5.
For your case it means that if 100 messages are queued, only ten are directly being executed and another 5 every next minute. You won't loose any messages, but they will queue up if your bucket-size and rate is too low.
In Task Queues code is executed to connect to the server side
through URL Fetch.
My file queue.yaml.
queue:
- Name: default
rate: 10 / m
bucket_size: 1
In such settings, Tusk performed all at once, simultaneously.
Specificity is that between the requests should be delayed at least 5
sec. Task must perform on stage with a difference> 5 sec. (but
does not parallel).
What are the values set in queue.yaml?
You can't specify minimum delays between tasks in queue.yaml, currently; you should do it (partly) in your own code. For example, if you specify a bucket size of 1 (so that more than one task should never be executing at once) and make sure the tasks runs for at least 5 seconds (get a start=time.time() at the start, time.sleep(time.time()-(5+start)) at the end) this should work. If it doesn't, have each task record in the store the timestamp it finished, and when it start check if the last task ended less than 5 seconds ago, and in that case terminate immediately.
The other way could be store the task data in table. In your task-queue add a id parameter. Fetch 1st task from table and pass its id to task queue processing servlet. In servlet at the end delay for 5 second and feth next task, pass its id and.... so on.