I have a bunch of data to monitor. My data are statistics that can only be retrieved every hour but can change every second and I want to store into a database as much values as I can for each data set.
I've though about several approaches for this problem and I finally chose to refresh and read all statistics at once instead of reading them independently.
So that, I came out with command mycommand which reads all my statics with the cost of several minutes (let's say 30) of execution. Now I would like to run this script every hour, but taking the script execution into account.
I actually run
* */1 * * * mycommand.sh
and receive many annoying error emails (actually one every hour) and I effectly retrieve my statistics every 2 hours.
1h 30 minutes is the half of 3 hours. So you could have two entries in crontab(5) running the same /home/gogaz/mycommand.sh script, one to run it at 1, 4, 7, ... hours (every 3 hours from 1am) and another to run it at 2:30, 5:30, 8:30 hours, ... (every 3 hours from 2:30am) etc
Writing these entries is left as an exercise to the reader.
See also anacrontab(5) and at(1). For example, you might run your script once using batch, but terminate your script with an at command rescheduling that same script (the drawback is handling of unexpected errors).
If you redirect your stdout and stderr in your crontab entry, you won't get any emails.
Related
I have an AppleScript that runs on loop every two hours to modify a calendar B based on updates from another calendar A.
The script uses the on idle command below to wait 2 hours every loop. What happens if the computer stays idle for 1.5 hours then goes to sleep for 10 hours? Will there be 0.5 hours left when it wakes up? Any other scenarios?
on idle
my_code()
return (120 * minutes)
end idle
The script truly only needs to run if there is an update to calendar A, which is a shared iCloud calendar and can get updates from multiple people. The two hour loop is what I could figure out so far but I feel it is not efficient. Any more robust suggestions? Is there a way I can trigger the script to run only when it detects an update in calendar A? Or, along the same line of thought, is there a way to get the last timestamp the calendar was updated?
Thanks
I can't test following. Not sure it is the best way to solve your problem. Try yourself:
property oldStampDates : {}
on run
tell application "Calendar" to tell calendar "Test Calendar" to set oldStampDates to get stamp date of events
end run
on idle
--> Script retrieves last modified date and time of indicated calendar events.
tell application "Calendar" to tell calendar "Test Calendar" to set newStampDates to get stamp date of events
if newStampDates is not oldStampDates then display notification "The changes was detected"
set oldStampDates to newStampDates
return 30 -- seconds, default setting
end idle
NOTE: 1) you can put instead of display notification call to your handler my_code(), 2) you can put instead of 30 seconds other value, for example, return 10 (checking every 10 seconds).
I'm using Gatling to run loadtest.
I've put all the queries (3million queries in total) in a log file and load it into a feed
val feeder = tsv(logFileLocation).circular
val search =
feed(feeder)
.exec(
http("/")
.get("${params}")
.headers(sentHeaders)
).pause(1)
As for the simulation, I want 50 users concurrently and peak request at 50/seconds so I set up this way
setUp(myservicetest.inject(atOnceUsers(50))).throttle(
reachRps(50) in (40 minutes), jumpToRps(20), holdFor(20 minutes)).maxDuration(60 minutes).protocols(httpProtocol)
My assumption is these 50 users each loads the queries and starts from fisr to last queries. Because there are enough queries to execute, these 50 users will always stay on line for the whole duration (60 mins)
But when I ran it, I saw user1 runs query1, user2 runs query2, .. user50 runs query50. Every user just runs 1 query and then quit. So exactly 50 queries were executed in the loadtest and it finished quickly.
SO my question is, say I have 3 million queries in tsv(logFileLocation).circular and multiple users, will each user starts from query1 and try to execute all 3m queries. Or each user is scheduled to run part of the 3m queries and if enough time is allocated, at the end of the test all 3m queries are executed for just once?
Thanks
Disclaimer: Gatling's author here
The latter: "at the end of the test all 3m queries are executed for just once".
Each virtual user performs its scenario.
Injection profiles only controls when new virtual users are started.
If you want virtual users to perform more than one request, you have to include that in your scenario, eg with loops and pauses.
Those are Gatling basics, I recommend you have a look at the new Gatling Academy courses we've just launched.
I am working with a path of jobs on autosys that runs every night.
One of these jobs needs to run only on certain dates at the beginning of each month (usually the first 4 days but it can change depending on the business) so currently I'm putting said job ON_ICE or OFF_ICE manually and I'm looking to automate this.
I currently have 2 ideas but I'm stuck either way I choose.
Option 1 (the cleanest ?)
I would create a new job in between daily_job and monthly_job, called let's say calendar_check.
this job would start a batch on my app serveur, checking the database of my app where the calendar is, and depending of the database check sending back to autosys the good command to put the next job ON or OFF_ICE.
My batch looks like this
%My_SQLPATH% -S %My_SQL_SERVER% -d %My_SQL_DB -h-1 -W -Q "SQL Query that returns 1 or 0 depending on the calendar in my application" output.txt
set /P bEndMonth= < output.txt
echo %bEndMonth%
del output.txt
IF %bEndMonth% == 0 (start "somthing i don't know what 'sendevent -j ON_ICE -e monthly_job'") ELSE (start "somthing i don't know what 'sendevent -e OFF_ICE -j monthly_job'")
That last line is what I don't know how to write or if it's even possible to get back to my autosys server and use the sendevent command.
Little diagram for clarity
Option 2 (more messy but maybe easier)
I create 2 new jobs. One, on my path of jobs put monthly_job ON_ICE everyday.
Another one, not on my path of jobs, reads an autosys calendar and runs only on the calendar dates to put monthly_job OFF_ICE.
Downside is I must maintain another calender in autosys but it's minor.
But again I don't know the syntaxe for the jil to ask one job to put another job on or off ice.
Little diagram again for clarity
Any help is welcome or any other idea on how i could implement this.
Thank you !
For executing sendevent command in Autosys, the autosys cli package has to be installed, local variable to be declared and then login to the particular instance.
This could be verified by the Scheduling Admin/Middleware team if any.
If the days of the run at the start of the month is fixed, like first 5 days or first 5 working days, extended calendar can be considered.
Alternative way:
Since the condition to run/hold the job is based on the output of the SQL query from the database, we would use user defined exit code based on which the monthly job would trigger.
Step1: Make a script that would fetch the SQL query output and based on it we can define user exit codes.
LOGIN Database;
EXEC SQL Query;
IF %bEndMonth% == 0
THEN exit 0;
ELSE
exit 1;
Generally if the job exit code is > than 0, it would trigger an alarm for job failure, if required to suppress this, add the following attribute to this job:
max_exit_success:1
Now the job would be marked as success incase the exit is 0 or 1.
Considering the name of the job is Job_Cal_Check and is defined daily to run.
Step 2:
The monthly job/box would have no calendar and it would only trigger if the Job_Cal_Check job has exit 1. Add the following attribute
condition: exitcode (Job_Cal_Check) = 1
Edits: Fix Mis Run of successive jobs
Create an intermediate job to sleep for x seconds, calculate this time +-2 or 3 seconds taken for the monthly job to start after completion of the Job_Cal_Check.
The job flow would be:
Non Monthly Job
Previous Chain -> Job_Cal_Check -> Sleep_Job -> Followup_Jobs *(would wait only for Sleep_Job as the Monthly job would have completed status from the previous run)*
Monthly Job
Previous Chain -> Job_Cal_Check -> Sleep_Job + Monthly_Job *(both these jobs would we activated simultaneously)* -> Followup_Jobs *(would wait for both the jobs to complete)*
Job attributes as follows:
Sleep_Job:
condition: success (Job_Cal_Check)
Followup_Jobs
condition: success (Sleep_Job) AND success(Monthly_Job)
On days of Non monthly job,
Followup_Jobs would only wait for the Sleep_Job to complete.
On days of Monthly job,
Followup_Jobs would first wait for the Sleep_Job to complete by this time the Montlhy_Job would have Activated/In Running/Completed and upon completion, the job flow would continue.
Hope this helps, if anything more required do ask.
In Task Queues code is executed to connect to the server side
through URL Fetch.
My file queue.yaml.
queue:
- Name: default
rate: 10 / m
bucket_size: 1
In such settings, Tusk performed all at once, simultaneously.
Specificity is that between the requests should be delayed at least 5
sec. Task must perform on stage with a difference> 5 sec. (but
does not parallel).
What are the values set in queue.yaml?
You can't specify minimum delays between tasks in queue.yaml, currently; you should do it (partly) in your own code. For example, if you specify a bucket size of 1 (so that more than one task should never be executing at once) and make sure the tasks runs for at least 5 seconds (get a start=time.time() at the start, time.sleep(time.time()-(5+start)) at the end) this should work. If it doesn't, have each task record in the store the timestamp it finished, and when it start check if the last task ended less than 5 seconds ago, and in that case terminate immediately.
The other way could be store the task data in table. In your task-queue add a id parameter. Fetch 1st task from table and pass its id to task queue processing servlet. In servlet at the end delay for 5 second and feth next task, pass its id and.... so on.
We recently got some data back on a benchmarking test from a software vendor, and I think I'm missing something obvious.
If there were 17 transactions (I assume they mean successfully completed requests) per second, and 1500 of these requests could be served in 5 minutes, then how do I get the response time for a single user? Is this sort of thing even possible with benchmarking? I have a lot of other data from them, including apache config settings, but I'm not sure how to do all the math.
Given the server setup they sent, I want to know how I can deduce the user response time. I have looked at other similar benchmarking tests, but I'm having trouble measuring requests to response time. What other data do I need to provide here to get that?
If only 1500 of these can be served per 5 minutes then:
1500 / 5 = 300 transactions per min can be served
300 / 60 = 5 transactions per second can be served
so how are they getting 17 completed transactions per second? Last time I checked 5 < 17 !
This doesn't seem to fit. Or am I looking at it wrongly?
I presume be user response time, you mean the time it takes to serve a single transaction:
If they can serve 5 per second than it takes 200ms (1/5) per transaction
If they can serve 17 per second than it takes 59ms (1/17) per transaction
That is all we can tell from the given data. Perhaps clarify how many transactions are being done per second.