I have an agent job set to run log backups every two hours from 2:00 AM to 11:59 PM (leaving a window for running a full or differential backup). A similar job is set up in every one of my 50 or so instances. I may be adding several hundred instances over time (we host SQL Servers for some of our customers). They all backup to the same SAN disk volume. This is causing latency issues and otherwise impacting performance.
I'd like to offset the job run times on each instance by 5 minutes, so that instance one would run the job at 2:00, 4:00, etc., instance two would run it at 2:05, 4:05, etc., instance three would run it at 2:10, 4:10, etc. and so on. If I offset the start time for the job on each instance (2:00 for instance one, 2:05 for instance two, 2:10 for instance three, etc.), can I reasonably expect that I will get my desired result of not having all the instances run the job at the same time?
If this is the same conversation we just had on twitter: when you tell SQL Server Agent to run every n minutes or every n hours, the next run is based on the start time, not the finish time. So if you set a job on instance 1 to run at 2:00 and run every 2 hours, the 2nd run will run at 4:00, whether the first run took 1 minute, 12 minutes, 45 minutes, etc.
There are some caveats:
there can be minor delays due to internal agent synchronization, but I've never seen this off by more than a few seconds
if the first run at 2:00 takes more than 2 hours (but less than 4 hours), the next time the job runs will be at 6:00 (the 4:00 run is skipped, it doesn't run at 4:10 or 4:20 to "catch up")
There was another suggestion to add a WAITFOR to offset the start time (and we should discard random WAITFOR, because that is probably not what you want - random <> unique). If you want to hard-code a different delay on each instance (1 minute, 2 minutes, etc.) then it is much more straightforward to do that with a schedule than by adding steps to all of your jobs. IMHO.
Perhaps you could setup a centralized DB that manages the "schedule" and have the jobs add/update a row when they run. This way each subsequent server can start the job that "polls" when it can start. This way any latency in the jobs will cause the others to wait so you don't have a disparity in your timings when one of the servers is thrown off.
Being a little paranoid I'd add a catchall scenario that says after "x" minutes of waiting proceed anyway so that a delay doesn't cascade far enough that the jobs don't run.
Related
I am trying to figure out the issue.
we have a couple of sql job agents that runs less than 15 seconds everyday to execute store procedures.
Whenever I tried to see the job history, there is no information about it and it shows "never executed"
I did a simple test and tried to let it run say more than 60 seconds. I see it appears on the job history.
Is there any settings that I need to set so that I could see the jobs in job history that runs less than 15 seconds.
Thanks.
We currently run scheduled overnight jobs to sync heavily calculated data into flat tables for use in reports. These processes can take anywhere between 5mins to 2hrs per database, depending on the size. This has been working fine for a long time.
The need we now have is to try to keep the data as up-to-date as possible with our current setup.
I wrote a sync routine that can sync the specific users' data as it gets modified.
The short version is a trigger inserts the userids into a holding table when their records get modified, and there is a job that runs every 10 seconds that checks that table and if a userid is found it then fires the sync for that user and updates the flat table (and then flags the record as complete). This can take anywhere between instant and ~1min depending again on how much data it needs to calculate. Far better than the 24hrs it used to be.
Now onto the 'problem'.
We have upwards of 30 databases that all need to be checked.
Currently, our overnight jobs have a step for each database and run through it in turn.
The problem I can foresee is if customer 1 has a lot of users that are syncing then it'll wait to finish that before moving on to customer 2's database, etc.. by the time you get to customer 30 it could be a relatively long wait before their sync even begins; regardless of how quick the actual sync is. Ie. 2 customers enter data at the same time: As far as customer 1 is concerned, their sync happened in seconds whereas customer 30 took 30 mins before their data updated, even though the sync routine itself only took seconds to complete as it had to wait for 29 other databases to finish their work first.
I have had the idea of changing how we do our scheduled job here and create a job for each database/customer. That way the sync check will run synchronously across all database and no customer at the end of the queue will be waiting for other customers' sync to finish before theirs starts.
Is this a viable solution? Is it a had idea to have 30 jobs checking 30 databases every 10 seconds? Is there another, better option I haven't considered?
Our servers are running Microsoft SQL Server Web and Standard currently, though I believe we may be upgrading to enterprise at some point. (If the version makes a difference to my options here)
Not a technical question as such, but I do wonder how other people deal with the Schedules in SQL Server in relation to SQL Agent.
Personally, I like to create a bunch of schedules and then reuse them for various jobs.
As an example, I like to name my schedules in the following manner:
Daily - Every 15 seconds midnight to midnight
Daily - Every 15 seconds between 03:20 and 23:55
Daily - Every 22 minutes from midnight to midnight
Daily - Every 3 hours - starting at 03:30 to 23:59:59
Daily - Every 30 seconds from 00:00:05
It's not that stringent, but it helps me to understand the schedules a little better.
And then, I like to associate my jobs with existing schedules (the rule in the team is : DO NOT modify schedules)
So I do end up with a number of schedules that are linked to numerous jobs.
Discussing this with a colleague and wondering what is the correct approach, most efficient approach to this, we tested things a little.
One behaviour I thought was surprising is as follows:
create a new job
during that process, create a new schedule and associate it to that job
run the job once (mine was "SELECT 1 as one") to be sure to be sure
then drop that SQLAgent job
I would have expected the schedule to remain, but as it turns out the newly created schedule is also dropped!
How do others feel about that? is this correct behaviour?
And do you prefer to create a new schedule for every new job you create? or re-use Schedules?
Hoping to hear interesting views,
Kindest,
B
I'm not sure this is entirely opinion based, but I also won't be surprised if it gets flagged as such.
In the shop where I had the most sway over job scheduling, we opted to create new schedules for each job. Sometimes just one, sometimes several. That way, if we were experiencing high load on any given server, we could tweak individual job schedules to spread out the load without impacting multiple jobs. Say, for instance, one of 6 jobs with the same start time started to process exponentially more data, that one job could easily be shifted by X minutes so it wasn't competing for resources, or causing competition.
I have a job that runs every 5 minutes. Every once in a while, if our database load is large, the job will take longer than that. The job will then no longer run, because the next scheduled time is in the past. Is there a way I can get SQL Server to work around this so that it runs again even after one of these long runs?
If you were to have a script that runs continuously, you can spawn a second script from within the script every 5 minutes (without waiting for it to exit). There are many alternative (better) ways to do scheduling in Windows involving custom applications.
But then you will have overlapping script runs if one goes beyond it's 5 minutes, which is probably not what you want. A workaround is to create a temp file when the second script starts and delete it when it's done, and check for its existence in the beginning of the script, if it exists, you exit.
I would like to know what happens if an agent job (with recurring interval) in MS SQL server runs long enough so that it overlaps recurring execution.
According to my tests, paralleled execution does not happen, which is good.
What I'm trying t find out is that will the next execution be ignored because the previous one is not finished yet, OR will it be queued?
By "queued" I mean executing the queued requests immediately after the previous completed discarding the schedule.
Thank you
It will not be queued, it will be skipped. Easy to test: Create a job with a WAITFOR DELAY '00:05:00';, schedule job for < 2 minutes from now, then start the job manually. It will run for 5 minutes, once.