What is the best practice to monitor SQL Server job steps?
I have a single job with 50 steps. Sometimes, some of the steps are taking longer to complete. I want to have a monitoring process that notifies me when a step takes longer than usual. The problem is that I'm not able to find any information about job steps while a step is running and before they get completed (at least not in sysjobsteps and sysjobs). In particular, I'm interested about the step_id, step_name and step_start_time.
I would appreciate any ideas.
The system table you're looking for is actually dbo.sysjobhistory, and specifically the run_date, run_time, and run_duration columns:
run_date - Date the job or step started execution. For an In Progress history, this is the date/time the history was written.
run_time - Time the job or step started in HHMMSS format.
run_duration - Elapsed time in the execution of the job or step in HHMMSS format.
I do realize you asked for information while a job step is in process, but as per the Microsoft Books Online:
In most cases the data is updated only after the job step completes and the table typically contains no records for job steps that are currently in progress, but in some cases underlying processes do provide information about in progress job steps.
So only in certain cases will you have access to that information.
Alternatively, I can only think to recommend monitoring the running queries on your server instead, if you want a more realtime approach. You can use tools like Adam Machanic's sp_WhoIsActive to help you accomplish that. You may find it difficult to correlate the job step itself to the query it's executing, unless the query is tagged with an identifiable comment in the beginning, but I also realize this is not ideal either.
Related
I am a BI developer and was entrusted with a task of a DBA. It's about monitoring SQL Server Agent jobs respectively getting information about the following key figures, which are to be prepared after all frontend:
Failed Server Agent Jobs and Jobsteps
Run-time of jobs and Jobsteps
Find out if a job has even started
Memory allocation: evaluation and monitoring of the moving and stored data volumes
The goal is to monitor and, if necessary, provide an indication of any discrepancies in the key figures.
I got the first two points. In each case I show in a table whether a job or whether steps are enabled/disabled. I have also recorded the terms of each step and thresholds can be used to warn if critical ranges are reached.
The biggest problems for me are the jobs that did not even start (Point 3.). To my knowledge, these are not recorded in the MSDB tables. I would like to know when a job or jobstep has not even started. Can you tell me how to get this info? Maybe someone already has a suitable script for this problem ready?
On the subject of memory usage, I'm interested in how much free space is left on the hard disk, how big the partition is and how the consumption changes over time.
On the internet, I could find nothing to the points 3rd and 4th. I would be very grateful for your help! (and forgive me my bad english :) )
I get as a result, when the next run of Job xy is planned. He does not appear in the table.
However, he appears on the SQL Server Agent in the job history. There, however, the information is missing when he should run the next time.
My plan was to take the next_scheduled_run_date column from the sysjobactivity table and then compare a run later to the run_requested_Date column.
But obviously some records are missing in the sysjobactivity table. The other table, which otherwise contains target_start times, is the sysjobschedules.
Unfortunately, this only has the currently scheduled date. I have not found another table that contains a history of the target run_dates.
Of course, one could have manual tables (analogous to the target values) but that would be too much effort.
I am creating a database medical system and then I came to a point where I am trying to create a notification feature and i will use SQL jobs in it, where the SQL job responsibility is to check some tables and the entities that will find it need to be notified for a change in certain data will put their ids in an entity called Notification and a trigger will be called for the app to check that table and send the notificiation.
what I want to ask is how many SQL jobs can a sql server handle ?
Does the number of running SQL jobs in background affect the performance of my application or the database performance in a way or another ?
NOTE: the SQL job will run every 10 seconds
I couldn't find any useful information online.
thanks in advance.
This question really doesn't have enough background to get a definitive answer. What are the considerations?
Do the queries in your ten-second job actually complete in ten seconds, even when your DBMS is under its peak transactional workload? Obviously, if the job routinely doesn't complete in ten seconds, you'll get jobs piling up.
Do the queries in your job lock up tables and/or indexes so the transactional load can't run efficiently? (You should use SET ISOLATION LEVEL READ UNCOMMITTED; as much as you can so database reads won't lock things unnecessarily.)
Do the queries in your job do a lot of rows' worth of inserts and updates, and so swamp the SQL Server transaction logs?
How big is your server? (CPU cores? RAM? IO capacity?) How big is your database?
If your project succeeds and you get many users, will your answers to the above questions remain the same? (Hint: no.)
You should spend some time on the execution plans for the queries in your job, and try to make them as efficient as possible. Add the necessary indexes. If necessary refactor the queries to make them more efficient. SSMS will show you the execution plans and suggest appropriate indexes.
If your job is doing things like deleting expired rows, you may want to build the expiration in your data model. For example, suppose your job does
DELETE FROM readings WHERE expiration_date >= GETDATE()
and your application does this, relying on your job to avoid getting expired readings.
SELECT something FROM readings
You can refactor your application query to say
SELECT something FROM readings WHERE expiration_date < GETDATE()
and then run your job overnight, at a quiet time, rather than every ten seconds.
A ten-second job is not the greatest idea in the world. If you can rework your application so it will function correctly with a ten-second, ten-minute, or twelve-hour job, you'll have a more resilient production system. At any rate if something goes wrong with the job when your system is very busy you'll have more than ten seconds to fix it.
Not a technical question as such, but I do wonder how other people deal with the Schedules in SQL Server in relation to SQL Agent.
Personally, I like to create a bunch of schedules and then reuse them for various jobs.
As an example, I like to name my schedules in the following manner:
Daily - Every 15 seconds midnight to midnight
Daily - Every 15 seconds between 03:20 and 23:55
Daily - Every 22 minutes from midnight to midnight
Daily - Every 3 hours - starting at 03:30 to 23:59:59
Daily - Every 30 seconds from 00:00:05
It's not that stringent, but it helps me to understand the schedules a little better.
And then, I like to associate my jobs with existing schedules (the rule in the team is : DO NOT modify schedules)
So I do end up with a number of schedules that are linked to numerous jobs.
Discussing this with a colleague and wondering what is the correct approach, most efficient approach to this, we tested things a little.
One behaviour I thought was surprising is as follows:
create a new job
during that process, create a new schedule and associate it to that job
run the job once (mine was "SELECT 1 as one") to be sure to be sure
then drop that SQLAgent job
I would have expected the schedule to remain, but as it turns out the newly created schedule is also dropped!
How do others feel about that? is this correct behaviour?
And do you prefer to create a new schedule for every new job you create? or re-use Schedules?
Hoping to hear interesting views,
Kindest,
B
I'm not sure this is entirely opinion based, but I also won't be surprised if it gets flagged as such.
In the shop where I had the most sway over job scheduling, we opted to create new schedules for each job. Sometimes just one, sometimes several. That way, if we were experiencing high load on any given server, we could tweak individual job schedules to spread out the load without impacting multiple jobs. Say, for instance, one of 6 jobs with the same start time started to process exponentially more data, that one job could easily be shifted by X minutes so it wasn't competing for resources, or causing competition.
I have an ETL and Cube solutions, which I process one after another in a SQL agent job.
In ETL I run 1 package, that in turn one by one runs all other packages.
Whole processing takes 10 hours.
For ETL:
How can I find out which package takes what amount of time to run within that one parent package, other than opening solution and record times?
For cube:
Here dimensions process fast. What do I measure here in order to find which part takes it so long? Maybe measures? How to track processing times of particular measure?
Maybe SQL Profiler will help? If so, is there a good article which describes which metrics there should I pay attention to?
To gather statistics about SSIS execution times, you can enable logging:
For package deployment model, you'll have to turn on logging in each package, go to SSIS > logging. In the dialogue choose the Pre and Post Execute events. Use a sql logging provide which will log to a system table called dbo.sysssislog. You'll need to join pre and post events on execution id.
For Project deployment model, it's probably already on. This can be configured in SSMS, Integration Services > SSISDB, right click and choose properties. Once you've executed the package, you can see the results in the standard reports. Right click the master package and choose Reports > Standard Reports > All Executions.
Lots more details on SSIS logging here: https://learn.microsoft.com/en-us/sql/integration-services/performance/integration-services-ssis-logging
For SSAS, I always tested this manually. Connect in SSMS, right click on each Measure group and do a process full (this assumes the dimensions have just been freshly processed.) The measures are more likely to be the cause of an issue because of the amount of data.
Once you understand which measure is slow, you can look at tuning the source query, if it has any complexity to it, or partitioning the measure group and doing incremental loading. Full processing could be scheduled periodically.
m
I have a situation where I have a job that runs every day (Job A), job that runs every 2 days (Job B) and another job that runs every weekend (Job C). I need to make sure that Job A runs before Job B. If Job A does not run appropriately then i don't want Job B to run. The same thing applies to Job C. Anyone have any thoughts on how to go about this?
Appreciate any help
I have used a product called SQL Sentry to do what you are trying to do. SQL Sentry has a lot of other advanced monitoring and control functionality (like killing jobs that hang, queuing low priority jobs, etc). Here is their website https://sentryone.com/platform/sql-server-performance-monitoring.
This is a quote from one of their advertising:
19. Chaining and Queuing
Did you ever wish you could find just a few more hours in your
maintenance window, or need to have jobs run in a particular sequence?
The advanced chaining features in SQL Sentry Event Manager can assure
that interdependent jobs run in the proper order without wasting time
or resources.
Chaining
SQL Sentry Event Manager allows you to chain
SQL Agent Jobs, Windows Tasks, or Oracle Jobs across servers. You can
enforce dependencies and automate workflow throughout your entire
enterprise, even across platforms! The graphical chaining interface
allows you to design the workflow using variables such as completion,
success, or failure. More details are available in the User Guide, but
take a few minutes to watch our two video tutorials on chaining -
Graphical Chaining Interface and Advanced Chaining.
I need to make sure that Job A runs before Job B. If Job A does not run appropriately then i don't want Job B to run. The same thing applies to Job C.
Create all jobs A,B,C and schedule only job A..At the end of job A ,success event ,Call job B like below
EXEC dbo.sp_start_job N'Weekly Sales Data Backup' ;
GO
Now the same things applies to job c,call job c on success event of job B..
I would go with this approach..You also can go with an approach of insert success ,failure values into a table and ensure job b or c reads those values before starting