Validating the SSIS config files configured on the SQL Server Agent Jobs - sql-server

Recently I faced an issue in SQL Server Agent Job. The error is "Login Time Out Expired".
I have analyzed it. It seems like the server name mentioned on the SSIS config file is wrong. I have corrected the server name now the job runs fine.
Our job design - SQL Server Agent Job invokes an SSIS package along with its Config file.
The actual problem is we have lot of sql server agent jobs (200 + jobs). All are running on its own schedule. Currently we are fixing these issues as soon as we get an error in the Job history. This is purely a manual approach. This is one of the environment. We have almost 10 plus environments which have the same set of jobs.
I am looking for an approach where we can pre validate all the config files configured on the SQL Server agent jobs and report the files which have the incorrect server names or incorrect file paths. As you know, doing this task manually is an headache process even while doing this we may miss some jobs / create other issues as well.
Is any way we can validate the config files prior to running the SQL jobs.

You can get the sqlagent job steps information by querying the msdb tables. In that, you can find out, which configuration file is being used in the job step.
Refer to sql agent jobs documentation
SELECT
[sJOB].[job_id] AS [JobID]
, [sJOB].[name] AS [JobName]
, [sJSTP].[step_uid] AS [StepID]
, [sJSTP].[step_id] AS [StepNo]
, [sJSTP].[step_name] AS [StepName]
, CASE [sJSTP].[subsystem]
WHEN 'ActiveScripting' THEN 'ActiveX Script'
WHEN 'CmdExec' THEN 'Operating system (CmdExec)'
WHEN 'PowerShell' THEN 'PowerShell'
WHEN 'Distribution' THEN 'Replication Distributor'
WHEN 'Merge' THEN 'Replication Merge'
WHEN 'QueueReader' THEN 'Replication Queue Reader'
WHEN 'Snapshot' THEN 'Replication Snapshot'
WHEN 'LogReader' THEN 'Replication Transaction-Log Reader'
WHEN 'ANALYSISCOMMAND' THEN 'SQL Server Analysis Services Command'
WHEN 'ANALYSISQUERY' THEN 'SQL Server Analysis Services Query'
WHEN 'SSIS' THEN 'SQL Server Integration Services Package'
WHEN 'TSQL' THEN 'Transact-SQL script (T-SQL)'
ELSE sJSTP.subsystem
END AS [StepType]
, [sPROX].[name] AS [RunAs]
, [sJSTP].[database_name] AS [Database]
, [sJSTP].[command] AS [ExecutableCommand]
, CASE [sJSTP].[on_success_action]
WHEN 1 THEN 'Quit the job reporting success'
WHEN 2 THEN 'Quit the job reporting failure'
WHEN 3 THEN 'Go to the next step'
WHEN 4 THEN 'Go to Step: '
+ QUOTENAME(CAST([sJSTP].[on_success_step_id] AS VARCHAR(3)))
+ ' '
+ [sOSSTP].[step_name]
END AS [OnSuccessAction]
, [sJSTP].[retry_attempts] AS [RetryAttempts]
, [sJSTP].[retry_interval] AS [RetryInterval (Minutes)]
, CASE [sJSTP].[on_fail_action]
WHEN 1 THEN 'Quit the job reporting success'
WHEN 2 THEN 'Quit the job reporting failure'
WHEN 3 THEN 'Go to the next step'
WHEN 4 THEN 'Go to Step: '
+ QUOTENAME(CAST([sJSTP].[on_fail_step_id] AS VARCHAR(3)))
+ ' '
+ [sOFSTP].[step_name]
END AS [OnFailureAction]
FROM
[msdb].[dbo].[sysjobsteps] AS [sJSTP]
INNER JOIN [msdb].[dbo].[sysjobs] AS [sJOB]
ON [sJSTP].[job_id] = [sJOB].[job_id]
LEFT JOIN [msdb].[dbo].[sysjobsteps] AS [sOSSTP]
ON [sJSTP].[job_id] = [sOSSTP].[job_id]
AND [sJSTP].[on_success_step_id] = [sOSSTP].[step_id]
LEFT JOIN [msdb].[dbo].[sysjobsteps] AS [sOFSTP]
ON [sJSTP].[job_id] = [sOFSTP].[job_id]
AND [sJSTP].[on_fail_step_id] = [sOFSTP].[step_id]
LEFT JOIN [msdb].[dbo].[sysproxies] AS [sPROX]
ON [sJSTP].[proxy_id] = [sPROX].[proxy_id]
ORDER BY [JobName], [StepNo]
Now, you need to look into Executable command to see the exact configuration file being used in the SSIS execution and accordingly take action.
[ExecutableCommand]: The actual command which will be executed by the subsystem.

Related

Get the SQL Server Agent job that has run a particular execution of a deployed SSIS package

I have a SSIS package deployed in SQL Server and there are 3 different SQL Server Agent jobs that runs this package in different steps and schedules.
My question is: if the package is showing as failed in the Integration Services Catalogs -> Reports in one of the execution, is there a way I can identify which is the job that run that execution which caused the package failed (not by cross checking time of failure from the history of the job and the package failed execution time)?
It is not very straight forward. Based on this stack exchange answer, you may try:
SELECT
history.*
,ex.*
,ex.status
, CASE ex.status
WHEN 1 THEN 'created'
WHEN 2 THEN 'running'
WHEN 3 then 'canceled'
WHEN 4 then 'failed'
WHEN 5 then 'pending'
WHEN 6 then 'ended unexpectedly'
WHEN 7 then 'succeeded'
WHEN 8 then 'stopping'
WHEN 9 then 'completed'
END as job_status
FROM (
SELECT
h.step_name,
-- h.message,
h.run_status,
h.run_date,
h.run_time,
SUBSTRING(h.message, NULLIF(CHARINDEX('Execution ID: ', h.message),0)+14 ,PATINDEX('%[^0-9]%',SUBSTRING(h.message, NULLIF(CHARINDEX('Execution ID: ', h.message),0)+14 ,20))-1) ExecutionId
FROM MSDB.DBO.SYSJOBHISTORY h) history
LEFT JOIN
SSISDB.CATALOG.EXECUTIONS ex on ex.execution_id = history.ExecutionId
WHERE project_name = '<ssisdb_project_name_here>'
It has many columns which you can ignore by replacing * in select. The important part is to join MSDB.DBO.SYSJOBHISTORY with MSDB.DBO.SYSJOBHISTORY.
Also, this works for project deployment mode and not package deployment mode of SSIS.

How to find connection of SSIS Package in a SQL Job Step

I have a list of jobs to run multiple ssis packages. I have a big list, I want to know if it is possible to run a tsql to msdb and get the information of the packages in the steps with the connection strings.
Super old thread but the below code is slightly adapted and much more useful as it shows package level details.
select
prj.name as 'ProjectName'
,pa.name as 'SSISPackageName'
,op.parameter_name as 'ParmaterName'
,op.design_default_value as 'ConnectionString'
from
catalog.object_parameters op
join catalog.projects prj
on op.project_id = prj.project_id
join [catalog].[packages] pa
on pa.project_id = prj.project_id
where op.parameter_name like '%ConnectionString%'
If your packages are stored in the SSISDB Catalog, then you would need to query on SSIDB database to get the actual connection strings. Here is what I would suggest, use the query that Rodrigo A has provided and tweak the column 'Command' further to get the packages name from it by using string functions and have the Step ID and Package Name as output. use the list of packages that you have obtained and put it in the IN clause of the following query and run it against SSISDB -
select prj.name as 'ProjectName'
,op.object_name as 'SSISPackageName'
,op.parameter_name as 'ParmaterName'
,op.design_default_value as 'ConnectionString'
from catalog.object_parameters op
join catalog.projects prj
on op.project_id = prj.project_id
where op.parameter_name like '%ConnectionString%'
and op.object_name in (
--put the package name list here
);
You can then join on the result sets, to get the step Id, PackageName and the connection strings in that package.
Run this query over the server you want to check:
SELECT [sJSTP].[step_id], Name, sJSTP.Command FROM [msdb].[dbo].[sysjobs] [sJOB]
LEFT JOIN [msdb].[dbo].[sysjobsteps] AS [sJSTP]
ON [sJOB].[job_id] = [sJSTP].[job_id]
AND [sJOB].[start_step_id] = [sJSTP].[step_id]
ORDER BY Name, step_id
Cheers!

Method for Extracting application name from SQL Server Audit log or SQL Audit trace?

I have two sample SQL Server 2012 and 2014 instances, and a central syslog server, I'm searching the web since weeks but cannot find a solution.
I would like to exctract the "application name" field or any similar from a relevant source, Audit log or audit trace. I've found the needed field in the .trc file, but nothing in Audit logs and also cannot find a method how to write this kind of log to a readable text file (in order to forward it to a syslog server).
It would be good to know the client application name (eg.: SQL Server Management Studio) or process name (pymssql) that's using database resources, connects do queries... etc.
Has somebody already faced this issue?
cannot find a method how to write this kind of log to a readable text file (in order to forward it to a syslog server).
SELECT TE.name AS [EventName] ,
T.DatabaseName ,
t.DatabaseID ,
t.NTDomainName ,
t.ApplicationName ,
t.LoginName ,
t.SPID ,
t.Duration ,
t.StartTime ,
t.EndTime
FROM sys.fn_trace_gettable(CONVERT(VARCHAR(150), ( SELECT TOP 1
f.[value]
FROM sys.fn_trace_getinfo(NULL) f
WHERE f.property = 2
)), DEFAULT) T
JOIN sys.trace_events TE ON T.EventClass = TE.trace_event_id
WHERE te.name = 'Data File Auto Grow'
OR te.name = 'Data File Auto Shrink'
ORDER BY t.StartTime ;
More Examples :
https://www.simple-talk.com/sql/performance/the-default-trace-in-sql-server-the-power-of-performance-and-security-auditing/
https://www.simple-talk.com/sql/database-administration/collecting-the-information-in-the-default-trace/

Automatically stop SQL Server Agent when no jobs are running [duplicate]

I have around 40 different sql server jobs in one instance. They all have different schedules. Some run once a day some every two mins some every five mins. If I have a need to stop sql server agent, how can I find the best time when no jobs are running so I won't interrupt any of my jobs?
how can I find the best time when no jobs are running so I won't interrupt any of my jobs?
You basically want to find a good window to perform some maintenance. #MaxVernon has blogged about it here with a handy script
/*
Shows gaps between agent jobs
-- http://www.sqlserver.science/tools/gaps-between-sql-server-agent-jobs/
-- requires SQL Server 2012+ since it uses the LAG aggregate.
Note: On SQL Server 2005, SQL Server 2008, and SQL Server 2008 R2, you could replace the LastEndDateTime column definition with:
LastEndDateTime = (SELECT TOP(1) s1a.EndDateTime FROM s1 s1a WHERE s1a.rn = s1.rn - 1)
*/
DECLARE #EarliestStartDate DATETIME;
DECLARE #LatestStopDate DATETIME;
SET #EarliestStartDate = DATEADD(DAY, -1, GETDATE());
SET #LatestStopDate = GETDATE();
;WITH s AS
(
SELECT StartDateTime = msdb.dbo.agent_datetime(sjh.run_date, sjh.run_time)
, MaxDuration = MAX(sjh.run_duration)
FROM msdb.dbo.sysjobs sj
INNER JOIN msdb.dbo.sysjobhistory sjh ON sj.job_id = sjh.job_id
WHERE sjh.step_id = 0
AND msdb.dbo.agent_datetime(sjh.run_date, sjh.run_time) >= #EarliestStartDate
AND msdb.dbo.agent_datetime(sjh.run_date, sjh.run_time) < = #LatestStopDate
GROUP BY msdb.dbo.agent_datetime(sjh.run_date, sjh.run_time)
UNION ALL
SELECT StartDate = DATEADD(SECOND, -1, #EarliestStartDate)
, MaxDuration = 1
UNION ALL
SELECT StartDate = #LatestStopDate
, MaxDuration = 1
)
, s1 AS
(
SELECT s.StartDateTime
, EndDateTime = DATEADD(SECOND, s.MaxDuration - ((s.MaxDuration / 100) * 100)
+ (((s.MaxDuration - ((s.MaxDuration / 10000) * 10000))
- (s.MaxDuration - ((s.MaxDuration / 100) * 100))) / 100) * 60
+ (((s.MaxDuration - ((s.MaxDuration / 1000000) * 1000000))
- (s.MaxDuration - ((s.MaxDuration / 10000) * 10000))) / 10000) * 3600, s.StartDateTime)
FROM s
)
, s2 AS
(
SELECT s1.StartDateTime
, s1.EndDateTime
, LastEndDateTime = LAG(s1.EndDateTime) OVER (ORDER BY s1.StartDateTime)
FROM s1
)
SELECT GapStart = CONVERT(DATETIME2(0), s2.LastEndDateTime)
, GapEnd = CONVERT(DATETIME2(0), s2.StartDateTime)
, GapLength = CONVERT(TIME(0), DATEADD(SECOND, DATEDIFF(SECOND, s2.LastEndDateTime, s2.StartDateTime), 0))
FROM s2
WHERE s2.StartDateTime > s2.LastEndDateTime
ORDER BY s2.StartDateTime;
The question title scared me a bit - I thought you wanted to programmatically shut the SQL Server agent down anytime there were no jobs running. My answer to that question would be "Why?" There is no need to.
But if you are just looking to do a planned restart or shut down and you don't have a third party tool like Sentry One's SQL Sentry Event Manager to have a visualization, I would just let the SQL Server Agent Job History and Job Activity Monitor help here. The Job Activity monitor can show you which jobs are running right now in the status column. You can also see the last execute and next execute dates and times.
In the object browser in SSMS, connect to your instance, then expand SQL Server Agent, then you'll see Jobs and under that you'll see "Job Activity Monitor" - this view should show you what you need.
Also - don't worry about shutting down before a job executes. If you do that, you will either have that job just missing its schedule and you can let it run when it is next due to (depending on the job and its purpose) or you can manually right click and execute the job.
For more on the activity monitor for jobs, see Monitor Job Activity in the product documentation.
I recommend creating a script that will disable your jobs. Disabled jobs still exist but will not be automatically launched by their schedules. Run this script (based on procedure sp_update_job in the msdb database) to disable jobs, wait for any currently running jobs to finish execution, then stop SQL agent. A similar script to re-enable disabled jobs would be useful. You might need to plan around jobs that are and should remain disabled.
A complete “SQL Agent shutdown” process could be fully scripted, but I question the wisdom of doing so. A bit of research implies that there is no 100% reliable way of programmatically telling if a given job is or is not running, and while there is an undocumented (where "undocumented" means "you really shouldn't be using this") system procedure for stopping and starting services, doing so from with SQL Server itself seems like a pretty bad idea.
You can query the system tables as shown by Dattatrey Sindol in the MSSQLTips.com article Querying SQL Server Agent Job Information:
SELECT
[sJOB].[job_id] AS [JobID]
, [sJOB].[name] AS [JobName]
, [sDBP].[name] AS [JobOwner]
, [sCAT].[name] AS [JobCategory]
, [sJOB].[description] AS [JobDescription]
, CASE [sJOB].[enabled]
WHEN 1 THEN 'Yes'
WHEN 0 THEN 'No'
END AS [IsEnabled]
, [sJOB].[date_created] AS [JobCreatedOn]
, [sJOB].[date_modified] AS [JobLastModifiedOn]
, [sSVR].[name] AS [OriginatingServerName]
, [sJSTP].[step_id] AS [JobStartStepNo]
, [sJSTP].[step_name] AS [JobStartStepName]
, CASE
WHEN [sSCH].[schedule_uid] IS NULL THEN 'No'
ELSE 'Yes'
END AS [IsScheduled]
, [sSCH].[schedule_uid] AS [JobScheduleID]
, [sSCH].[name] AS [JobScheduleName]
, CASE [sJOB].[delete_level]
WHEN 0 THEN 'Never'
WHEN 1 THEN 'On Success'
WHEN 2 THEN 'On Failure'
WHEN 3 THEN 'On Completion'
END AS [JobDeletionCriterion]
FROM
[msdb].[dbo].[sysjobs] AS [sJOB]
LEFT JOIN [msdb].[sys].[servers] AS [sSVR]
ON [sJOB].[originating_server_id] = [sSVR].[server_id]
LEFT JOIN [msdb].[dbo].[syscategories] AS [sCAT]
ON [sJOB].[category_id] = [sCAT].[category_id]
LEFT JOIN [msdb].[dbo].[sysjobsteps] AS [sJSTP]
ON [sJOB].[job_id] = [sJSTP].[job_id]
AND [sJOB].[start_step_id] = [sJSTP].[step_id]
LEFT JOIN [msdb].[sys].[database_principals] AS [sDBP]
ON [sJOB].[owner_sid] = [sDBP].[sid]
LEFT JOIN [msdb].[dbo].[sysjobschedules] AS [sJOBSCH]
ON [sJOB].[job_id] = [sJOBSCH].[job_id]
LEFT JOIN [msdb].[dbo].[sysschedules] AS [sSCH]
ON [sJOBSCH].[schedule_id] = [sSCH].[schedule_id]
ORDER BY [JobName]

CRM Auto Pre-filter doesn't pass a query

I've created a simple SSRS report using Visual Studio 2012,
I'm using CRMAF_ prefix to use CRM's auto filtering, and achieve a context-based report.
I'm using two datasets to achieve this; dsFiltered for the filtered data, and dsApprovalSummary for my report.
This is the query dsFiltered uses :
declare #sql as nVarchar(max)
set #sql = 'SELECT vrp_investdocumentid
FROM (' + #CRM_Filteredvrp_investdocument + ') as CRMAF_vrp_investdocument'
exec(#sql)
This is the query dsApprovalSummary uses :
select doc.vrp_name as 'Yatırım Dosyası',
act.vrp_actioncode as 'Aksiyon Kodu',
cfg.vrp_description as 'Aksiyon Açıklaması',
act.OwnerIdName as 'Aksiyon Sorumlusu',
act.ModifiedOn as 'Son Değiştirme Tarihi'
from vrp_action act
inner join vrp_investdocument as doc on act.RegardingObjectId=doc.vrp_investdocumentId
inner join vrp_actionconfig as cfg on act.vrp_actioncode = cfg.vrp_actioncode
where cfg.vrp_reporttask=1 and act.RegardingObjectId = #documentId
order by act.ModifiedOn
The parameters are :
#CRM_Filteredvrp_investdocument - The parameter CRM should have been populated with a query, defaults to null
#CRM_vrp_investdocumentId - Comes from dsFiltered (CRMAF_vrp_investdocument.vrp_investdocumentid); allows null.
The report works perfectly on the development server. However, when i deploy the report into the production server, it does not ask me to select a filter, or does not have a default filter; tries to run directly and then gives an rsProcessingAborted. I've checked the logs, and saw it said SYNTAX ERROR NEAR )-.
This is from the report server logs :
processing!ReportServer_0-20!13ec!11/11/2014-13:45:04:: w WARN: Data source 'srcApprovalSummary': Report processing has been aborted.
processing!ReportServer_0-20!13ec!11/11/2014-13:45:04:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ProcessingAbortedException: ,
Microsoft.ReportingServices.ReportProcessing.ProcessingAbortedException: An error has occurred during report processing.
---> Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Query execution failed for dataset 'dsFiltered'.
---> System.Data.SqlClient.SqlException: Incorrect syntax near ')'
UPDATE : On the development server, we have everything installed on the same machine; CRM Frontend, Services, SQL Server, Report Server etc. But on the production environment, each one of these servers are different machines. Could this be the source of error?
UPDATE 2 : Running the profiler gave me that #CRM_Filteredvrp_investdocument comes in NULL. See the query below from the profiler :
exec sp_executesql N'declare #sql as nVarchar(max)
set #sql = ''SELECT vrp_investdocumentid
FROM ('' + #CRM_Filteredvrp_investdocument + '') as CRMAF_vrp_investdocument''
exec(#sql)',N'#CRM_Filteredvrp_investdocument nvarchar(4000)',#CRM_Filteredvrp_investdocument=NULL
It turns out to be a collation problem, i've been trying to use a custom data source with this connection string :
Data Source=myprodsqlserver; Initial Catalog=myorganization_MSCRM;
I've rewritten it lowercase, and replaced the data source with localhost the problem is magically gone.
data source=localhost; initial catalog=myorganization_MSCRM;
In the report editor, try rebuilding the datasource used by each of your datasets using the connection string builder (don't type it manually). Build them so they point to your Prod CRM database and then test the report completely in the report editor. This will determine if the problem is lies with the report or CRM.

Resources