There's been a string of random occurrences of the following error in the SQL Server Agent scheduled jobs lately that I have been unable to find a solution to.
The error occurs infrequently, but usually once a week for a daily scheduled job, but in any number of different jobs and not always the same one. Each job shares the fact that it executes an SSIS package from the same server that is running the job. It also always runs for almost exactly 30 seconds elapsed time, which I guess is the timeout threshold. I'm not sure why it would timeout if the server is just connecting to its own SSIS catalog. Also of note is that it never actually gets to the point where it executes the SSIS package, and this occurs regardless of which package is trying to be executed.
During my research I came across many people suggesting that simply updating SQL Server 2012 to the latest CU* or SP2 would solve the problem. However, upgrading the server to SP2 has not.
One solution tried (which admittedly was ugly) was to simply have a single retry upon failure of the job step, which actually did solve the problem in about 30% of the cases.
I would welcome anyone who has experience with this error, or anyone who has any suggestions.
The error message is as follows:
Date 16/07/2014 6:00:11 AM
Log Job History ({$jobname})
Step ID 1
Server {$productionserver}
Job Name {$jobname}
Step Name {$stepname}
Duration 00:00:31
Sql Severity 0
Sql Message ID 0
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0
Message
Executed as user: {$user}.
Microsoft (R) SQL Server Execute Package Utility Version 11.0.5058.0 for 64-bit Copyright (C) Microsoft Corporation. All rights reserved.
Started: 6:00:11 AM Failed to execute IS server package because of error 0x80131904.
Server: {$productionserver},
Package path: {$packagepath},
Environment reference Id: NULL.
Description: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Source: .Net SqlClient Data Provider
Started: 6:00:11 AM Finished: 6:00:42 AM
Elapsed: 31.122 seconds. The package execution failed. The step failed.
Try this:
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding
And this
https://connect.microsoft.com/SQLServer/feedback/details/783291/ssis-package-fails-to-start-application-lock-timeout-in-ssisdb-catalog-create-execution
Looks like its a known bug.
Check what else is/was running on the instance at the time of the package failures (e.g. a database integrity check or similarly intensive operation).
The SQL Agent is timing out talking to its own SSIS catalog (a 30 second timeout). It's not actually executing the packages, so it's nothing to do with the packages themselves and everything to do how busy the instance is at the time of the execution.
(Answering this question since it comes up in a Google search)
I know this is an older question. but I'm having the same problem and this doesn't have an accepted answer.
The job fails in 1.5 seconds so I believe it is NOT a timeout issue.
I can confirm 0x80131904 is (or can be) a permissions issue. I had my SSIS package running under a SQL Agent job just fine with sysadmin and network admin privileges. when i switched it to an account with fewer permissions, i get this error.
For me, the problem was because i was not assigning permissions in all the correct places. I already set Read/Execute permissions in the Project Properties. Then (this is the step I didn't do) I had to assign Read permissions on the folder containing Projects and Environments.
Hope this helps someone.
We have experienced this error when attempting to start several SSIS packages at the same instant. Service packs were supposed to fix it, but have not. We have implemented a staggered schedule for SSIS packages so only one package is starting at any given moment.
We also experienced the same bug. As a workaround, we created the following stored procedure. If you put this into a job that runs every f.e. 10 minutes, it makes sure that if there are random failures, the job gets restarted continuously until you reach an occurence without timeout failure.
USE [msdb]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[usp_StartTimedOutJob]
AS
DECLARE #jobid NVARCHAR(100)
, #jobname NVARCHAR(250)
, #stepname NVARCHAR(250)
, #varMail VARCHAR(MAX)
DECLARE cJobs CURSOR FOR
-- CTE selects all jobs that are currently not running and orders them by most recent
WITH CTE_NotRunning AS (
SELECT S.job_id
, S.step_name
, S.[message]
, rownum = ROW_NUMBER() OVER (PARTITION BY S.job_id ORDER BY S.run_date DESC, S.run_time DESC)
FROM msdb.dbo.sysjobhistory AS S
LEFT OUTER JOIN (SELECT DISTINCT ja.job_id
FROM msdb.dbo.sysjobactivity ja
LEFT JOIN msdb.dbo.sysjobhistory jh ON ja.job_history_id = jh.instance_id
JOIN msdb.dbo.sysjobs j ON ja.job_id = j.job_id
JOIN msdb.dbo.sysjobsteps js
ON ja.job_id = js.job_id
AND ISNULL(ja.last_executed_step_id,0)+1 = js.step_id
WHERE
ja.session_id = (
SELECT TOP 1 session_id FROM msdb.dbo.syssessions ORDER BY agent_start_date DESC
)
AND start_execution_date is not null
AND stop_execution_date is NULL) AS R
ON S.job_id = R.job_id
WHERE R.job_id IS NULL)
-- only select the jobs into the cursor set for which the most recent job had a timeout issue
SELECT job_id
, step_name
FROM CTE_NotRunning
WHERE [message] LIKE '%0x80131904%time%out%' -- error message that corresponds to timed out jobs, error code: 0x80131904
AND rownum = 1
OPEN cJobs
FETCH NEXT FROM cJobs
INTO #jobid, #stepname
WHILE ##FETCH_STATUS = 0
BEGIN
-- for each of the timed out jobs in the cursor, start the job again from the step that caused the timeout
SET #jobname = (SELECT [name] FROM msdb.dbo.sysjobs WHERE job_id = #jobid)
EXECUTE dbo.sp_start_job #job_id = #jobid, #step_name = #stepname
END
CLOSE cJobs
DEALLOCATE cJobs
GO
I had this exact same issue. SQL Agent was running SSIS Jobs perfectly fine then suddenly I came across this error. Spent about an hour looking for a fix online. Found out the server admin had installed new windows updates.
I simply restarted the Server (which hosts the SSIS catalog and SQL Server/Agent). After server restart jobs ran fine again.
Hope server restart works for the next person that goes through this.
Sometimes this kind of error occurs when the package is deployed twice under SQL Integration Service Catalogs. You also may have changed the package name but there are other related auto-generated configurations are unique like the Environment reference Id and others .
So if you have a scheduled job, you will need to create a new one and point it to the .
Good Luck
I had the same problem and error message on SQL Server 2017.
My problem was on the SSISDB database, that was too big and had to be maintained (no more space available). After having cleaned up the SSISDB database, the jobs ran well again on this server.
Related
This question is related to: Debezium How do I correctly register the SqlServer connector with Kafka Connect - connection refused
In Windows 10, I have Debezium running on an instance of Microsoft SQL Server that is outside of a Docker container. I am getting the following warning every 390 milliseconds:
No maximum LSN recorded in the database; please ensure that the SQL
Server Agent is running
[io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource]
I checked Debezium's code on Github and the only place that I can find this warning states in the code comments that this warning should only be thrown if the Agent is not running. I have confirmed that the SQL Server Agent is running.
Why is this warning showing up and how do I fix it?
Note:
My current solution appears to only work in a non-production environment - per Docker's documentation.
LSN is the "pieces" of information related about your SQL Server changes. If you don't have LSN, is possible that your CDC is not running or not configured properly. Debezium consumes LSNs to replicate so, your SQL Server need to generate this.
Some approaches:
Did you checked if your table are with CDC enabled? This will list your tables with CDC enabled:
SELECT s.name AS Schema_Name, tb.name AS Table_Name
, tb.object_id, tb.type, tb.type_desc, tb.is_tracked_by_cdc
FROM sys.tables tb
INNER JOIN sys.schemas s on s.schema_id = tb.schema_id
WHERE tb.is_tracked_by_cdc = 1
Your CDC database are enabled and runnig? (see here)
Check if enabled:
SELECT *
FROM sys.change_tracking_databases
WHERE database_id=DB_ID('MyDatabase')
And check if is running:
EXECUTE sys.sp_cdc_enable_db;
GO
Your CDC service are running on SQL Server? See in docs
EXEC sys.sp_cdc_start_job;
GO
On enabling table in CDC, I had some issues with rolename. For my case, configuring at null solved my problem (more details here)
EXEC sys.sp_cdc_enable_table
#source_schema=N'dbo',
#source_name=N'AD6010',
#capture_instance=N'ZZZZ_AD6010',
#role_name = NULL,
#filegroup_name=N'CDC_DATA',
#supports_net_changes=1
GO
Adding more to William's answer.
For the case SQL Server Agent is not running
You can enable it by following :
Control panel >
Administrative Tools >
Click "Services"
Look for SQL Server Agent
Right click and Start
Now you can fire cdc job queries in your mssql.
PS: you need to have login access to windows server.
Another possibility of this error (I just ran into this warning myself this morning trying to bring a new DB online) is the SQL login does not have the permissions needed. Debezium runs the following SQL. Check that the SQL login you are using has access to run this stored procedure and it returns the tables you have set up in CDC. If you get an error or zero rows returned, work with your DBA to get the appropriate permissions set up.
EXEC sys.sp_cdc_help_change_data_capture
We have a SQL Server Agent job that runs every hour. Twice a month, the server is rebooted for Microsoft patching. The reboot is done by a different department, so we only know that it will occur during a 4 hour window on a specific day. When the server comes back up, is there a way to make any jobs that didn't start during this time frame autostart?
To my knowledge, there is no "built-in" way. But maybe you could do another job, that checks the last restart date of SQL Server.
SELECT sqlserver_start_time FROM sys.dm_os_sys_info
Then, if it detects that the server was restarted not long ago, it could check when your job ran last time.
select j.job_id, name, last_run_date, last_run_time, last_run_outcome, last_run_duration
from msdb.dbo.sysjobs as j (NoLOCK)
inner join msdb.dbo.sysjobservers as s (NOLOCK) on s.job_id = j.job_id
where enabled=1
And if necessary, tell SQL to start the job immediately
EXEC msdb.dbo.sp_start_job N'YourJobName'
The sp_procoption system stored procedure can be used to run one (or multiple) user defined stored procedures upon the SQL Server service starting. A stored procedure must be created in the master database to be used by this system SP. You can then create a stored procedure that queries the msdb tables for specific (or any as described in your post) jobs that are inactive and then use sp_start_job to run the associated job. The scan for startup procs server configuration option must be enabled to use sp_procoption, however executing this stored procedure automatically enables this which you can verify using sp_configure. An example of adding a stored procedure to execute when the SQL Server service starts via sp_procoption is below.
USE [master]
EXEC SP_PROCOPTION #ProcName = 'uspCheckInactiveJobs', #OptionName = 'startup', #OptionValue = 'on';
I have an SSIS package (built w/Visual Studio 2012) with various Execute SQL Tasks in it. It's running as a job on SQL Server 2012. In the package properties I have TransactionOption set to Required; all of the tasks' TransactionOption settings are Supported. IsolationLevel is set to Serializable on everything. Suddenly yesterday, one of the tasks started hanging. If I run this task individually, it runs fine. If I disable it and run the rest of the package, there's no problem. However if this task is enabled and I try to run the package, it hangs at this task, and blocks all sessions behind it. The only way I can get the entire package to run properly is to change the package's TransactionOption setting to Supported (i.e., turn transactions off). Here's what the offending task does:
update timekeep
set tkemail = u.EMail,
tkmoddate = getdate(),
tkmoduser = 'appmgr'
from ALLUSERDATA.AllUserData.dbo.[User] u
where u.Login COLLATE SQL_Latin1_General_CP1_CI_AI = tkinit
and tkemail <> u.EMail COLLATE SQL_Latin1_General_CP1_CI_AI
I've tried putting WITH (ROWLOCK) after the UPDATE statement, but that doesn't help. The timekeep table is used by multiple people throughout the company all day, but the SSIS package is scheduled to run at 3:45 am, and it's still hanging. Again, this just started yesterday morning. The DBA says nothing on the server has changed. Any ideas on what could be causing this?
I have an interesting problem going on. I recently moved 2 SQL databases to SQL Azure for a client and all seemed to be going well...at first. Mid-morning I get a spike of error emails for various things, but a few common ones:
-The request limit for the database is 90 and has been reached.
-Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
-A transport-level error has occurred when receiving results from the server.
There's obviously some database related issues going on with the move to Azure, or the existing code in general. The errors that seemed to happen the most were request limit and timeouts. Once they started, they never seemed to stop. And I don't think there were many users using the site today. It almost seemed like the connection continued to try to connect on a different thread in the background if this makes any sense. This is in reference to the "The timeout period elapsed prior to completion of the operation or the server is not responding." I would get an error email, I'd check the page it referenced myself, and it would load immediately. I checked with the user who threw the error and they reported everything was fine. Strange. Yet I continued every few minutes to get the same error email.
I currently have them on the S1 Tier which limits the requests to 90 concurrently. I did some digging and found the following SQL query:
select * from sys.dm_exec_connections
I ran this, and it showed I had over 90 active connections, some of which were opened some time ago. This was strange to me as the site was currently not being used (it's really late at night and I know no one is using the site). I wanted to end all the connections so I came up with the following query:
DECLARE #sessionId int
DECLARE #SQL nvarchar(1000)
DECLARE #clientIP nvarchar(50)
set #clientIP = 'XX.XX.XX.XX'
select #sessionId = min( session_id ) from sys.dm_exec_connections where client_net_address = #clientIP
while #sessionId is not null
begin
SET #SQL = 'KILL ' + CAST(#sessionId as varchar(4))
EXEC (#SQL)
select #sessionId = min( session_id ) from sys.dm_exec_connections where session_id > #sessionId and client_net_address = #clientIP
end
I tried running this command, but the connections came right back. I went on the web server and manually stopped the site in IIS, ran the KILL command again but the connections remained. I put up the app_offline file and took the site down for about a half hour to see if any lingering connections would drop, but they didn't. And I still continued to get error emails for pages I KNEW were not accessible because I stopped the Site AND app pool. I went on the server and manually stopped the w3wp process and ran SQL KILL statements to kill the connections. They finally went away! I put the app back online and hit a single page. I kept running the above query to see the active connections and sure enough every time I ran the query the active connection count kept creeping up. It stops around 102 as of right now. And that's me as a user hitting a single page. I'm guessing this isn't normal? Does this indicate connections are lingering out there and not being dropped or closed?
I just made code changes recently adding Entity Framework. Wherever I'm grabbing data through EF, im using so with a using statement on the context. The rest of the app is sort of old and is using TableAdapters. I see in some places it's following the same pattern with using statements, other places Dispose is being called. I haven't had a chance to track down all the usages yet. Is this a good place to start looking? Anyone have any suggestions on how to track this 'leak' down? I'm not super knowledgeable with SQL so any help would be greatly appreciated!
Is it possible to set up somehow Microsoft SQL Server to run a stored procedure on regular basis?
Yes, in MS SQL Server, you can create scheduled jobs. In SQL Management Studio, navigate to the server, then expand the SQL Server Agent item, and finally the Jobs folder to view, edit, add scheduled jobs.
If MS SQL Server Express Edition is being used then SQL Server Agent is not available. I found the following worked for all editions:
USE Master
GO
IF EXISTS( SELECT *
FROM sys.objects
WHERE object_id = OBJECT_ID(N'[dbo].[MyBackgroundTask]')
AND type in (N'P', N'PC'))
DROP PROCEDURE [dbo].[MyBackgroundTask]
GO
CREATE PROCEDURE MyBackgroundTask
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- The interval between cleanup attempts
declare #timeToRun nvarchar(50)
set #timeToRun = '03:33:33'
while 1 = 1
begin
waitfor time #timeToRun
begin
execute [MyDatabaseName].[dbo].[MyDatabaseStoredProcedure];
end
end
END
GO
-- Run the procedure when the master database starts.
sp_procoption #ProcName = 'MyBackgroundTask',
#OptionName = 'startup',
#OptionValue = 'on'
GO
Some notes:
It is worth writing an audit entry somewhere so that you can see that the query actually ran.
The server needs rebooting once to ensure that the script runs the first time.
A related question is: How to run a stored procedure every day in SQL Server Express Edition?
Yes, if you use the SQL Server Agent.
Open your Enterprise Manager, and go to the Management folder under the SQL Server instance you are interested in. There you will see the SQL Server Agent, and underneath that you will see a Jobs section.
Here you can create a new job and you will see a list of steps you will need to create. When you create a new step, you can specify the step to actually run a stored procedure (type TSQL Script). Choose the database, and then for the command section put in something like:
exec MyStoredProcedure
That's the overview, post back here if you need any further advice.
[I actually thought I might get in first on this one, boy was I wrong :)]
Probably not the answer you are looking for, but I find it more useful to simply use Windows Server Task Scheduler
You can use directly the command sqlcmd.exe -S "." -d YourDataBase -Q "exec SP_YourJob"
Or even create a .bat file. So you can even 2x click on the task on demand.
This has also been approached in this HERE
I'll add one thing: where I'm at we used to have a bunch of batch jobs that ran every night. However, we're moving away from that to using a client application scheduled in windows scheduled tasks that kicks off each job. There are (at least) three reasons for this:
We have some console programs that need to run every night as well. This way all scheduled tasks can be in one place. Of course, this creates a single point of failure, but if the console jobs don't run we're gonna lose a day's work the next day anyway.
The program that kicks off the jobs captures print messages and errors from the server and writes them to a common application log for all our batch processes. It makes logging from withing the sql jobs much simpler.
If we ever need to upgrade the server (and we are hoping to do this soon) we don't need to worry about moving the jobs over. Just re-point the application once.
It's a real short VB.Net app: I can post code if any one is interested.
You could use SQL Server Service Broker to create custom made mechanism.
Idea (simplified):
Write a stored procedure/trigger that begins a conversation (BEGIN DIALOG) as loopback (FROM my_service TO my_service) - get conversation handler
DECLARE #dialog UNIQUEIDENTIFIER;
BEGIN DIALOG CONVERSATION #dialog
FROM SERVICE [name]
TO SERVICE 'name'
...;
Start the conversation timer
DECLARE #time INT;
BEGIN CONVERSATION TIMER (#dialog) TIMEOUT = #time;
After specified number of seconds a message will be sent to a service. It will be enqueued with associated queue.
CREATE QUEUE queue_name WITH STATUS = ON, RETENTION = OFF
, ACTIVATION (STATUS = ON, PROCEDURE_NAME = <procedure_name>
, MAX_QUEUE_READERS = 20, EXECUTE AS N'dbo')
, POISON_MESSAGE_HANDLING (STATUS = ON)
Procedure will execute specific code and reanable timer to fire again.
You can find fully-baked solution(T-SQL) written by Michał Gołoś called Task Scheduler
Key points from blog:
Pros:
Supported on each version (from Express to Enterprise). SQL Server Agent Job is not available for SQL Server Express
Scoped to database level. You could easiliy move database with associated tasks (especially when you have to move around 100 jobs from one enviromnent to another)
Lower privileges needed to see/manipulate tasks(database level)
Proposed distinction:
SQL Server Agent (maintenance):
backups
index/statistics rebuilds
replication
Task Scheduler (business processes):
removing old data
preaggregations/cyclic recalculations
denormalization
How to set it up:
get source code from section: "Do pobrania" - To download
(enabling broker/setting up schema tsks/configuration table + triggers + stored procedure)/setting up broker things)
set up configuration table [tsks].[tsksx_task_scheduler] to add new tasks (columns names are self-descriptive, sample task included)
Warning: Blog is written in Polish but associated source code is in English and it is easy to follow.
Warning 2: Before you use it, please make sure you have tested it on non-production environment.
Using Management Studio - you may create a Job (unter SQL Server Agent)
One Job may include several Steps
from T-SQL scripts up to SSIS Packages
Jeb was faster ;)
You should look at a job scheduled using the SQL Server Agent.