I am trying to create a directory in SQL server of the current SSIS packages we have which would include the jobs they are connected to and in turn which table they would load. I am wondering is it possible to see any queries used in an SSIS package such as either the execute sql task or the table which the ssis package is moving data to. Currently have been using the following query which gives me good info but hoping to expand it to include any queries which are used within side SSIS.
SELECT event_message_id ,
execution_path ,
package_name ,
package_path_full ,
event_name ,
message_source_name ,
package_path
FROM internal.event_messages (NOLOCK)
Related
session Connection full information
I am currently running the query and capturing session connection information which collects login, NT User, Host, Application/Program, Server and database information but i am also looking more granular detail which also tell me if it's executing SSIS package then which SSIS package, if it's tableau reports as application/program then which Database objects it's pulling the reports or other application/batch job doing any DML activities.
If i can't get all the above details but as a example if it's running SSIS package then which SSIS package running i can get it?
I am running currently couple different query as a daily sql job and storing the data into table.
SELECT ##ServerName AS SERVER
,NAME
,login_time
,last_batch
,getdate() AS DATE
,STATUS
,hostname
,program_name
,nt_username
,loginame
FROM sys.databases d
LEFT JOIN sys.sysprocesses sp ON d.database_id = sp.dbid
WHERE database_id NOT BETWEEN 0 AND 4
AND loginame IS NOT NULL
Any other way i can more details as we will be doing migration wanted to make sure?
Ex. If SSIS loading data into DB, we need to find that SSIS package
Currently I am able to identify all connections established to the instance and where they are coming from, but
we need to know which of those connections are feeding data and which are consuming data.
Thanks for all your help!
The simple answer is No.
When an application connects, it can have an ApplicationName passed in the connection details. You can retrieve it easily enough from the program_name column when you query sys.dm_exec_sessions. But it has to be configured in the connection string (in the SSIS data source) for you to be able to retrieve it.
Adding an ApplicationName to your connection strings is a good habit, not just for this, but for many tuning/monitoring reasons.
You can query the SSISDB catalog system view, catalog.executions, to find SSIS package executions on the server:
SELECT * FROM ssisdb.catalog.executions
In SQL Server Data Tools 2015, I would like to move data from a SQL Server 2012 database into a new access database(2005) and need to create the access table as part of the process. Can this be done all in one Execute SQL process under control flow. This will be part of a loop to run through a list of tables that need to be dynamically created and loaded into an empty access db.
I have created a connection manager and that is in the connection field for the access database and put the code into the SQL statement field under the general tab of the Execute SQL Task component.
Both databases are on my local machine.
"SELECT a.* into providers from OPENROWSET('SQLNCLI', 'Server=localhost;Trusted_Connection=yes;', 'SELECT * FROM newResults.dbo.providers') as a"
I get the following error:
SSIS package "C:\Users\chris\source\repos\Integration Services
Project5\Integration Services Project5\Package1.dtsx" starting. Error:
0xC002F210 at Execute SQL Task 2, Execute SQL Task: Executing the
query "SELECT a.* into providers from OPENQUERYSET('SQLN..." failed
with the following error: "Syntax error in FROM clause.". Possible
failure reasons: Problems with the query, "ResultSet" property not set
correctly, parameters not set correctly, or connection not established
correctly. Task failed: Execute SQL Task 2 SSIS package
"C:\Users\chris\source\repos\Integration Services Project5\Integration
Services Project5\Package1.dtsx" finished: Success.
The SQL contained in the Execute SQL Task is executed in the destination's context. The SELECT INTO FROM OPENQUERYSET statement is being passed to Access. Access doesn't have the OPENQUERYSET function and even if it did, your source is SQL Server, which Access doesn't know about unless you have made a connection to SQL Server in Access. Copy your SQL statement into Access and try to execute it and you'll see the same or a similar error. That's what the Execute SQL Task is doing.
Dynamic data is one of the more challenging problems in SSIS. The COZYROC tools include a lot of support for handling dynamic scenarios. Check out the videos for their Data Flow Task Plus for some ideas.
I am creating SSIS 2016 package which contains multiple data flow tasks. Each data flow task contains oledb source and oledb destination.
I have created two Execute SQL tasks one in the end and the other at the end of the entire process. I have named it Audit start and Audit End respectively.
I am trying to capture the following information in the audit table
PackageName, StartTime, EndTime, NumberOfRecords, MaxFinancialInstanceId
The audit start will insert the PackageName and startime while audit end will call the update stored procedure to add EndTime, NumberOfRecords, MaxFinancialInstanceId
I am aware that most of this information can be obtained by querying SSIDB
For e.g
SELECT execution_id,
package_name,
source_component_name,
destination_component_name,
rows_sent,
created_time
FROM [internal].[execution_data_statistics]
But in my case I would have some additional column information captured as well. Please note that my database and package would reside on different servers.
Is it wise to do the way I am manually doing or connect to the package server and get the information.
If the manual way is recommended then how do I capture the total records transmitted ?
I have around 4 servers(test1 , test2, test3, test4) and each server with 5 - 6 databases. Each server has around 60 SQL server agent jobs scheduled.
Each each job inside the servers has combination of sql statements and SSIS packages. Inside SQL statements, we have stored procs, which access tables from another servers.
Now i want to check whether server test2, is being used in sql server agent jobs of other servers.
i tried with the following query but not able to get any result
SELECT Job.name AS JobName, Job.enabled AS ActiveStatus,
JobStep.step_name AS JobStepName, JobStep.command AS JobCommand
FROM sysjobs Job INNER JOIN
sysjobsteps JobStep
ON Job.job_id = JobStep.job_id
WHERE JobStep.command LIKE '%test2%'
but when i check manually in test1 server , i can see test2 server being used in stored procs and ssis packages.
How can this be achieved ?
This is a little tricky. You will need to search the procedures, file system and SQL Server separately. The job table only contains the entry point, not the definition for each step.
SSIS Packages, stored in SQL Server
SSIS packages are stored in the msdb table sysssispackages. The definition is stored in the column packagedata. You'll need to cast the data, before you can read/search it.
-- How to read package data column.
SELECT TOP 10
[name],
packagedata AS [Before],
CAST(CAST(packagedata AS VARBINARY(MAX)) AS XML) AS [After]
FROM
sysssispackages
;
SSIS Packages, stored on the file system
These are XML text files. You can search using your favorite method. If you want to build your own solution .Net has some great file handling ablities.
Stored Procedures
There are several methods, all of which return data from the system tables. One approach is to use the object definition function.
-- Returns definition of stored procedure.
SELECT
OBJECT_DEFINITION(OBJECT_ID(N'[Your-Schema].[Your-Procedure]'))
;
If you want to bring this altogether
Using the table sysjobsteps, you could build a package/app that searches for a given server name. The final product would need to support all three search methods. You might be able to use the subsystem column, to choose which search method should be used for each step.
I have an SSIS package which imports the data into the SQL Server 2008 database. I have set up the schedule job in the SQL Server Agent to run that package. When I check the history, I could only see whether the job ran successfully or not. I could not see other messages apart from that.
I would like to know how many records are imported whenever the job is executed. How can I monitor that? Should I use the additional components in SSIS package or set some configurations in SQL Server Agent Job Setup?
I found some logging facilities in SQL Server Agent Job Setup but I am not sure it can fulfill my requirements or not.
If you are just interested in knowing the columns being processed and not interested with the info for further use, one possible option is making use of the SSIS logging feature. Here is how it works for data flow tasks.
Click on the SSIS package.
On the menus, select SSIS --> Logging...
On the Configure SSIS Logs: dialog, select the provider type and click Add. I have chosen SQL Server for this example. Check the Name checkbox and provide the data source under Configuration column. Here SQLServer is the name of the connection manager. SSIS will create a table named dbo.sysssislog and stored procedure dbo.sp_ssis_addlogentry in the database that you selected. Refer screenshot #1 below.
If you need the rows processed, select the checkbox OnInformation. Here in the example, the package executed successfully so the log records were found under OnInformation. You may need to fine tune this event selection according to your requirements. Refer screenshot #2 below.
Here is a sample package execution within data flow task. Refer screenshot #3 below.
Here is a sample output of the log table dbo.sysssislog. I have only displayed the columns id and message. There are many other columns in the table. In the query, I am filtering the output only for the package named 'Package1' and the event 'OnInformation'. You can notice that records with ids 7, 14 and 15 contain the rows processed. Refer screenshot #4 below.
Hope that helps.
Screenshot #1:
Screenshot #2:
Screenshot #3:
Screenshot #4:
use the below procedure for getting SSIS errors with execution id
CREATE PROCEDURE [dbo].[get_ssis_status] #EXECUTION_ID INT\n
AS
BEGIN
SELECT o.operation_id EXECUTION_ID
,convert(datetimeoffset,OM.message_time,109) TIME
,D.message_source_desc ERROR_SOURCE
,OM.message ERROR_MESSAGE
,CASE ex.STATUS
WHEN 4 THEN 'Package Failed'
WHEN 7 THEN CASE EM.message_type
WHEN 120 THEN 'package failed'
WHEN 130 THEN 'package failed' ELSE 'Package Succeed'END
END AS STATUS
FROM SSISDB.CATALOG.operation_messages AS OM
INNER JOIN SSISDB.CATALOG.operations AS O ON O.operation_id = OM.operation_id
INNER JOIN SSISDB.CATALOG.executions AS EX ON o.operation_id = ex.execution_id
INNER JOIN (VALUES (- 1,'Unknown'),(120,'Error'),(110,'Warning'),(130,'TaskFailed')) EM(message_type, message_desc) ON EM.message_type = OM.message_type
INNER JOIN (VALUES
(10,'Entry APIs, such as T-SQL and CLR Stored procedures')
,(20,'External process used to run package (ISServerExec.exe)')
,(30,'Package-level objects')
,(40,'Control Flow tasks')
,(50,'Control Flow containers')
,(60,'Data Flow task')
) D(message_source_type, message_source_desc) ON D.message_source_type = OM.message_source_type
WHERE ex.execution_id = #EXECUTION_ID
AND OM.message_type IN (120,130,-1);
END
Here's another approach for when SQL Server job history is not showing output from SSIS packages: use DTEXEC command lines.
(Upside: this approach puts the job's output where anyone else supporting it would expect to find it: in job history.
Downside for big packages: if you have a long SSIS package, with lots of tasks or components, and lots of output, then the job history will split package output into many lines of job history, making the approach in the previous answer--logging to a table--easier to read.)
To show SSIS package output in the job's View History:
(1) Change the job steps from type "SQL Server Integration Services Package", to "Operating system (CmdExec)",
(2) Use DTEXEC command lines, to execute the packages.
Example of command line:
DTExec /DTS "\MSDB\myPkgName" /DECRYPT pkgPass /MAXCONCURRENT " -1 " /CHECKPOINTING OFF
Note that if the SSIS package requires 32-BIT execution (true for exporting to Excel, for example), then use the DTEXEC utility in "Program Files (x86)" by fully qualifying it. Example, where the SQL Server application was installed on an "E:" drive, and where SQL Server 2014 is being used:
"E:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\DTExec.exe" /DTS "\MSDB\myPkgName" /DECRYPT pkgPass /MAXCONCURRENT " -1 " /CHECKPOINTING OFF
If your SSIS packages are in the file system (as ".dtsx" files), then replace "/DTS" with "/FILE".
If your SSIS packages were placed in SSISDB (using the "project deployment model", which is available starting with SQL Server 2012, instead of the older "package deployment model"), then replace "/DTS" with "/ISSERVER".
Next, go into the job step's "Advanced" page, and make sure that the box is checked for "Include step output in history".
Lastly, consider your job step's "Run as": if your job steps "Run as" were already set to a proxy, on job steps of type "SQL Server Integration Services Package", then you already made that proxy active to the subsystem "SQL Server Integration Services Package". Now, to do command lines like the above, check the proxy's properties, and make sure it is also active to the subsystem "Operating system (CmdExec)".
MSDN reference: SSIS Output on Sql Agent history
If you have deployed the package to the database's Integration Services Catalog (rather than load it from a file system) you can easily get detailed reporting.
Open the catalog node in SQL Server Management Studio, right click the Package name, select Reports | Standard Reports | All Executions and see details about every step of the job and its subcomponents, including records imported.