I am looking through the SSIS DB Project Catalog to find the Begin and End datetime for every SSIS package called from a Master package. Is there any query to conduct this?
After looking at [internal].[executions], and SSISDB.catalog.operation_messages, etc
Running this query does not work, if I there is a master package calling Child Packages.
select start_time,end_time,*
from catalog.executions cc order by cc.start_time desc
I am trying to find the begin and end time for every child pacakge.
The CATALOG.EXECUTABLE_STATISTICS DMV logs execution statistics for components within a package, including execute package tasks. The START_TIME and END_TIME columns store the time that the component began and completed execution. The EXECUTION_DURATION column holds the time that an executable, in this case the child packages, took to execute in milliseconds. This can of course be converted to seconds, minutes, etc. depending on what you need. While this has a column for the execution path of the component within the parent package, it doesn't have a column for the direct name of component, thus CATALOG.EXECUTABLES is included for the EXECUTABLE_NAME, and this DMV can be omitted if you only want to view the execution path (EXECUTION_PATH column) instead. CATALOG.EXECUTIONS has columns for the folder and project name, and you can join to this to apply filters for the specific project and folder that the package is located in. You can also apply a filter on the EXECUTION_ID column to only view details for a specific execution. Executing a package at the basic logging level with allow execution details to be logged for the components.
SELECT
EX.FOLDER_NAME,
EX.PROJECT_NAME,
E.EXECUTABLE_NAME,
EX.PACKAGE_NAME,
ES.START_TIME AS ComponentStartTime,
ES.END_TIME AS ComponentEndTime,
EX.start_time AS PackageStartTime,
EX.end_time AS PackageEndTime,
ES.EXECUTION_DURATION AS ComponentExecutionTimeInMilliseconds
FROM SSISDB.CATALOG.EXECUTIONS EX
INNER JOIN SSISDB.CATALOG.EXECUTABLES E on EX.EXECUTION_ID = E.EXECUTION_ID
INNER JOIN SSISDB.CATALOG.EXECUTABLE_STATISTICS ES on E.EXECUTABLE_ID = ES.EXECUTABLE_ID AND EX.EXECUTION_ID = ES.EXECUTION_ID
--PACKAGE_NAME- parent package
WHERE E.PACKAGE_NAME = 'Package Name.dtsx' AND EX.PROJECT_NAME = 'Project Name'
AND EX.FOLDER_NAME = 'Folder Name'
Related
I'm testing SnowFlake. To do this I created an instance of SnowFlake on GCP.
One of the tests is to try the daily load of data from a STORAGE INTEGRATION.
To do that I had generated the STORAGE INTEGRATION and the stage.
I tested the copy
copy into DEMO_DB.PUBLIC.DATA_BY_REGION from #sg_gcs_covid pattern='.*data_by_region.*'
and all goes fine.
Now it's time to test the daily scheduling with the task statement.
I created this task:
CREATE TASK schedule_regioni
WAREHOUSE = COMPUTE_WH
SCHEDULE = 'USING CRON 42 18 9 9 * Europe/Rome'
COMMENT = 'Test Schedule'
AS
copy into DEMO_DB.PUBLIC.DATA_BY_REGION from #sg_gcs_covid pattern='.*data_by_region.*';
And I enabled it:
alter task schedule_regioni resume;
I got no errors, but the task don't loads data.
To resolve the issue i had to put the copy in a stored procedure and insert the call of the storede procedure instead of the copy:
DROP TASK schedule_regioni;
CREATE TASK schedule_regioni
WAREHOUSE = COMPUTE_WH
SCHEDULE = 'USING CRON 42 18 9 9 * Europe/Rome'
COMMENT = 'Test Schedule'
AS
call sp_upload_c19_regioni();
The question is: this is a desired behavior or an issue (as I suppose)?
Someone can give to me some information about this?
I've just tried ( but with storage integration and stage on AWS S3) and it works fine also using copy command inside sql part of the task, without calling a stored procedure.
In order to start investigating the issue, I would check following info (maybe for debugging I would create the task scheduling it every few minutes):
check task_history and verify executions
select *
from table(information_schema.task_history(
scheduled_time_range_start=>dateadd('hour',-1,current_timestamp()),
result_limit => 100,
task_name=>'YOUR_TASK_NAME'));
if previous step is successfull, check copy_history and verify the input file name , target table and number of records/errors are the expected ones
SELECT *
FROM TABLE (information_schema.copy_history(TABLE_NAME => 'YOUR_TABLE_NAME',
start_time=> dateadd(hours, -1, current_timestamp())))
ORDER BY 3 DESC;
Check if the results are the same you get when the task with sp call is executed.
Please also confirm that you are loading new files not yet loaded into your table with COPY command (otherwise you need to specify FORCE = TRUE parameter in the copy command or remove metadata information truncating your target table to reload the same files).
I have a list of jobs to run multiple ssis packages. I have a big list, I want to know if it is possible to run a tsql to msdb and get the information of the packages in the steps with the connection strings.
Super old thread but the below code is slightly adapted and much more useful as it shows package level details.
select
prj.name as 'ProjectName'
,pa.name as 'SSISPackageName'
,op.parameter_name as 'ParmaterName'
,op.design_default_value as 'ConnectionString'
from
catalog.object_parameters op
join catalog.projects prj
on op.project_id = prj.project_id
join [catalog].[packages] pa
on pa.project_id = prj.project_id
where op.parameter_name like '%ConnectionString%'
If your packages are stored in the SSISDB Catalog, then you would need to query on SSIDB database to get the actual connection strings. Here is what I would suggest, use the query that Rodrigo A has provided and tweak the column 'Command' further to get the packages name from it by using string functions and have the Step ID and Package Name as output. use the list of packages that you have obtained and put it in the IN clause of the following query and run it against SSISDB -
select prj.name as 'ProjectName'
,op.object_name as 'SSISPackageName'
,op.parameter_name as 'ParmaterName'
,op.design_default_value as 'ConnectionString'
from catalog.object_parameters op
join catalog.projects prj
on op.project_id = prj.project_id
where op.parameter_name like '%ConnectionString%'
and op.object_name in (
--put the package name list here
);
You can then join on the result sets, to get the step Id, PackageName and the connection strings in that package.
Run this query over the server you want to check:
SELECT [sJSTP].[step_id], Name, sJSTP.Command FROM [msdb].[dbo].[sysjobs] [sJOB]
LEFT JOIN [msdb].[dbo].[sysjobsteps] AS [sJSTP]
ON [sJOB].[job_id] = [sJSTP].[job_id]
AND [sJOB].[start_step_id] = [sJSTP].[step_id]
ORDER BY Name, step_id
Cheers!
What I need to find is the procedure of recreating some table, what data sources were used, which scripts if any &c. So is it possible to differentiate somehow, maybe in system views or similar, if the table was created manually or by query and if the data was imported from external data or from already existing table/view in the database? I already know who created and when. I’ve pretty much screened whole database without results and now I am looking for hints in metadata.
If the table was created recently, you can glean information from the default trace. The query below will list object created and altered events. The default trace is a rollover trace so forensic information will be limited based on activity.
SELECT
trace.DatabaseName
,trace.ObjectName
,te.name AS EventName
,tsv.subclass_name
,trace.EventClass
,trace.EventSubClass
,trace.StartTime
,trace.EndTime
,trace.NTDomainName
,trace.NTUserName
,trace.HostName
,trace.ApplicationName
,trace.Spid
FROM (SELECT REVERSE(STUFF(REVERSE(path), 1, CHARINDEX(N'\', REVERSE(path)), '')) + N'\Log.trc' AS path
FROM sys.traces WHERE is_default = 1) AS default_trace_path
CROSS APPLY fn_trace_gettable(default_trace_path.path, DEFAULT) AS trace
JOIN sys.trace_events AS te ON
trace.EventClass=te.trace_event_id
JOIN sys.trace_subclass_values AS tsv ON
tsv.trace_event_id = EventClass
AND tsv.subclass_value = trace.EventSubClass
WHERE te.name IN(N'Object:Altered', N'Object:Created')
AND tsv.subclass_name = 'Commit'
ORDER BY trace.StartTime;
First of all I have never attempted something like this in SSIS and I am very new to SSIS package development.
I need to build a component in my package that will run through a table of data (say 80 rows) and set a field titled DisplayOrder to the auto incremented number. The catch is that one of the records HAS to be set to 0 and then the rest of he records set to the auto incremented number.
In regards to code, I am not even sure what code to attach to this question or even what screenshots.
I finally figured it out and there is no need for a loop.
Create a SQL Task to clear the linked Table.
Script Used
DELETE FROM [Currency].[ExchangeRates]
Create a SQL Task to clear the main table.
Script Used
DELETE FROM [Currency].[CurrencyList]
Load the values into the main table.
Actions Used
Load values from XML Source
Dump values to [ExchangeRates] Table
Create a SQL Task to load the Values from the main table to the linked table.
Script Used
INSERT INTO [Currency].[CurrencyList] (CurrencyCode, CurrencyName, ExchangeRateID, DisplayOrder) SELECT [er].[TargetCurrency] AS [CurrencyCode], [er].[TargetName] AS [CurrencyName], [er].[ID] AS [ExchangeRateID], ROW_NUMBER() OVER (ORDER BY [ER].[TargetName]) AS [DisplayOrder] FROM [Currency].[ExchangeRates] AS [er] ORDER BY [CurrencyName]
Create a SQL Task to load a new record to the main table for use as DisplayOrder 0.
Script Used
INSERT INTO [Currency].[ExchangeRates] ([Title], [Link], [Description], [PubDate], [BaseCurrency], [TargetCurrency], [TargetName], [ExchangeRate]) VALUES ('1 USD = 1 USD','http://www.floatrates.com/usd/usd/','1 U.S. Dollar = 1 U.S. Dollar',(SELECT TOP 1 [PubDate] FROM [Currency].[ExchangeRates]),'USD','USD','United States Dollar','1')
Create a SQL Task to reference the newly created record from the main table.
Script Used
INSERT INTO [Currency].[CurrencyList] (CurrencyCode, CurrencyName, ExchangeRateID, DisplayOrder) SELECT [er].[TargetCurrency] AS [CurrencyCode], [er].[TargetName] AS [CurrencyName], [er].[ID] AS [ExchangeRateID], 0 AS [DisplayOrder] FROM [Currency].[ExchangeRates] AS [er] WHERE [er].[TargetCurrency] = 'USD'
We have different SSIS package that we use in daily tasks (updates, ETL...) and we have a kind of complicated structure, where a package calls different other packages. And there are primarily about 10 principal jobs that call secondary ones. So these 10 jobs are always on success even if a step fails so it wouldn't block other executions. Although we would like to retrieve the steps (and their status) that are related to these jobs via a SQL Query but we couldn't join between the steps and their calling jobs and at the same time retrieve the status (The step status in this case and not the jobs).
I searched a lot on the net and i always find a script that joins the steps and calling jobs without the status or steps and status without knowing which job is calling...
(for example this link and this one )
so to sum it all up, we are trying to do a Query where we can join the jobs, their Status and their parent job.
Any help in this matter would be really appreciated and thanks in advance.
EDIT
Thanks to the link in #BaconBits comment i was able to create a query joining three tables (msdb.dbo.sysjobsteps, msdb.dbo.sysjobs, msdb.dbo.sysjobhistory) that retrieves something like the following:
Job_name1 Step_name1 Job1_status
Job_name1 Step_name2 Job1_status
Job_name1 Step_name3 Job1_status
Job_name2 Step_name1 Job2_status
Job_name2 Step_name2 Job2_status
But I still couldn't retrieve the step status (which is what i need in this case since the job outcome is always on success even if a step fails)
Query:
select j.name, s.step_name,
CASE WHEN s.last_run_outcome=0 THEN 'Failed'
WHEN s.last_run_outcome=1 THEN 'Success'
WHEN s.last_run_outcome=2 THEN 'Retry'
WHEN s.last_run_outcome=3 THEN 'Canceled'
END
,h.run_date, s.output_file_name
from msdb.dbo.sysjobsteps s
inner join msdb.dbo.sysjobs j on s.job_id=j.job_id
inner join msdb.dbo.sysjobhistory h
on h.job_id=j.job_id or s.step_id=h.step_id
--where j.name like '%Dem%'
order by h.run_date, j.name
Thank you #BaconBits and anyone for any further help.