I've got a SQL Job set up to run a SSIS package. The SSIS package has a number of parameters. I want to see what the parameters look like when the job is sending the parameter list.
I have an issue where the package works in dev, but not in prod, as it says it has a mismatched quote. I have a few parameters in a row that have a unc path. Once I remove the trailing backslash, it works, but I want to see what it's sending.
Is there a way to log or see how the parameter list is built from a job running a SSIS package?
Thank you.
Assuming you are using SSIS Catalog.
If you navigate to the SSIS project inside the Integration Service Catalog, and run the report Standard Report->All Executions. You can check the last execution run and inside that report will list all the parameters.
Or check the SSISDB tables.
SELECT TOP 100 *
FROM SSISDB.internal.execution_parameter_values
WHERE execution_id = <execution_id>
When running the following query, it showed me SQL was sending both, prod and dev parameters, so it was not using all the parameters set in the job.
It was using some parameters from the Catalog > Configure section mixed with ones from the job.
I ended up having to ensure the parameters were set at the Catalog level: right click the project > Configure. (Typically the packages is saved with all dev parameters, then changed on the prod server.)
Once I had the parameters set at the config level, as well as the in the job, everything worked. However, I think there is a bug issue with the software. I can't tell if the parameter is coming from the Sql job, or the catalog configure area. I do have a case where I know at least one is coming from a job area.
In case it's helpful, just put in a partial project name and this will show you the parameters for the latest execution of the project you are working on. Most of this code is not original to me, I just adapted it.
USE SSISDB
GO
DECLARE #SSISprgName VARCHAR(100) = 'ENTER PARTICAL PROJECT NAME HERE'
IF OBJECT_ID('tempdb..#t1') IS NOT NULL DROP TABLE #t1
SELECT TOP 1 execution_id, process_id,start_time, *
FROM catalog.executions AS e --Where end_time is null
WHERE project_name LIKE '%'+ #SSISprgName +'%'
ORDER BY e.start_time DESC
IF OBJECT_ID('tempdb..#t1') IS NOT NULL DROP TABLE #t1
SELECT TOP 1 e.execution_id
INTO #t1
FROM catalog.executions AS e --Where end_time is null
WHERE project_name LIKE '%'+ #SSISprgName +'%'
ORDER BY e.start_time DESC
SELECT TOP 100 *
FROM SSISDB.internal.execution_parameter_values
WHERE execution_id IN (SELECT execution_id FROM #t1)
Related
Is it possible to get exactly the same parameters as shown in the All execution Overview report (see the printscreen below)?
I was trying to use the table [internal].[execution_parameter_values] from SSISDB and filter it via execution_id, nevertheless it returns much more parameters than in the report. I have also tried to filter it with table attribute "value_set", "object_type", etc. but still it did not return the same list as in the report.
Reference:
https://learn.microsoft.com/en-us/sql/integration-services/system-views/views-integration-services-catalog?view=sql-server-2017
execution_parameter_value:
Displays the actual parameter values that
are used by Integration Services packages during an instance of
execution.
Whenever the package is executed, records are inserted into that table. You need to determine the execution_id that you want to filter on.
You can get that from [catalog].[executions] in the SSIS DB. Filter based on your project or package and when it was executed.
Or you will also see that in the execution overview report as "Operation ID":
You can then filter based on that value:
SELECT * FROM [internal].[execution_parameter_values]
WHERE [execution_id] = 16529
Overview report in the SSIS catalog shows only TOP 25 used parameters sorted by parameter_name ASC.
Also, it is needed to filter out the records with parameter_name without "." character.
So the result T-SQL script would be:
SELECT TOP 25
[parameter_name]
,[parameter_value]
,[parameter_data_type]
FROM [SSISDB].[internal].[execution_parameter_values]
WHERE execution_id = #execution_id AND parameter_name not like '%.%'
ORDER BY parameter_name
How do we configure our TFS gated build validation to catch parser validation errors?
For example, we had a stored procedure with an invalid statement that kept passing the gated TFS check-in XAML build validation and even passed SSDT publish to a SQL Server 2008R2 database for two whole years!
CREATE PROCEDURE This_Should_Fail
AS BEGIN
DECLARE #TableVariable TABLE(
ID INT IDENTITY(1, 1),
Name VARCHAR(MAX)
)
SELECT DISTINCT TOP 1 Name
FROM #TableVariable
ORDER BY ID
END
When I run this statement I get the appropriate error, however when I run the full version in TFS there is no error!
Msg 145, Level 15, State 1, Procedure This_Should_Fail, Line 7
ORDER BY items must appear in the select list if SELECT DISTINCT is specified.
But the error was never encountered until recently when the definition changed to be like this, which doesn't even throw any errors when run!
CREATE PROCEDURE This_Should_Fail
AS BEGIN
DECLARE #TableVariable TABLE(
ID INT IDENTITY(1, 1),
Name VARCHAR(MAX),
ForeignKeyID INT
)
--create some sample data
INSERT INTO #TableVariable(Name, ForeignKeyID)
VALUES ('Obj1', 999), ('Obj2', 999), ('Obj3', 0), ('Obj4', 0)
DECLARE #ForeignKeyID INT = 999--some lookup here
SELECT DISTINCT TOP 1 Name
FROM #TableVariable
WHERE ForeignKeyID = #ForeignKeyID
END
And it took a whole month after that change was made before the change in the TFS branch failed to deploy anywhere. In both cases, there was no ordering by all selected columns.
This is confusing me even more because neither version in TFS (which I can't include because it's my company's IP) throws any error even when manually run in SSMS 2014!
Thanks.
Gated check-in is a form of continuous integration build. In TFS, it creates a shelveset containing the code that's being validated, then runs a build of that code. Only if that code builds successfully and all configured unit tests pass does the code actually get checked in.
In other words, the validation is controlled by your build/test directly not by TFS. If the error is not caught by your build or not test out by your test. Then it will effect after actually deployed to a server.
One way to do this is catching the log info of your build and then use Logging Commands and exit 1 to fail the build. More details you could refer this question: How to fail the build from a PowerShell task in TFS 2015 (for vNext Build)
I’m doing the following query on a remote database:
DECLARE #SQL varchar(max) =
'SELECT ts, value
FROM history
WHERE name = ''SOME_ID''
EXEC (#SQL) AT SOME_LINKED_SERVER
So the expected output is like that:
ts value
----------------
ts1 value1
ts2 value2
… …
I’m doing this query for almost 100 different names and am willing to save a different CSV for each output. I know I can do it manually by clicking with the right button on the query’s output and selection “Save Result As...”, but it would take too long, specially because each query takes about 10 minutes to finish.
So I’d like to do it automatically, making the procedure export all the different CSV’s after getting the data. My ideia is to loop the search through an array of names, do the query and export the output as a CSV.
How can I do that? Before trying to loop through the array, I'm already struggling to output a CSV for a single query result.
Why do it one name at a time when you can do this:
SELECT [ts], [value], [name]
FROM [SOME_LINKED_SERVER].[database_name].[dbo].[history] H (NOLOCK)
WHERE [name] IN (<name list>)
If you want to store the result on the local end, then insert the data into a local table and work with the data there. Like so:
SELECT [ts], [value], [name]
INTO [local_history]
FROM [SOME_LINKED_SERVER].[database_name].[dbo].[history] H (NOLOCK)
WHERE [name] IN (<name list>)
SELECT * FROM [local_history] -- WHERE [name] = 'SOME_ID'
Then export, either by using Save Results As..., or copy and pasting the results into Excel, and doing a Save as... (F12).
Even better: if you have SSIS installed, use that to build a package that runs the query, and does the export for you. SSIS can even loop through the list of names if it comes down to that. If you don't have SSIS installed, install it, it comes with SQL Server. SSIS can be made to automate this process entirely.
I've inherited a database recently which contains thousands of stored procedures and functions, however most of them are deprecated and no longer in use.
I've started adding a piece of code to the stored procedures one at time to notify me if they run, but this process is really quite manual.
Is there any way to start an audit, and see which stored procedures run in the next month or two without adding a piece of code to each stored procedure manually?
Thanks,
Eric
I believe you need to be on SQL Server 2005 SP2 or higher. In prior versions of SQL Server, the OBJECT_NAME function only accepts a parameter for object_id.
Hopefully this should work for you:
SELECT DB_NAME(dest.[dbid]) AS 'databaseName'
, OBJECT_NAME(dest.objectid) AS 'procName'
, MAX(deqs.last_execution_time) AS 'last_execution'
FROM sys.dm_exec_query_stats AS deqs
CROSS APPLY sys.dm_exec_sql_text(deqs.sql_handle) AS dest
WHERE dest.[TEXT] LIKE '%yourTableName%' -- replace
And dest.[dbid] = DB_ID() -- exclude ad-hocs
GROUP BY DB_NAME(dest.[dbid])
, OBJECT_NAME(dest.objectid)
ORDER BY databaseName
, procName
OPTION (MaxDop 1);
So I have a stored procedure in SQL Server. I've simplified its code (for this question) to just this:
CREATE PROCEDURE dbo.DimensionLookup as
BEGIN
select DimensionID, DimensionField from DimensionTable
inner join Reference on Reference.ID = DimensionTable.ReferenceID
END
In SSIS on SQL Server 2012, I have a Lookup component with the following source command:
EXECUTE dbo.DimensionLookup WITH RESULT SETS (
(DimensionID int, DimensionField nvarchar(700) )
)
When I run this procedure in Preview mode in BIDS, it returns the two columns correctly. When I run the package in BIDS, it runs correctly.
But when I deploy it out to the SSIS catalog (the same server the database is on), point it to the same data sources, etc. - it fails with the message:
EXECUTE statement failed because its WITH RESULT SETS clause specified 2 column(s) for result set number 1, but the statement sent
3 column(s) at run time.
Steps Tried So Far:
Adding a third column to the result set - I get a different error, VS_NEEDSNEWMETADATA - which makes sense, kind of proof there's no third column.
SQL Profiler - I see this:
exec sp_prepare #p1 output,NULL,N'EXECUTE dbo.DimensionLookup WITH RESULT SETS ((
DimensionID int, DimensionField nvarchar(700)))',1
SET FMTONLY ON exec sp_execute 1 SET FMTONLY OFF
So it's trying to use FMTONLY to get the result set data ... needless to say, running SET FMTONLY ON and then running the command in SSMS myself yields .. just the two columns.
SET NOTCOUNT ON - Nothing changed.
So, two other interesting things:
I deployed it out to my local SQL 2012 install and it worked fine, same connections, etc. So it may be a server / database configuration. Not sure what if anything it is, I didn't install the dev server and my own install was pretty much click through vanilla.
Perhaps the most interesting thing. If I remove the join from the procedure's statement so it just becomes
select DimensionID, DimensionField from DimensionTable
It goes back to just sending 2 columns in the result set! So adding a join, without adding any additional output columns, ups the result set to 3 columns. Even if I add 6 more joins, just 3 columns. So one guess is its some sort of metadata column that only gets activated when there's a join.
Anyway, as you can imagine, it's driving me kind of mad. I have a workaround to load the data into a temp table and just return that, but why won't this work? What extra column is being sent back? Why only when I add a join?
Gah!
So all credit to billinkc: The reason is because of a patch.
In Version 11.0.2100.60, SSIS Lookup SQL command metadata is gathered using the old SET FMTONLY method. Unfortunately, this doesn't work in 2012, as the Books Online entry on SET FMTONLY helpfully notes:
Do not use this feature. This feature has been replaced by sp_describe_first_result_set.
Too bad they didn't follow their own advice!
This has been patched as of version 11.0.2218.0. Metadata is correctly gathered using the sp_describe_first_result_set system stored procedure.
This can happen if the specified WITH results set in SSIS identifies that there are more columns than being returned by the stored proc being called. Check your stored proc and ensure that you have the correct number of output columns as the WITH results set.