I have the following stored procedure which work as expected when run it run it using the EXEC command below in SSMS.
the CustomerAlternateKey column in both the [dbo].[DimCustomer] and dbo.TargetTable tables are encrypted using Always Encrypted Deterministic encryption using the Azure Key Vault option.
create or alter procedure uspMoveToTarget #keyvalue nvarchar(15)
as
begin
insert into TargetTable(CustomerKey, CustomerAlternateKey)
select
CustomerKey,
CustomerAlternateKey
from [dbo].[DimCustomer]
where CustomerAlternateKey = #keyvalue
end
declare #variable nvarchar(15) = 'AW00011000'
exec uspMoveToTarget #variable -- works
I have created a ADO.Net connection in SSIS with the Column Encryption Setting = "Enabled". I have added an "Execute SQL Task" and referenced the ADO.Net connection i just mentioned and added the value above into a package variable.
When i run my SSIS package i get the following error:
Error: 0xC002F210 at Execute SQL Task, Execute SQL Task: Executing the query "uspMoveToTarget" failed with the following error: "Failed to decrypt a column encryption key. Invalid key store provider name: 'AZURE_KEY_VAULT'. A key store provider name must denote either a system key store provider or a registered custom key store provider. Valid system key store provider names are: 'MSSQL_CERTIFICATE_STORE', 'MSSQL_CNG_STORE', 'MSSQL_CSP_PROVIDER'. Valid (currently registered) custom key store provider names are: . Please verify key store provider information in column master key definitions in the database, and verify all custom key store providers used in your application are registered properly.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
Task failed: Execute SQL Task
I am not sure where to go from here as everything works as expected in SSMS. Are the additional configurations i need to make in SSIS for this to work?
Thanks in advance
Related
I'm using Azure SQL Databases.
I'm trying to create an external data source for a database 1 to a database 2 in the same server, so both the databases are located on the same server. After creating the external data source for the database 1 I create an external table in database 1 that have the same definition as a table that exist in database 2. After creating the table I run the following query on database 1:
SELECT * FROM [dbo].[SomeTableName]
The query results in an error with the following message
Msg 46832, Level 16, State 3, Line 1
An error occurred while establishing connection to remote data source: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user '[credential_identity used to create db scoped cred]'.[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user '[credential_identity used to create db scoped cred]'.
It seems like a credential issue at first glance, but, to clarify on that, all databases on the server use the same credentials to access them.
The db scoped credential that I used, was one that was already available for me to use and the credential_id seems to be the correct one as it is the same one I use to access all of the databases. I did not create the scoped credential, I just use one that was created by another person for the same purpose, but, to connect to a different db. The query I used to create the external data source is the following:
CREATE EXTERNAL DATA SOURCE <database 2> WITH
(TYPE = RDBMS,
LOCATION = '<server_name>.database.windows.net',
DATABASE_NAME = '<database 2>',
CREDENTIAL = <scoped_credential>,
) ;
And after that I created an external table that uses the external data source created with the following query:
CREATE EXTERNAL TABLE [dbo].[someTableName]
(
<... respective definitions>
)
WITH
( DATA_SOURCE = MyElasticDBQueryDataSrc)
This seems to be like an obvious error, I think I would probably have to create a new scoped credential, but right now I'm trying to solve the issue on my own, and trying this implies asking for permission from supervisor and following defined processes for the matter. Just trying to expose my problem to the public to see if I'm missing something important. Thanks!
I got similar error when I tried to reproduce but after checking it was because of wrong credentials.
Msg 46832, Level 16, State 3, Line 31
An error occurred while establishing connection to remote data source: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user 'username.[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user 'username'.
CREATE MASTER KEY ENCRYPTION BY PASSWORD='pratik#2961';
-- enter credetials to login to Azure SQL Database
CREATE DATABASE SCOPED CREDENTIAL mycred12
WITH
IDENTITY = 'username',
SECRET = 'password';
-- CREATE EXTERNAL DATASOURCE
CREATE EXTERNAL DATA SOURCE mydatasource1 WITH
(
TYPE = RDBMS,
LOCATION = N'your_servername.database.windows.net',
DATABASE_NAME = N'databasename',
CREDENTIAL = mycred12
)
-- CREATE EXTERNAL TABLE
CREATE EXTERNAL TABLE AzureDBtable1234
(
AirlineCode nvarchar(MAX),
Name nvarchar(MAX),
Date nvarchar(MAX)
) -- must ensure data structure matches the one on SQL Database
WITH(
DATA_SOURCE = mydatasource12,
SCHEMA_NAME = 'dbo',
OBJECT_NAME = 'Airlines'
)
SELECT * FROM AzureDBtable1234
OUTPUT
I have an SSIS Package, which contains multiple flows.
Each flow is responsible for creating a "staging" table, which gets filled up after creation.
These tables are global temporary tables.
I added 1 extra flow (I did not make the package) which does exactly as stated above, for another table. However, for some reason, the package fails intermittently on this flow, while it is exactly the same as others, besides some table names.
The error that keeps popping up:
Update - Insert Data Flow:Error: SSIS Error Code DTS_E_OLEDBERROR. An
OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is
available. Source: "Microsoft SQL Server Native Client 11.0"
Hresult: 0x80004005 Description: "Unspecified error". An OLE DB
record is available. Source: "Microsoft SQL Server Native Client
11.0" Hresult: 0x80004005 Description: "The metadata could not be determined because statement 'select * from
'##TmpMcsConfigurationDeviceHistory86B34BFD041A430E84CCACE78DA336A1'' uses a temp table.".
Creation expression:
"CREATE TABLE " + #[User::TmpMcsConfigurationDeviceHistory] + " ([RecId] [bigint] NULL,[DataAreaID] [nvarchar](4) COLLATE database_default NULL,[Asset] [bigint] NULL,[Code] [nvarchar](255) COLLATE database_default NULL,[Configuration] [bigint],[StartdateTime] [datetime] NULL,[EndDateTime] [datetime] NULL)
"
Parsed expression (=evaluated):
CREATE TABLE ##TmpMcsConfigurationDeviceHistory764E56F088DC475C9CC747CC82B9E388 ([RecId] [bigint] NULL,[DataAreaID] [nvarchar](4) COLLATE database_default NULL,[Asset] [bigint] NULL,[Code] [nvarchar](255) COLLATE database_default NULL,[Configuration] [bigint],[StartdateTime] [datetime] NULL,[EndDateTime] [datetime] NULL)
Using WITH RESULT SETS to explicitly define the metadata will allow SSIS to skip the sp_describe_first_result_set step and use the metadata that you define. The upside is that you can use this to get SSIS to execute SQL that contains a temporary table (for me, that performance helped a lot); the downside is, you have to manually maintain and update this if anything changes.
Query sample (stored procedure:)
EXEC ('dbo.MyStoredProcedure')
WITH RESULT SETS
(
(
MyIntegerColumn INT NOT NULL,
MyTextColumn VARCHAR(50) NULL,
MyOtherColumn BIT NULL
)
)
Query sample (simple SQL:)
EXEC ('
CREATE TABLE #a
(
MyIntegerColumn INT NOT NULL,
MyTextColumn VARCHAR(50) NULL,
MyOtherColumn BIT NULL
)
INSERT INTO #a
(
MyIntegerColumn,
MyTextColumn,
MyOtherColumn
)
SELECT
1 AS MyIntegerColumn,
''x'' AS MyTextColumn,
0 AS MyOtherColumn
SELECT MyIntegerColumn, MyTextColumn, MyOtherColumn
FROM #a')
WITH RESULT SETS
(
(
MyIntegerColumn INT NOT NULL
,MyTextColumn VARCHAR(50) NULL
,MyOtherColumn BIT NULL
)
)
Another option (kind of a hack, but it works and doesn't require you to change your use of global temp tables) is to use a SET FMTONLY ON command in front of your actual query to send a fake "First result set" to SSIS with your correct column structure. So you can do something like
SET FMTONLY ON
select 0 as a, 1 as b, 'test' as C, GETDATE() as D
SET FMTONLY OFF
select a, b, c, d from ##TempTable
When SSIS runs sp_describe_first_result_set, it will return the metadata and column names of your FMTONLY command, and won't complain about not being able to determine the metadata of your temp table because it won't even try.
If you are working on SSIS 2012, then it uses system stored procedure sp_describe_first_result_set to fetch the metadata of the tables and it does not support temporary tables. But you can go for other options like table variables and CTEs which are going to work fine. https://connect.microsoft.com/SQLServer/feedback/details/629077/denali-engine-metadata-discovery-shuns-temp-tables
I had faced a similar issue when SSSI packages were migrated from 2008 to 2016. The latest version uses sp_describe_first_result_set to fetch metadata and it does not work with temporary tables. As a workaround, I used the below query in the OLEDB source editor. I did not change the SQL stored procedure, and it still uses a temporary table. Do be sure to use the Parse Query and Preview option to ensure it works fine. See the image below.
Query:
EXEC [dbo].[spGetNames]
WITH RESULT SETS((
FirstName varchar(50),
LastName varchar(50)
));
Had the same issue as we use temp table for staging. After spending some time, found a work around.
In the OLE DB/ADO Destination of Data flow task where you specify the name of the staging table .
Change the AccessMode property to SQL command instead of OpenRowSet and specify SQL Command property to "select * from #temp".
Hurray, Its working as expected.
Catch here is when you specify Access mode other than SQL Command, SSIS expects that to be a table / view and it changed the SSIS to call sp_describe_first_result_set to get the meta data. but when you specify SQL Command, it's expecting a query or SP command etc. so luckily it still uses the old way of getting the meta data .
http://social.msdn.microsoft.com/Forums/sqlserver/en-US/cfe1c7c1-910a-4f52-9718-c3406263b177/usage-of-temp-tables-in-ssis-2012?forum=sqlintegrationservices#cfe1c7c1-910a-4f52-9718-c3406263b177
I found that the problem lied in a GUID duplicate issue, I copied elements (like the one to create temp tables) and they all received the same guid upon copying. I used a tool to reset all these guids in my package and this solved my problem.
Thanks!
I am working in SQL Server 2008 and BIDS (SSIS). I am trying to generate a "load ID" for when a package is executed and store that ID in a load history table (which then populates subsequent tables).
My basic SSIS control flow is the following:
Execute SQL Task, Data Flow Task
The load table is created via the following:
CREATE TABLE dbo.LoadHistory
(
LoadHistoryId int identity(1,1) NOT NULL PRIMARY KEY,
LoadDate datetime NOT NULL
);
The editor for the Execute SQL Task is as follows:
General:
ResultSet = None
ConnectionType = OLE DB
SQLStatement:
INSERT INTO dbo.LoadHistory (LoadDate) VALUES(#[System::StartTime]);
SELECT ? = SCOPE_IDENTITY()
Parameter Mapping:
Variable Name = User::LoadID
Direction = Output
Data Type = LONG
Parameter Name = 0
Parameter Size = -1
SSIS is throwing the following error:
[Execute SQL Task] Error: Executing the query "INSERT INTO dbo.LoadHistory
..." failed with the following error: "Multiple-step OLE DB operation generated errors. Check each OLE DB status value, if available. No work was done.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
This error message doesn't really help me find the problem. My best guess is that it's due to the parameter mapping, but I don't see my mistake. Can anybody point out my problem and provide the fix?
I figured out my problem. System::StartTime needs to have DATE as its data type, not DBTIMESTAMP.
I was passing three parameters.
In the Parameter Name property I had:
0
1
3
Corrected it to:
0
1
2
It works now, no multiple-step operation generated errors message.
I have created a table called DimInternationalFunction.
IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[DimInternationalFunction]') AND type in (N'U'))
DROP TABLE [DimInternationalFunction]
Go
Create Table DimInternationalFunction
(IntFunctionKey int NOT NULL identity primary key,
SubSubFunctionString char(10),
FunctionCode char(3),
SubFunctionCode char(6),
SubSubFunctionCode char(10),
SubSubFunctionName nvarchar(60),
SubFunctionName nvarchar(60),
FunctionName nvarchar(60))
I have initially inserted records in this table in SSMS.
After inserting the initial records manually in SSMS, now my manager wants me to insert "new records only" using SSIS.
I have tried using this in SSMS and it worked. Either it gives me 0 records inserted or sometimes it gives me 5 records inserted as a result. My manager wants me to do this in SSIS.
I tried using this script inside the OLE DB Source under Data Access Mode: SQL Command and SQL Command text:
insert into DWResourceTask.dbo.DimInternationalFunction
select f.SubSubFunctionString,
f.FunctionCode,
f.SubFunctionCode,
f.SubSubFunctionCode,
f.SubSubFunctionName,
f.SubFunctionName,
f.FunctionName
from ODS_Function F
where FunctionViewCode = 'INT'
and not exists (select * from DWResourceTask.dbo.DimInternationalFunction I
where (f.SubSubFunctionString=i.SubSubFunctionString
and f.FunctionCode=i.FunctionCode
and f.SubFunctionCode=i.SubFunctionCode
and f.SubSubFunctionCode=i.SubSubFunctionCode
and f.SubSubFunctionName=i.SubSubFunctionName
and f.SubFunctionName=i.SubFunctionName
and f.FunctionName=i.FunctionName)
)
The error message that I got after clicking preview is
The component reported the following warnings:
Error at Int Function [International Function Table [33]]: No column information was returned by the SQL command.
Choose OK if you want to continue with the operation.
Choose Cancel if you want to stop the operation.
Is there another component in SSIS that can do this? or can I just use either exec sql task component or ole db source?
I am thinking of using exec sql task connected to a data flow task, inside the data flow task I will put ole db source containing a staging table and do a delete on that or is there any other way to do it. Please help. Thanks in advance.
You could do it with an Execute SQL task.
If you want to do it "the pure SSIS way", you could use a lookup component. Set the "rows with no matching" handler to "Redirect to no match output", and configure the target table as connection. Then use the "No Match Output" only, ignoring the "Match Output". And send the records from the "No Match Output" to the target.
In spite of its name, the "Lookup" component can be used to filter data in many cases.
But I would assume the Execute SQL task would be more efficient for large data sets, keeping all data within the database engine.
I have spent a few days trying to fix this problem. I have a SSIS package with 2 execute SQL tasks within a sequence container, one is a simple delete from table and the next one an simple insert the delete precedes the insert. The delete works fine so the connection etc is ok.
The Insert is failing with the following vague and unhelpful message.
failed with the following error: "Syntax error, permission violation, or other nonspecific error". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
The insert has 1 input parameter which is a date which is bound to a datetime variable set to 01/01/2011. When I replace the ? in the sql task query with a hard coded date the task works. I have also looked at the locals at a pre-execute event break point on the insert task and the variable is ok.
I also fired up a SQL Profiler session and I see the delete query hitting the DB but nothing for the insert (when it uses the input parameter).
I am using Visual Studio 2005 Pro SP1 (Not my choice) and SQL Server 2005 SP3.
Regards
Mark
I know that you have found an answer to your question. However, I would like to clarify here that the following query you were executing using OleDb connection is valid and it does work within the Execute SQL Task in SSIS package.
INSERT INTO dbo.table1 (DateCol, OtherCol, OtherCol1)
SELECT ?, SourceCol1, SourceCol2 FROM dbo.SourceTable
Following example shows a successful implementation of the above mentioned query using SSIS 2005 (Business Intelligence Development Studio (BIDS) 2005)
Step-by-step process:
Create two tables named dbo.Source and dbo.Destination using the scripts provided under the Scripts section. Populate the table dbo.Source with data as shown in screenshot #1. Table dbo.Destination will initially be empty and will be populated with source data using an Execute SQL Task.
On the SSIS package, create an OLE DB Connection named SQLServer in the Connections Managers pointing to a SQL Server instance of your preference. Refer screenshot #2.
On the SSIS package, create a variable named RecordDate as shown in screenshot #3.
On the SSIS package, place an Execute SQL Task as shown in screenshot #4. Configure the task as shown in screenshots #5 and #6.
Screenshot #7 shows sample package execution.
Screenshot #8 shows data in the tables dbo.Source and dbo.Destination after package execution.
Hope that helps.
Scripts:
.
CREATE TABLE [dbo].[Destination](
[Id] [int] IDENTITY(1,1) NOT NULL,
[DateValue] [datetime] NOT NULL,
[ItemNumber] [varchar](50) NOT NULL,
[Qty] [int] NOT NULL,
CONSTRAINT [PK_Destination] PRIMARY KEY CLUSTERED ([Id] ASC)) ON [PRIMARY]
GO
CREATE TABLE [dbo].[Source](
[Id] [int] IDENTITY(1,1) NOT NULL,
[ItemNumber] [varchar](50) NOT NULL,
[Qty] [int] NOT NULL,
CONSTRAINT [PK_Source] PRIMARY KEY CLUSTERED ([Id] ASC)) ON [PRIMARY]
GO
Screenshot #1:
Screenshot #2:
Screenshot #3:
Screenshot #4:
Screenshot #5:
Screenshot #6:
Screenshot #7:
Screenshot #8:
You need to make sure your SQL Statement is of the correct type to be parameterized according to your connection manager.
If you're using OLE DB - your Insert statement needs to be of some kind like INSERT INTO Tbl(col) VALUES (?)
Resultset should be "None" (As there's nothing to return on your INSERT), and the Parameter Mapping tab should have a single parameter (or as many as ?'s you have, and Parameter Name should (for OLE DB) start on 0, then 1, 2 ... ,n. If you were using an ADO connection you would have to name the parameters Param1, Param2, ... ParamN.
You can see the reference for passing variables to the respective connection managers here: http://technet.microsoft.com/en-us/library/cc280502.aspx
Proper answer as per your comment:
You cannot use a parameter mapping in a query of that kind. But there is an easy way to do it.
Make a new variable sqlCommand (type string). click it in the list of variables, and press F4 to see properties. Change "EvaluateAsExpression" to true, and click the expression box to get up the expression window. In here make your query in a format like this "INSERT INTO tbl(dateCol,intCol,charCol) SELECT CONVERT(DATETIME,'" + (DT_STR,20,1252)#[User::dateVar] + "',104) AS dateCol, intCol, charCol from anotherTbl"
When you click Evaluate Expression you'll see a valid SQL statement being formed that you can use.
Now go back to the Execute SQL task, remove the parameter mapping. Close the dialog box, click the Execute SQL task, press F4, find the expressions line, click the ... and add an expression on the Property "SqlStatementSource" with expression #[User::sqlCommand] (or whatever you named your variable).
Now run the task and it should work without a problem.
You can find the expression here:
Objective:
To update datetime field (date+time) in parameter in sql task editor of SSIS package.
Description:
Step01: Created a variable having datatype as string.
e.g:
Variable name: ReconcileStartDateTime
Data Type: String
Step02: Assign value to a variable. create 'Execute SQL Task'
General:
SQL Command-> select cast(getdate() as nvarchar(100)) as StartDateTime
ResultSet-> Single Row
ByPassprepare-> True
Result set tab:
Result Name: StartDateTime
Variable Name: User::ReconcileStartDateTime
Step 03: Create Execute SQL Task and use query as below:
SqlStatement: Update OrderDetail set StartDate = cast(? as datetime) where ID= 101;
Parameter mapping: click ADD button, set variable name User::ReconcileStartDateTime; Datatype as Nvarchar; Parameter to 0
Result: When execute the SSIS package, datetime is set accordingly. SQL Profile would help to see the output.