Getting error when trying to run Sql Command in SSIS package.
Task: DataFlow Task
Connection: ADO.NET
Data Access mode: Sql Command
Sql text:
select from table where field1 = ? and field2 = ?
Error:" No value given for one or more required parameters"
More Information:
Execute Sql task in package:
(General tab)
- Connection: ADO.NET
- SQL Statement: exec storedprocedureX ?,?
(Parameter Mapping tab)
User::field1 , Input , String , 0 , -1
User::field2, Input, String, 1, -1
Variables set in package
field1 value 12C
field2 value 15A
What am I missing that is causing the variable values to not be read at Data flow level? I have no problem at the Execute SQL task level.
An OLE DB Command in the data flow is different compared to the Execute SQL Task in the control flow. You seem to be describing the Execute SQL Task correctly.
To use a variable in the data flow, you need to add it to the data flow -- the easiest way is to use a Derived Column with an expression. Add a Derived Column to your Data Flow before the OLE DB Command and configure it as follows: Derived Column Name: field1; Derived Column: add as new column; Expression: #[User::field1]. Then in the OLE DB Command, under Column Mappings map the columns as Input Column: field1; Destination Column: Param_0, etc.
Related
The DFT source someday returns no result.So have to insert/update the table based on some default value which is defined in the derived column. As you can see in the attached figure i am using conditional split and checking the Rowcount == 0 if this condition satisfy this will process the OLEDB command where i am using a simple Insert statement
INSERT INTO [dbo].[Invalid]
(Code]
,[Date]
,[CreatedDate]
,[UpdatedDate])
VALUES
(?
,?
,GETDATE()
,GETDATE())
The code and the date value is defined in the derived column and mapped in the OLEDB command column mappings.
the data flow task looks like this
When the Oledb source rowcount > 0 then the value is inserting in the OLEDB destination.
but when rowcount = 0 the value is not inserting through OLEDB command. The dataflow task is not returning any error. When i debug the rowcount variable it showing its value as 0 but still it is not doing the insert.
Please let me know how to solve this.
I don't want to use MERGE JOIN/Lookup for this as it is always insert/update single row.
For sure, if row count = 0 then it will not process any command since no rows are found in the pipeline, which will not execute the OLE DB Command. I think you are looking for an Execute SQL Task.
You should remove the conditional split and the OLE DB Command. And outside the data flow task, add an Execute SQL Task after the data flow task and set an expression in the precedence constraint (if Row count = 0) that links the Data flow task and the Execute SQL task.
For additional information about precedence constraint you can refer to the following article:
Working with Precedence Constraints in SQL Server Integration Services
I have a huge (26GB) sqlite database that I want to import to SQL Server with SSIS.
I have everything setup correctly. Some of the data flows are working correctly and importing the data.
Data flows are simple. They just consist of source and destination.
But when it comes to a table that has 80 million rows, data flow fails with this unhelpful message:
Code: 0xC0047062
Source: Data Flow Task Source 9 - nibrs_bias_motivation [55]
Description: System.Data.Odbc.OdbcException (0x80131937): ERROR [HY000] unknown error (7)
at System.Data.Odbc.OdbcConnection.HandleError(OdbcHandle hrHandle, RetCode retcode)
at System.Data.Odbc.OdbcCommand.ExecuteReaderObject(CommandBehavior behavior, String method, Boolean needReader, Object[] methodArguments, SQL_API odbcApiMethod)
at System.Data.Odbc.OdbcCommand.ExecuteReaderObject(CommandBehavior behavior, String method, Boolean needReader)
at System.Data.Odbc.OdbcCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.Odbc.OdbcCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior)
at Microsoft.SqlServer.Dts.Pipeline.DataReaderSourceAdapter.PreExecute()
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPreExecute(IDTSManagedComponentWrapper100 wrapper)
And before this task fails, memory usage goes up to 99%, then the task fails. This made me think its a memory issue. But I don't know how can I solve this.
I tried setting DelayValidation to true on all data flow tasks. Nothing changed.
I played with the buffer sizes. Nothing.
What can I do?
Step by Step guide
Since the error is thrown when reading from a large dataset, try reading data by chunks, to achieve that you can follow these steps:
Declare 2 Variables of type Int32 (#[User::RowCount] and #[User::IncrementValue])
Add an Execute SQL Task that execute a select Count(*) command and store the Result Set into the variable #[User::RowCount]
Add a For Loop with the following preferences:
Inside the for loop container add a Data flow task
Inside the dataflow task add an ODBC Source and OLEDB Destination
In the ODBC Source select SQL Command option and write a SELECT * FROM TABLE query *(to retrieve metadata only`
Map the columns between source and destination
Go back to the Control flow and click on the Data flow task and hit F4 to view the properties window
In the properties window go to expression and Assign the following expression to [ODBC Source].[SQLCommand] property: (for more info refer to How to pass SSIS variables in ODBC SQLCommand expression?)
"SELECT * FROM MYTABLE ORDER BY ID_COLUMN
LIMIT 500000
OFFSET " + (DT_WSTR,50)#[User::IncrementValue]"
Where MYTABLE is the source table name, and IDCOLUMN is your primary key or identity column.
Control Flow Screenshot
References
ODBC Source - SQL Server
How to pass SSIS variables in ODBC SQLCommand expression?
HOW TO USE SSIS ODBC SOURCE AND DIFFERENCE BETWEEN OLE DB AND ODBC?
SQLite Limit
I have to consolidate data from from 1000+ databases having the same structure/tables in one unique DB.
DBs may be added and removed potentially on a daily basis so I need to retrieve the list of DBs dynamically and run the dynamically generated SQL query to extract data on each of them.
I designed the Data Flow with a query from a variable that is working fine if executed with a static value:
With a SQL task I get the list of instances, I loop over the them and with a nested Foreach Loop/SQL task I retrieve the database names and create the dynamic SQL with the following statement (DB name is anonymized):
SELECT 'select ''' + name + ''' as DatabaseName, ID from ' + name + '.[dbo].[Orders] as querytext FROM sys.databases WHERE name LIKE ( 'XXX%_%' );
This part is also working fine:
How can I use the result of the SQL task "Execute SQL Task - Get query text" as query to be executed in the Source "OLE DB Source 1" (part of "Data Flow Task 3")?
I tried mapping an Object variable "User::SqlCommandFromSQLTask" in the result set of the SQL task, then set it up as ADO object source variable and with a Script task convert it to string and pass the value to the variable SqlStringFromSQLTask3 (used as source in "OLE DB Source 1") but I get the error Violation of PRIMARY KEY constraint, like if the data flow is always running with a static value I set up as default:
While, if I remove the value from the variable panel, I get the error "Command text was not set for the command object.", even changing the property DelayValidation of the Data Flow to false.
Any help is much appreciated.
When I have used SSIS to connect to multiple SQL Server boxes, I have stored those SQL Server connection strings in a table in a central database. Then I use a query of that table as the input to the foreach loop data flow task. If we ever have to change a sql server connection string, which does happen, we just update that table with the newest value.
My best attempt at visualizing this:
ForEach row in dbo.runThese
**** Start Loop
(grab select statements from sql table)
dbo.runThese
Output:
ID db2_script sql_script
---------------------------------------------------------------------------
1 'select count(\*) from db2_cstmr' 'select count(*) from sql_cstmr'
(Run each script on an individual connection to the DB2 and SQL Server database)
(create a combined string with each result)
149, 149
(Insert the combined results into a SQL Server table)
INSERT INTO dbo.storeResults
VALUES (149,149)
**** End Loop
I see three different ways to do this, but I'll provide the one I see is the most elegant. I will split the tasks based on location within the package:
1. Variables
New variable "Statements" of Object data type, which will hold a list of db2 and sql server statements
db2_script: String
sql_script: String
id: int32
2. Control Flow
Execute SQL Task: Get all the records (sql statements) into the object variable using something like: SELECT id, db2_script, sql_script FROM dbo.StatementsToExecute. You need to set ResultSet property of the component to "Full result set" and configure the object variable in the Result Set pane
For Each Loop: using the object variable as enumerator (Foreach From Variable Enumerator) in the Collection pane, and assigning into db2_script, sql_script and id variables in the Variable Mapping pane
Data Flow component (see next)
3. Data Flow
OLEDB Source for DB2 database: specify variable db2_script for source statement (Data access mode: SQL command from variable)
OLEDB Source for SQL Server database: specify variable sql_script for source statement (Data access mode: SQL command from variable)
Edit both sources with Advanced Editor, got to "Input and Output Properties" tab, click on "OLE DB Source Output", set IsSorted=True, click on "OLE DB Source Output"->"Output Columns"->db2_count/sql_count
MERGE: merge both sources into one single pipeline and two different output columns
OLEDB Target: map db2_count and sql_count to the target columns
Note: you would need to provide aliases for the Counts in each Select statement (e.g. SELECT COUNT(*) AS db2_count FROM ...) because they will give names to the columns in the Data Flow's pipeline. Another way is to edit in advanced mode both sources and give ad-hoc names
I am working in SQL Server 2008 and BIDS (SSIS). I am trying to generate a "load ID" for when a package is executed and store that ID in a load history table (which then populates subsequent tables).
My basic SSIS control flow is the following:
Execute SQL Task, Data Flow Task
The load table is created via the following:
CREATE TABLE dbo.LoadHistory
(
LoadHistoryId int identity(1,1) NOT NULL PRIMARY KEY,
LoadDate datetime NOT NULL
);
The editor for the Execute SQL Task is as follows:
General:
ResultSet = None
ConnectionType = OLE DB
SQLStatement:
INSERT INTO dbo.LoadHistory (LoadDate) VALUES(#[System::StartTime]);
SELECT ? = SCOPE_IDENTITY()
Parameter Mapping:
Variable Name = User::LoadID
Direction = Output
Data Type = LONG
Parameter Name = 0
Parameter Size = -1
SSIS is throwing the following error:
[Execute SQL Task] Error: Executing the query "INSERT INTO dbo.LoadHistory
..." failed with the following error: "Multiple-step OLE DB operation generated errors. Check each OLE DB status value, if available. No work was done.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
This error message doesn't really help me find the problem. My best guess is that it's due to the parameter mapping, but I don't see my mistake. Can anybody point out my problem and provide the fix?
I figured out my problem. System::StartTime needs to have DATE as its data type, not DBTIMESTAMP.
I was passing three parameters.
In the Parameter Name property I had:
0
1
3
Corrected it to:
0
1
2
It works now, no multiple-step operation generated errors message.