I am trying to create an SSIS package that queries data from a table, and calls a stored procedure in another database with each row.
In my old DTS package, I was doing this:
EXEC myStoredProcedure ?, ?, ?
...and then I mapped the parameters. However, in SSIS, I can't figure out how to make this work.
I have a Data Flow task, which first runs a query for the data. It passes the data to an OLE DB Destination. I set the Data access mode to "SQL command", but when I try to put in the SQL above, I get "Invalid Parameter Count" when it parses the SQL. I can't get to the Mappings screen. Any ideas?
In the Data Flow, the OLE DB Command can be used to execute a SQL statement for each row in a dataflow - (MSDN documentation)
Alternatively, you can store the source result set in a variable of data type object and use a Foreach Loop container in the Control Flow (example here).
You will need to use an Execute SQL Task. In the SQLStatement section you can add the code to execute the stored procedure.
In order to pass in parameters, use the ? syntax and specify the parameters in the "Parameter Mapping" section.
A good example can be found here.
Related
Having seen other questions with answers that don't totally address what I am after, I am wondering how in SSIS to use an OLE DB Command transformation to do an Insert and immediately get the resulting primary key for each row inserted as a new column, all within the same Data Flow Task. That sounds like it should be a common, built-in, fairly simple thing to ask for in SSIS, right?
So the obvious first choice for me would be to use an OLE DB Command where I do a SELECT and include an OUTPUT clause in my command:
INSERT INTO dbo.MyReleaseTable(releaseDate)
OUTPUT ?=Inserted.id
VALUES (?)
Only I can't figure out how to do this in an OLE DB Command (with an output) and it not complain. I've read about using stored procedures to do this, so am I required to use a stored procedure if I want to do this?
Let's say this won't work. I could use a Script Transformation and execute direct SQL in that, right? Well if that's what I must do, then the line between using custom code and SSIS block-components gets blurred and I am tempted to throw SSIS away and just do the whole ETL in code.
Then I hear talk about using an Execute SQL task. So now I can't even do 1 data flow within 1 data flow task? Am I getting that right? I'd like to keep 1 single data flow contained within 1 data flow task and not have to break my 1 flow out between separate tasks.
If it turns out that this seemingly simple data flow objective is not built into SSIS then I will consider dumping SSIS altogether. Talend has a free ETL offering, don't they?
Well, this can be done with SSIS inside DataFlow, but with some tricks. You need to create a stored procedure with input and output parameters and reuse it in DataFlow, as described here, fetching result value.
Drawbacks of this approach:
You need to create a Stored Procedure
Each row is processed with SP, which causes implicit transactions, instead of batch processing. This can slow down your package.
Solution without performance penalty - do it in two DataFlows, first doing value insert into some temp table, and the second DF - doing SQL MERGE command at OLE DB source and handling output data as you wish. All this inside transaction, handled either by MSDTC or by your own.
I am new to SSIS and would appreciate any help on my issue below -
I have a stored procedure which does not take any input and returns a temp table. This is used for data validation and will be run every day thus, I need to create an SSIS package for the same (the requirement is such). I created an Execute SQL task where I have the result set in a variable of type Object but I want to add a condition of checking rowcount >1 and if yes then write to an excel file.
Thanks,
Hiral
Answer depends on whether you are using SSIS 2012+ or 2008.
Assuming you are with SSIS 2012+:
Approach - declare variable type int for rowcount. Process received Object variable in Foreach loop, more details and screenshots are available in this article. Inside loop - increment variable. Then you can do a conditional execution based on rowcount variable value > 0.
Alternatives - valid for SSIS 2008 as well:
Object with result set is nothing more than a .NET dataset. Create a Script task which counts rows in the first table of the dataset, and stores this in a variable. Then use this variable in package as described above.
Instead of using Execute SQL Script task for SP run, use DataFlow with OLE DB Source and specify the stored procedure as the data source. Then you can count the resulting rows, store value in a variable etc.
Change property Retainsameconnection =true for connection manager
Doing so you can access temp table created in previous step .
You could use a global temp table instead if you want the creation done in a separate step. But this comes with its own issues like being accessible to everyone/every process.
I would like to execute a stored procedure MyProc with certain input parameters for each row in the table MyTable. Column for each row will act like input values for MyProc.
How do I accomplish this in SSIS? I have been told by upper management to use SSIS and I don't have any choice in this matter.
So.. I have done the following:
Create package
Create Execute SQL Task
Created query inside ExecuteSQLTask
Create Foreach Loop Container
Create Data Flow Task inside Foreach loop container.
Inside Data Flow Task I created OLE DB Source
Connected OLEDBSource to Derived Column
Enable DataViewer in connection in step 3.3
It looks like the DataFlowTask in step 3.1 is being called for all rows in step 2.1
What I want is to invoke DataFlowTask in step 3.1 to get called for each record in the row in step 2.1 AS OPPOSED TO all the records.
Once this works, then I intend to ExecuteSQLTask inside a ForEachLoopContainer.
All you need is a Dataflow Task with an OLEDB source and an OLE DB Command transformation.
The OLEDB source SELECTs from the table that you want to perform the row-by-row stored procedure on.
It then is followed by an OLE DB Command transformation that calls the stored procedure and passes columns from the data flow to the parameters of the stored procedure.
This will result in the stored procedure being executed once for each row of your table. You don't need the initial Execute SQL, and you don't need the For-Each loop.
I have consumed simple web service for addition of two numbers and I am getting the result in xml format that I have stored in variable supposed named as my_data, now I have to insert that result in my sql table named as Data_result by writing query in oledb source editor, but unable to insert data using variable.I just want to store all the result calculated by my web service in sql table.
Since I am newbie to SSIS, any help is greatly appreciated
When I am trying to insert the data by the following query command:
"INSERT INTO Data_Result(Result) SELECT '"+#USER::my_data+"'"
It's giving error:
Error at Data Flow Task [OLE DB Source [55]]: No Column Transformation was return by SQL Command
--------Updated explanation of errors based on the solution given
Error for executing query
If I place the expression without the "" quotes then I get following error
If I place the query in double quotes then following error is shown
And if I remove User from User::Data Variable and place the query in double quotes then I get following screen
although the expression evaluates but after proceeding further on this evaluated expression when I am trying to search for the variable in expression column of Execute sql Task, then I am unable to locate the newly created variable, as shown below
------------Updated question according to other query----------
Here is the picture of my whole work flow
This is what I have did inside the for each loop container under collection tab
And this below setting I have done between Variable mapping tab,
And in below screen shot, I am using Execute SQL Task to enter my data obtained from web service task into database using an insert query, but unable to fire proper insert query,
And below is my XML file,
<?xml version="1.0" encoding="utf-16"?>
Reshma
1988-09-23T00:00:00
,
This name and birthdate I received from web service I want to insert that into database table
--------------Updated question for foreach loop container--------
Below is the foreach loop container I am using,
But still I am not getting node value into variable and also I am suppose to make use of object type variable or it can work with string type of variable.
I suggest you build this SQL statement in another variable, as explained here:
http://consultingblogs.emc.com/jamiethomson/archive/2005/12/09/2480.aspx
Then use that variable in your execute SQL statement
Normally you would try and parameterise the query but in this case (as in many others) it won't work.
Here are some more detailed instructions:
A. Generate the dynamic SQL
Create a new SSIS variable of type string
In properties, set it's 'Evaluate as Expression' to TRUE
In properties, click on 'Expression' and build an expression as follows:
"INSERT INTO Data_Result(Result) VALUES ('"+#[USER::my_data]+"')"
Press Evaluate and verify that the output is valid SQL (you may even wish to paste it into SSMS and test)
B. Get an Execute SQL task to run your SQL:
Drop a execute SQL task on your page. Set it's connection
With the execute SQL task highlighted, go to properties and click expressions
Pick the property 'SQLStatementSource'. In the expression pick your variable that you built in step A and press evaluate.
Press OK on everything
Now whatever change occurs in my_data will automatically be applied to your execute SQL task
Try that for starters and get back to me.
Here i am new in Developing the SSIS package
I need your support to come up with the solution.
I have 10 different set of stored procedures which I have to export into text file, all 10 procedures will return the same set of columns (only calling parameters are different).
I am not getting the solution how to do ?
Could you please help me to understand how to export the data from a stored procedure output to tab delimited text file?
Please let me know how to build the ssis package ?
Thanks
This is very hard to do without putting pictures in on each step. I do not seem to be able to put pictures in so I will try to describe in is much detail as I can.
You have to first set up a connection to the database where you are going to run the stored procedures from. This means creating a connection manager for "New OLE DB Connection". You will need vaild login to the database information to create this connection.
In the control area I would set up a "Execute SQL Task". I would set the result set to full result set and set the connection to the one you named in the prior step. To call a stored procedure from a SQL task use something like "exec ? = dbo.usp_check_load_table_all #JobCode = ?, #TransId = ? , #Status = ?, #TurnStatusOff = ?" The first ? is the return code from the stored procedure. The others are the parameters to run the stored procedure. Now you are running 10 different stored procedures and I only know how to run one but you could create ten packages, one to run each and concatenate the files when it is done. In the parameter mapping you set the values for the variables to run with. Make sure to create a USER::ReturnValue type long for the return code. The results set needs one entry a USER::Results of type object.
You now put in a foreach loop for a ADO enumerator putting in the USER::Results in as the variable. This allows you to read in each row one at a time. You must create user variables for the variable mapping to go into.
I would then do a data flow task and put a derived column task and set up each of the fields you want to write to the file from the USER:: fieids you created for the foreach loop.
I would create a flat file connection in the connection manager as a delimited file, tab delimited. You will need a file that looks like the output you desire as you will need to map each field in the file.
Add a flat file destination to under the deriried column task and map it to the flat file you just created. Now map each field to the output.
I hope this is helpful as I was once new SSIS myself.