I have consumed simple web service for addition of two numbers and I am getting the result in xml format that I have stored in variable supposed named as my_data, now I have to insert that result in my sql table named as Data_result by writing query in oledb source editor, but unable to insert data using variable.I just want to store all the result calculated by my web service in sql table.
Since I am newbie to SSIS, any help is greatly appreciated
When I am trying to insert the data by the following query command:
"INSERT INTO Data_Result(Result) SELECT '"+#USER::my_data+"'"
It's giving error:
Error at Data Flow Task [OLE DB Source [55]]: No Column Transformation was return by SQL Command
--------Updated explanation of errors based on the solution given
Error for executing query
If I place the expression without the "" quotes then I get following error
If I place the query in double quotes then following error is shown
And if I remove User from User::Data Variable and place the query in double quotes then I get following screen
although the expression evaluates but after proceeding further on this evaluated expression when I am trying to search for the variable in expression column of Execute sql Task, then I am unable to locate the newly created variable, as shown below
------------Updated question according to other query----------
Here is the picture of my whole work flow
This is what I have did inside the for each loop container under collection tab
And this below setting I have done between Variable mapping tab,
And in below screen shot, I am using Execute SQL Task to enter my data obtained from web service task into database using an insert query, but unable to fire proper insert query,
And below is my XML file,
<?xml version="1.0" encoding="utf-16"?>
Reshma
1988-09-23T00:00:00
,
This name and birthdate I received from web service I want to insert that into database table
--------------Updated question for foreach loop container--------
Below is the foreach loop container I am using,
But still I am not getting node value into variable and also I am suppose to make use of object type variable or it can work with string type of variable.
I suggest you build this SQL statement in another variable, as explained here:
http://consultingblogs.emc.com/jamiethomson/archive/2005/12/09/2480.aspx
Then use that variable in your execute SQL statement
Normally you would try and parameterise the query but in this case (as in many others) it won't work.
Here are some more detailed instructions:
A. Generate the dynamic SQL
Create a new SSIS variable of type string
In properties, set it's 'Evaluate as Expression' to TRUE
In properties, click on 'Expression' and build an expression as follows:
"INSERT INTO Data_Result(Result) VALUES ('"+#[USER::my_data]+"')"
Press Evaluate and verify that the output is valid SQL (you may even wish to paste it into SSMS and test)
B. Get an Execute SQL task to run your SQL:
Drop a execute SQL task on your page. Set it's connection
With the execute SQL task highlighted, go to properties and click expressions
Pick the property 'SQLStatementSource'. In the expression pick your variable that you built in step A and press evaluate.
Press OK on everything
Now whatever change occurs in my_data will automatically be applied to your execute SQL task
Try that for starters and get back to me.
Related
My Execute SQL Task uses a simple query that should be returning a single value. I want to write that value to a variable so that I can use it in a data flow task later but the result never seems to get written to my variable.
The SQL statement is simple:
select max (rec_id) from [dbo].[RX_BILLING_TEST]
and I believe that I've set up the task correctly:
When I execute the task, it completes successfully but the variables window shows that the value of my variable didn't update. I set breakpoints on variable value changed and on post execute but that didn't help.
As much as I hate errors, this is a case where I would appreciate getting one to point me in some direction. Any ideas what might be wrong?
In a similar case there are many things you have to check:
Try adding an alias to the aggregate function
select max (rec_id) AS Max_Rec_Id from [dbo].[RX_BILLING_TEST]
Check that your package doesn't have more than one variable User::v_Recid with different scopes. or that User::v_Recid scope's is Execute SQL Task. (it must be the Package)
Test your SQL Command using Sql Server Management Studio
Try adding the database name to your Command
select max (rec_id) AS Max_Rec_Id from [MyDB].[dbo].[RX_BILLING_TEST]
I am new to SSIS and would appreciate any help on my issue below -
I have a stored procedure which does not take any input and returns a temp table. This is used for data validation and will be run every day thus, I need to create an SSIS package for the same (the requirement is such). I created an Execute SQL task where I have the result set in a variable of type Object but I want to add a condition of checking rowcount >1 and if yes then write to an excel file.
Thanks,
Hiral
Answer depends on whether you are using SSIS 2012+ or 2008.
Assuming you are with SSIS 2012+:
Approach - declare variable type int for rowcount. Process received Object variable in Foreach loop, more details and screenshots are available in this article. Inside loop - increment variable. Then you can do a conditional execution based on rowcount variable value > 0.
Alternatives - valid for SSIS 2008 as well:
Object with result set is nothing more than a .NET dataset. Create a Script task which counts rows in the first table of the dataset, and stores this in a variable. Then use this variable in package as described above.
Instead of using Execute SQL Script task for SP run, use DataFlow with OLE DB Source and specify the stored procedure as the data source. Then you can count the resulting rows, store value in a variable etc.
Change property Retainsameconnection =true for connection manager
Doing so you can access temp table created in previous step .
You could use a global temp table instead if you want the creation done in a separate step. But this comes with its own issues like being accessible to everyone/every process.
I have some variables (at package level, at dataflow level, doesn't matter) and I need to load them by running an Execute Sql Task.
I added an Execute Sql Task, I have wrote my sql statement in filed SqlStatement, I have changed the ForcedExecutionValueType to string.
Now I want that when the Execute Sql Task is executed the return value that that select returns, to be the exact value for my variable.
example:
Select employer_name from employers where id=1
Returs="John Doe"
value for #employer_name must be "John Doe"
Any hints?
Your more detailed discussion of the issue is a little difficult to follow, so I aplogize if my following steps cover items with you are already familiar.
Steps:
Under the Control Flow tab, drag over an Execute SQL Task and open it.
Select General on the left.
Define the Connection (based on a database source created in Connection Managers below).
Define the SQL Statement.
For Result Set, select Single Row.
Select Result Set on the left.
Select the Add button and notice that a new record was added above.
Under the Result name column header for the new record, change "NewResultname" to "0".
Under Variable Name column header for the new record, select either an existing User variable you've already created or scroll to the top and create a new one.
Select OK and you're done.
Running the Execute SQL Task by itself will now populate the variable. Make certain you've verified that the SQL will return only one value and no more. Otherwise, you will need to modify your SQL with "TOP 1" in it to be on the safe side. When multiple values are expected, then you to apply a variable defined with a Data Type of "Object" and use "Full Result set" instead of "Single Row" in the Execute SQL Task.
Hope this helps.
I am working with SSIS 2008. I have a select query name sqlquery1 that returns some rows:
aq
dr
tb
This query is not implemented on the SSIS at the moment.
I am calling a stored procedure from an OLE DB Source within a Data Flow Task. I would like to pass the data obtained from the query to the stored procedure parameter.
Example:
I would like to call the stored procedure by passing the first value aq
storedProdecure1 'aq'
then pass the second value dr
storedProdecure1 'dr'
I guess it would be something like a cycle. I need this because the data generated by the OLE DB Source through the stored procedure needs to be sent to another destination and this must be done for each record of the sqlquery1.
I would like to know how to call the query sqlquery1 and pass its output to call another stored procedure.
How do I need to do this in SSIS?
Conceptually, what your solution will look like is an execute your source query to generate your result set. Store that into a variable and then you'll need to do iterate through those results and for each row, you'll want to call your stored procedure with that row's value and send the results into a new Excel file.
I'd envision your package looking something like this
An Execute SQL Task, named "SQL Load Recordset", attached to a Foreach Loop Container, named "FELC Shred Recordset". Nested inside there I have a File System Task, named "FST Copy Template" which is a precedence for a Data Flow Task, named "DFT Generate Output".
Set up
As you're a beginner, I'm going to try and explain in detail. To save yourself some hassle, grab a copy of BIDSHelper. It's a free, open source tool that improves the design experience in BIDS/SSDT.
Variables
Click on the background of your Control Flow. With nothing selected, right-click and select Variables. In the new window that pops up, click the button that creates a New Variable 4 times. The reason for clicking on nothing is that until SQL Server 2012, the default behaviour of variable creation is to create them at the scope of the current object. This has resulted in many lost hairs for new and experienced developers alike. Variable names are case sensitive so be aware of that as well.
Rename Variable to RecordSet. Change the Data type from Int32 to Object
Rename Variable1 to ParameterValue. Change the data type from Int32 to String
Rename Variable2 to TemplateFile. Change the data type from Int32 to String. Set the value to the path of your output Excel File. I used C:\ssisdata\ShredRecordset.xlsx
Rename Variable 4 to OutputFileName. Change the data type from Int32 to String. Here we're going to do something slightly advanced. Click on the variable and hit F4 to bring up the Properties window. Change the value of EvaluateAsExpression to True. In Expression, set it to "C:\\ssisdata\\ShredRecordset." + #[User::ParameterValue] + ".xlsx" (or whatever your file and path are). What this does, is configures a variable to change as the value of ParameterValue changes. This helps ensure we get a unique file name. You're welcome to change naming convention as needed. Note that you need to escape the \ any time you are in an expression.
Connection Managers
I have made the assumption you are using an OLE DB connection manager. Mine is named FOO. If you are using ADO.NET the concepts will be similar but there will be nuances pertaining to parameters and such.
You will also need a second Connection Manager to handle Excel. If SSIS is temperamental about data types, Excel is flat out psychotic-stab-you-in-the-back-with-a-fork-while-you're-sleeping about data types. We're going to wait and let the data flow actually create this Connection Manager to ensure our types are good.
Source Query to Result Set
The SQL Load Recordset is an instance of the Execute SQL Task. Here I have a simple query to mimic your source.
SELECT 'aq' AS parameterValue
UNION ALL SELECT 'dr'
UNION ALL SELECT 'tb'
What's important to note on the General tab is that I have switched my ResultSet from None to Full result set. Doing this makes the Result Set tab go from being greyed out to usable.
You can observe that I have assigned the Variable Name to the variable we created above (User::RecordSet) and I the Result Name is 0. That is important as the default value, NewResultName doesn't work.
FELC Shred Recordset
Grab a Foreach Loop Container and we will use that to "shred" the results that were generated in the preceding step.
Configure the enumerator as a Foreach ADO Enumerator Use User::RecordSet as your ADO object source variable. Select rows in the first table as your Enumeration mode
On the Variable Mappings tab, you will need to select your variable User::ParameterValue and assign it the Index of 0. This will result in the zerotth element in your recordset object being assigned to the variable ParameterValue. It is important that you have data type agreement as SSIS won't do implicit conversions here.
FST Copy Template
This a File System Task. We are going to copy our template Excel File so that we have a well named output file (has the parameter name in it). Configure it as
IsDestinationPathVariable: True
DestinationVarible: User::OutputFileName
OverwriteDestination: True
Operation: Copy File
IsSourcePathVariable: True
SourceVariable: User::TemplateFile
DFT Generate Output
This is a Data Flow Task. I'm assuming you're just dumping results straight to a file so we'll just need an OLE DB Source and an Excel Destination
OLEDB dbo_storedProcedure1
This is where your data is pulled from your source system with the parameter we shredded in the Control Flow. I am going to write my query in here and use the ? to indicate it has a parameter.
Change your Data access mode to "SQL Command" and in the SQL command text that is available, put your query
EXECUTE dbo.storedProcedure1 ?
I click the Parameters... button and fill it out as shown
Parameters: #parameterValue
Variables: User::ParameterValue
Param direction: Input
Connect an Excel Destination to the OLE DB Source. Double click and in the Excel Connection Manager section, click New... Determine if you're needing 2003 or 2007 format (.xls vs .xlsx) and whether you want your file to have header rows. For you File Path, put in the same value you used for your #User::TemplatePath variable and click OK.
We now need to populate the name of the Excel Sheet. Click that New... button and it may bark that there is not sufficient information about mapping data types. Don't worry, that's semi-standard. It will then pop up a table definition something like
CREATE TABLE `Excel Destination` (
`name` NVARCHAR(35),
`number` INT,
`type` NVARCHAR(3),
`low` INT,
`high` INT,
`status` INT
)
The "table" name is going to be the worksheet name, or precisely, the named data set in the worksheet. I made mine Sheet1 and clicked OK. Now that the sheet exists, select it in the drop down. I went with the Sheet1$ as the target sheet name. Not sure if it makes a difference.
Click the Mappings tab and things should auto-map just fine so click OK.
Finally
At this point, if we ran the package it would overwrite the template file every time. The secret is we need to tell that Excel Connection Manager we just made that it needs to not have a hard coded name.
Click once on the Excel Connection Manager in the Connection Managers tab. In the Properties window, find the Expressions section and click the ellipses ... Here we will configure the Property ExcelFilePath and the Expression we will use is
#[User::OutputFileName]
If your icons and such look different, that's to be expected. This was documented using SSIS 2012. Your work flow will be the same in 2005 and 2008/2008R2 just the skin is different.
If you run this package and it doesn't even start and there is an error about the ACE 12 or Jet 4.0 something not available, then you are on a 64bit machine and need to tell BIDS/SSDT that you want to run in 32 bit mode.
Ensure the Run64BitRuntime value is False. This project setting can be found by right clicking on the project, expand the Configuration Properties and it will be an option under Debugging.
Further reading
A different example of shredding a recordset object can be found on How to automate the execution of a stored procedure with an SSIS package?
I am trying to merge records from an Oracle database table to my local SQL table.
I have a variable for the package that is an Object, called OWell.
I have a data flow task that gets the Oracle data as a SQL statment (select well_id, well_name from OWell order by Well_ID), and then a conversion task to convert well_id from a DT_STR of length 15 to a DT_WSTR; and convert well_name from a DT_STR of length 15 to DT_WSTR of length 50. That is then stored in the recordset OWell.
The reason for the conversions is the table that I want to add records to has an identity field: SSIS shows well_id as a DT_WSTR of length 15, well_name a DT_WSTR of length 50.
I then have a SQL task that connects to the local database and attempts to add records that are not there yet. I've tried various things: using the OWell as a result set and referring to it in my SQL statement. Currently, I have the ResultSet set to None, and the following SQL statment:
Insert into WELL (WELL_ID, WELL_NAME)
Select OWELL_ID, OWELL_NAME
from OWell
where OWELL_ID not in
(select WELL.WELL_ID from WELL)
For Parameter Mapping, I have Paramater 0, called OWell_ID, from my variable User::OWell. Parameter 1, called OWell_Name is from the same variable. Both are set to VARCHAR, although I've also tried NVARCHAR. I do not have a Result set.
I am getting the following error:
Error: 0xC002F210 at Insert records to FLEDG, Execute SQL Task: Executing the query "Insert into WELL (WELL_ID, WELL_NAME)
Select OWELL..." failed with the following error: "An error occurred while extracting the result into a variable of type (DBTYPE_STR)". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
I don't think it's a data type issue, but rather that I somehow am not using the resultset properly. How, exactly, am I supposed to refer to that recordset in my SQL task, so that I can use the two recordset fields and add records that are missing?
Your problem is that you are trying to read an object variable into a sql task, and refer to that variable in the sql task.
To do what you are trying to do, you can use a foreach loop task. You can set the enumerator of a for each to an object (recordset) variable and map its columns to variables that you can then pass as parameters into your sql task. Your sql code in the example above has another flaw in that you are trying to reference a variable in your package as if it were a table in your database. You need to change your sql to be something like Insert into well(?,?)
This approach however leaves out the step where you can check to see if the records exists before you insert it. A better overall approach would be to do this all in a dataflow.
Do everything you are doing in your select from Oracle dataflow. At the last step, instead of using a recordset destination pointing to variable USER::OWell, add a lookup from the local sql table. Set your sql statement there to be select WELL.WELL_ID from WELL. On the columns tab in your lookup match Well_ID from your dataflow (fields on the left) to Well_ID from your lookup (fields on the right) by dragging the well_id field from the left to the right to form a connector between the boxes. At the bottom of the dialog box, click on Configure Error Output and set the error column value for the lookup output row to be Redirect Row. Choose OK to save and close this lookup. Next, add a oledb destination to the data flow and connect it to the error output of the lookup (the red arrow). Point the destination to the sql table and map the columns from the dataflow to the appropriate columns in the output table. This will pass the rows from the oracle dataflow that do not exist in the sql table into the bulk insert of the sql table.
To infer missing rows we either used a lookup task and then directed the unfound rows to an ordinary OLEDB destination (you just don't supply the identity column, obviously) or (where we were comparing a whole table) the SQLBI.com TableDifference component and routed the new rows to a similar OLEDB destination.
Individual INSERTs in SQL Command task aren't terribly quick.