I am new to SSIS and would appreciate any help on my issue below -
I have a stored procedure which does not take any input and returns a temp table. This is used for data validation and will be run every day thus, I need to create an SSIS package for the same (the requirement is such). I created an Execute SQL task where I have the result set in a variable of type Object but I want to add a condition of checking rowcount >1 and if yes then write to an excel file.
Thanks,
Hiral
Answer depends on whether you are using SSIS 2012+ or 2008.
Assuming you are with SSIS 2012+:
Approach - declare variable type int for rowcount. Process received Object variable in Foreach loop, more details and screenshots are available in this article. Inside loop - increment variable. Then you can do a conditional execution based on rowcount variable value > 0.
Alternatives - valid for SSIS 2008 as well:
Object with result set is nothing more than a .NET dataset. Create a Script task which counts rows in the first table of the dataset, and stores this in a variable. Then use this variable in package as described above.
Instead of using Execute SQL Script task for SP run, use DataFlow with OLE DB Source and specify the stored procedure as the data source. Then you can count the resulting rows, store value in a variable etc.
Change property Retainsameconnection =true for connection manager
Doing so you can access temp table created in previous step .
You could use a global temp table instead if you want the creation done in a separate step. But this comes with its own issues like being accessible to everyone/every process.
Related
This seems ridiculously easy, but I can't find it anywhere...
I have a VERY simple sequence container with two tasks: Truncate a SQL table, and repopulate it from production. But this container will be repeated for about 50 tables. The container's name (entered manually) = the name of both the source and destination tables.
I have two variables:
"TableName" is entered manually.
"DelTable" is an expression that uses #[User::TableName] to generate a simple SQL statement.
I'm super-lazy and would like to use an expression to set "TableName" = the name of the current scope so I only have to enter it once.
Ideas???
THANK YOU!
if you are truncating all tables in a DB and replacing with exactly the same structure, how about this approach:
Execute SQL:
select table_name
from INFORMATION_SCHEMA.TABLES --Add a where to limit the tables to the ones you want
Save results to an object variable called TABLES
Add a for each loop:
Loop through ADO Object setting value to a string variable called table
Add Execute SQL to FE LOOP: truncate table ? and map parameter.
Add a 2nd Execute SQL statement:
INSERT INTO SERVER.DB.SCHEMA.?
select * from ?
Again map the parameters.
If you are having trouble mapping parameters set up variables and use them to create the SQL statements to run.
#TomPhillips is correct, I cannot unfortunately comment or make that answer useful. hence commenting here.
There's no easy quick fix to use a loop/automate unless all the 50 tables are same structure which is rare by any stretch of imagination.
BIML is the way to go if you are lazy :)
SSIS is not dynamic. Data Flows require fixed input and output at compile time, not runtime. You cannot simply change the table name and have it work.
If you have a list of 50 tables to do the same function on, you can use BIML to dynamically generate the SSIS package(s). But the DF itself cannot be dynamic.
I want to copy and merge data from tables with identical structure (in a number of different source databases) to a single table of similar structure in a destination database. From time to time I need to add or remove a source database.
This is currently achieved using a Data Flow Task containing an OLEDB source with a SQL query within which there is a UNION for each of the databases I am extracting from. There is quite a lot of SQL within each UNION so, if I need to add fields, I need to add the same additional SQL to each UNION. Similarly, when I add or remove a source database I need to add or remove a UNION.
I was hoping that, rather than use such a UNION with a lot of duplicated code, I could, instead, use a Foreach Loop Container that executes SQL contained in a variable using parameters to substitute the name of the database and other database dependent items within the SQL on each iteration but I hit problems with that as I assume the Data Flow Task within the loop could not interpret the incoming fields because of the use what is effectively dynamic SQL.
Any suggestions as to how I might best achieve this without duplicating a lot of SQL?
It sounds like you have your loop figured out for moving from database to database. As long as the table schemas are identical (other than names as noted) from database to database, this should work for you.
Inside the For Each Loop container, create either a Script Task or an Execute SQL Task, whichever you're more comfortable working with.
Use that task to dynamically generate the SQL of your OLE DB Source query, changing the Customer Code prefix for each iteration. Assign the SQL text to a variable, either directly in a Script Task, or by assigning the Result Set of your Execute SQL Task (the result set being the query text) to a variable.
Inside your Data Flow Task, in the OLE DB Source, under Data Access Mode select "SQL Command from variable". Select the variable that you populated with your query in the last task.
You'll also need to handle changing the connection string between iterations, but, again, it sounds like you have a handle on that part already.
My Execute SQL Task uses a simple query that should be returning a single value. I want to write that value to a variable so that I can use it in a data flow task later but the result never seems to get written to my variable.
The SQL statement is simple:
select max (rec_id) from [dbo].[RX_BILLING_TEST]
and I believe that I've set up the task correctly:
When I execute the task, it completes successfully but the variables window shows that the value of my variable didn't update. I set breakpoints on variable value changed and on post execute but that didn't help.
As much as I hate errors, this is a case where I would appreciate getting one to point me in some direction. Any ideas what might be wrong?
In a similar case there are many things you have to check:
Try adding an alias to the aggregate function
select max (rec_id) AS Max_Rec_Id from [dbo].[RX_BILLING_TEST]
Check that your package doesn't have more than one variable User::v_Recid with different scopes. or that User::v_Recid scope's is Execute SQL Task. (it must be the Package)
Test your SQL Command using Sql Server Management Studio
Try adding the database name to your Command
select max (rec_id) AS Max_Rec_Id from [MyDB].[dbo].[RX_BILLING_TEST]
My SSIS package has an execute SQL task which has a query that needs a datetime filter at runtime.
The value of this filter is supposed to be the last datetime in which the package ran successfully.
What is the standard/optimal methodology to retrieve, persist and use this lastrun datetime?
For that kind of thing, I have a "config" table in the database to store the value. Then this can be read and updated each time the package runs. You could also use a text file, but that is not as secure.
Edit:
I achieve this by invoking a SQL Task at the end of the Package that calls a stored procedure. This SP accepts a bit parameter indicating success (1) or failure (0). The SP uses GetDate() to record the time that the Proc ran (which is when the Package finishes).
As DeanOC posted, I always have a step in my package that stores this kind of stuff. It can be as simple as a insert select current timestamp... kind of thing. or it may be the max of a timestamp column in the data I'm processing, so that next run I can filter by ...> StoredMaxTimestamp.
I have consumed simple web service for addition of two numbers and I am getting the result in xml format that I have stored in variable supposed named as my_data, now I have to insert that result in my sql table named as Data_result by writing query in oledb source editor, but unable to insert data using variable.I just want to store all the result calculated by my web service in sql table.
Since I am newbie to SSIS, any help is greatly appreciated
When I am trying to insert the data by the following query command:
"INSERT INTO Data_Result(Result) SELECT '"+#USER::my_data+"'"
It's giving error:
Error at Data Flow Task [OLE DB Source [55]]: No Column Transformation was return by SQL Command
--------Updated explanation of errors based on the solution given
Error for executing query
If I place the expression without the "" quotes then I get following error
If I place the query in double quotes then following error is shown
And if I remove User from User::Data Variable and place the query in double quotes then I get following screen
although the expression evaluates but after proceeding further on this evaluated expression when I am trying to search for the variable in expression column of Execute sql Task, then I am unable to locate the newly created variable, as shown below
------------Updated question according to other query----------
Here is the picture of my whole work flow
This is what I have did inside the for each loop container under collection tab
And this below setting I have done between Variable mapping tab,
And in below screen shot, I am using Execute SQL Task to enter my data obtained from web service task into database using an insert query, but unable to fire proper insert query,
And below is my XML file,
<?xml version="1.0" encoding="utf-16"?>
Reshma
1988-09-23T00:00:00
,
This name and birthdate I received from web service I want to insert that into database table
--------------Updated question for foreach loop container--------
Below is the foreach loop container I am using,
But still I am not getting node value into variable and also I am suppose to make use of object type variable or it can work with string type of variable.
I suggest you build this SQL statement in another variable, as explained here:
http://consultingblogs.emc.com/jamiethomson/archive/2005/12/09/2480.aspx
Then use that variable in your execute SQL statement
Normally you would try and parameterise the query but in this case (as in many others) it won't work.
Here are some more detailed instructions:
A. Generate the dynamic SQL
Create a new SSIS variable of type string
In properties, set it's 'Evaluate as Expression' to TRUE
In properties, click on 'Expression' and build an expression as follows:
"INSERT INTO Data_Result(Result) VALUES ('"+#[USER::my_data]+"')"
Press Evaluate and verify that the output is valid SQL (you may even wish to paste it into SSMS and test)
B. Get an Execute SQL task to run your SQL:
Drop a execute SQL task on your page. Set it's connection
With the execute SQL task highlighted, go to properties and click expressions
Pick the property 'SQLStatementSource'. In the expression pick your variable that you built in step A and press evaluate.
Press OK on everything
Now whatever change occurs in my_data will automatically be applied to your execute SQL task
Try that for starters and get back to me.