(I am not a vested enough member to include screen shots in my post - here is a link to a shared OneDrive folder which has article with images to explain better https://1drv.ms/f/s!Ai9bJ2dV87SLg9h5FP84If74hyUK4Q)
I am trying to log what particular stored procs have inserted, updated and deleted after they were executed via Execute SQL task within a SSIS package workflow. There was a custom logging method that a 3rd party implemented, but it worked by relating a System ParentContainer ID to a user Task ID which served as a parameter to a stored procedure which logged such information. 1) I don’t think this will work from an Execute SQL Task and 2) I want a level of detail that extends past what DML function occurred.
I have been successful in logging a “single row” by setting up a result set, using variables and via an adjacent Data Flow task using a derived column task to retrieve the variables and insert into a log table.
As an example of my current working method:
The Exec SQL Task setup
Detail of the data flow part that logs
I am now coming across stored procedures that perform multiple inserts, thus I have the need to log the additional detail - more than one row. I created variables in the proc to retrieves this multiple INSERT scenario and have a union select in the SP that yields the following result set.
I understand that I now need to use the Full Result Set setting now but for this application what method is used for to persist the result set to another step (for me a destination in order to log). Amongst research I understand how one may use a Foreach loop container but this appears to configurable to one variable which needs to be value type object. I currently have 4 variables here and unable to setup as such in the Collection section of the Foreach object.
I would appreciate any insight in achieving this or suggestion to another method altogether.
many thanks in advance!
Your INSERT_B_BUDGET SQL task generates more than 1 row, you want to use Foreach Loop to catch the full result set, correct?
create a variable, LoopObject, data type as Object.
edit INSERT_B_BUDGET task,
in General tab, change ResultSet to Full result set
in Result Set tab, Variable Name as LoopObject.
add Foreach loop container after INSERT_B_BUDGET task, move your Logging 1 task into the container.
in Collection tab, Enumerator, select Foreach ADO Enumerator, ADO object source variable as LoopObject, Enumeration mode, select Rows in the first table.
in Variable Mappings, add your existing 4 variables.
You have to set up a profiler trace to track what procs are being executed. You should set up filters on the trace by database and user and use the SQLProfilerTSQL_SPs template.
Related
I have the below within the Data-flow area. The problem I'm experiencing is that even if the result is 0, it is still creating the file.
Can anyone see what I'm doing wrong here?
This is pretty much expected and known annoying behavior.
SSIS will create an empty flat file, even if unchecked: "column names in a first data row".
The workarounds are:
remove such file by a file system task if #RowCountWriteOff = 0 just after the execution of a dataflow.
as alternative, do not start a dataflow if expected number of rows in the source is 0:
Update 2019-02-11:
Issue I have is that I have 13 of these export to csv commands in the
data flow and they are costly queries
Then double querying a source to check a row-count ahead will be even more expensive and perhaps better to reuse a value of variable #RowCountWriteOff.
Initial design has 13 dataflows, adding 13 constraints and 13 filesystem tasks the main control flow will make package more complex and harder to maintain
Therefore, suggestion is to use a OnPostExecute event handler, so cleanup logic is isolated to some certain dataflow:
Update 1 - Adding more details based on OP comments
Based on your comment i will assume that you want to loop over many tables using SQL Commands, check if table contains row, if so then you should export rows to flat files, else you should ignore the tables. I will mention the steps that you need to achieve that and provide links that contains more details for each step.
First you should create a Foreach Loop container to loop over tables
You should add an Execute SQL Task with a count command SELECT COunt(*) FROM ....) and store the Resultset inside a variable
Add a Data Flow Task that import data from OLEDB Source to Flat File Destination.
After that you should add a precedence constraint with expression, to the Data Flow Task, with expression similar to #[User::RowCount] > 0
Also, it is good to check the links i provided because they contains a lot of useful informations and step by step guides.
Initial Answer
Preventing SSIS from creating empty flat files is a common issue that you can find a lot of references online, there are many workarounds suggested and many methods that may solves the issue:
Try to set the Data Flow Task Delay Validation property to True
Create another Data Flow Task within the package, which will be used only to count rows in the Source, if it is bigger than 0 then the precedence constraint should led to the other Data Flow Task
Add a File System Task after the Data Flow Task which delete the output file if RowCount is o, you should set the precedence constraint expression to ensure that.
References and helpful links
How to prevent SSIS package creating empty flat file at the destination
Prevent SSIS from creating an empty flat file
Eliminating Empty Output Files in SSIS
Prevent SSIS for creating an empty csv file at destination
Check for number of rows returned and do not create empty destination file
Set the Data Flow Task Delay Validation property to True
I have some variables (at package level, at dataflow level, doesn't matter) and I need to load them by running an Execute Sql Task.
I added an Execute Sql Task, I have wrote my sql statement in filed SqlStatement, I have changed the ForcedExecutionValueType to string.
Now I want that when the Execute Sql Task is executed the return value that that select returns, to be the exact value for my variable.
example:
Select employer_name from employers where id=1
Returs="John Doe"
value for #employer_name must be "John Doe"
Any hints?
Your more detailed discussion of the issue is a little difficult to follow, so I aplogize if my following steps cover items with you are already familiar.
Steps:
Under the Control Flow tab, drag over an Execute SQL Task and open it.
Select General on the left.
Define the Connection (based on a database source created in Connection Managers below).
Define the SQL Statement.
For Result Set, select Single Row.
Select Result Set on the left.
Select the Add button and notice that a new record was added above.
Under the Result name column header for the new record, change "NewResultname" to "0".
Under Variable Name column header for the new record, select either an existing User variable you've already created or scroll to the top and create a new one.
Select OK and you're done.
Running the Execute SQL Task by itself will now populate the variable. Make certain you've verified that the SQL will return only one value and no more. Otherwise, you will need to modify your SQL with "TOP 1" in it to be on the safe side. When multiple values are expected, then you to apply a variable defined with a Data Type of "Object" and use "Full Result set" instead of "Single Row" in the Execute SQL Task.
Hope this helps.
I ran into a very odd situation with an SSIS package I have been working with.
I have a foreach loop that contains a data flow task which executes for every record in a recordset object populated prior to the loop.
In the dataflow task I have an ADO.NET source, a lookup, and finally a destination script component. The script component initializes a datatable in PreExecute, adds each row from the input, and post sends the datatable as a table valued parameter to a stored procedure. The lookup task is set up to only send non-matching rows to the script component.
The odd situation is that I found when all rows on the lookup were matched the script component was still being executed with 0 rows sent to it. This resulted in an error in the stored proc as I was only ever expecting it to be called when the script component received more than 0 rows. I have corrected the issue in the stored proc itself by first doing a check that there is actually data passed in.
In other packages I have very similar situations with destination script components set up after lookups where 0 records get passed in, and have not seen this behavior occur there - the only difference is that they're not within foreach loops.
So, I'd really like to know, for future reference, if it is the expected behavior of an an SSIS script component that it would execute with 0 rows passed in?
PreExecute and PostExecute methods will execute even if there are no rows to redirect from the following Lookup Transformation outputs to the Script Component configured as Destination component within Data Flow Task.
Lookup Match Output
Lookup No Match Output
Make sure that the package that you believe is working differently than the one not working is configured in the same manner.
I am working with SSIS 2008. I have a select query name sqlquery1 that returns some rows:
aq
dr
tb
This query is not implemented on the SSIS at the moment.
I am calling a stored procedure from an OLE DB Source within a Data Flow Task. I would like to pass the data obtained from the query to the stored procedure parameter.
Example:
I would like to call the stored procedure by passing the first value aq
storedProdecure1 'aq'
then pass the second value dr
storedProdecure1 'dr'
I guess it would be something like a cycle. I need this because the data generated by the OLE DB Source through the stored procedure needs to be sent to another destination and this must be done for each record of the sqlquery1.
I would like to know how to call the query sqlquery1 and pass its output to call another stored procedure.
How do I need to do this in SSIS?
Conceptually, what your solution will look like is an execute your source query to generate your result set. Store that into a variable and then you'll need to do iterate through those results and for each row, you'll want to call your stored procedure with that row's value and send the results into a new Excel file.
I'd envision your package looking something like this
An Execute SQL Task, named "SQL Load Recordset", attached to a Foreach Loop Container, named "FELC Shred Recordset". Nested inside there I have a File System Task, named "FST Copy Template" which is a precedence for a Data Flow Task, named "DFT Generate Output".
Set up
As you're a beginner, I'm going to try and explain in detail. To save yourself some hassle, grab a copy of BIDSHelper. It's a free, open source tool that improves the design experience in BIDS/SSDT.
Variables
Click on the background of your Control Flow. With nothing selected, right-click and select Variables. In the new window that pops up, click the button that creates a New Variable 4 times. The reason for clicking on nothing is that until SQL Server 2012, the default behaviour of variable creation is to create them at the scope of the current object. This has resulted in many lost hairs for new and experienced developers alike. Variable names are case sensitive so be aware of that as well.
Rename Variable to RecordSet. Change the Data type from Int32 to Object
Rename Variable1 to ParameterValue. Change the data type from Int32 to String
Rename Variable2 to TemplateFile. Change the data type from Int32 to String. Set the value to the path of your output Excel File. I used C:\ssisdata\ShredRecordset.xlsx
Rename Variable 4 to OutputFileName. Change the data type from Int32 to String. Here we're going to do something slightly advanced. Click on the variable and hit F4 to bring up the Properties window. Change the value of EvaluateAsExpression to True. In Expression, set it to "C:\\ssisdata\\ShredRecordset." + #[User::ParameterValue] + ".xlsx" (or whatever your file and path are). What this does, is configures a variable to change as the value of ParameterValue changes. This helps ensure we get a unique file name. You're welcome to change naming convention as needed. Note that you need to escape the \ any time you are in an expression.
Connection Managers
I have made the assumption you are using an OLE DB connection manager. Mine is named FOO. If you are using ADO.NET the concepts will be similar but there will be nuances pertaining to parameters and such.
You will also need a second Connection Manager to handle Excel. If SSIS is temperamental about data types, Excel is flat out psychotic-stab-you-in-the-back-with-a-fork-while-you're-sleeping about data types. We're going to wait and let the data flow actually create this Connection Manager to ensure our types are good.
Source Query to Result Set
The SQL Load Recordset is an instance of the Execute SQL Task. Here I have a simple query to mimic your source.
SELECT 'aq' AS parameterValue
UNION ALL SELECT 'dr'
UNION ALL SELECT 'tb'
What's important to note on the General tab is that I have switched my ResultSet from None to Full result set. Doing this makes the Result Set tab go from being greyed out to usable.
You can observe that I have assigned the Variable Name to the variable we created above (User::RecordSet) and I the Result Name is 0. That is important as the default value, NewResultName doesn't work.
FELC Shred Recordset
Grab a Foreach Loop Container and we will use that to "shred" the results that were generated in the preceding step.
Configure the enumerator as a Foreach ADO Enumerator Use User::RecordSet as your ADO object source variable. Select rows in the first table as your Enumeration mode
On the Variable Mappings tab, you will need to select your variable User::ParameterValue and assign it the Index of 0. This will result in the zerotth element in your recordset object being assigned to the variable ParameterValue. It is important that you have data type agreement as SSIS won't do implicit conversions here.
FST Copy Template
This a File System Task. We are going to copy our template Excel File so that we have a well named output file (has the parameter name in it). Configure it as
IsDestinationPathVariable: True
DestinationVarible: User::OutputFileName
OverwriteDestination: True
Operation: Copy File
IsSourcePathVariable: True
SourceVariable: User::TemplateFile
DFT Generate Output
This is a Data Flow Task. I'm assuming you're just dumping results straight to a file so we'll just need an OLE DB Source and an Excel Destination
OLEDB dbo_storedProcedure1
This is where your data is pulled from your source system with the parameter we shredded in the Control Flow. I am going to write my query in here and use the ? to indicate it has a parameter.
Change your Data access mode to "SQL Command" and in the SQL command text that is available, put your query
EXECUTE dbo.storedProcedure1 ?
I click the Parameters... button and fill it out as shown
Parameters: #parameterValue
Variables: User::ParameterValue
Param direction: Input
Connect an Excel Destination to the OLE DB Source. Double click and in the Excel Connection Manager section, click New... Determine if you're needing 2003 or 2007 format (.xls vs .xlsx) and whether you want your file to have header rows. For you File Path, put in the same value you used for your #User::TemplatePath variable and click OK.
We now need to populate the name of the Excel Sheet. Click that New... button and it may bark that there is not sufficient information about mapping data types. Don't worry, that's semi-standard. It will then pop up a table definition something like
CREATE TABLE `Excel Destination` (
`name` NVARCHAR(35),
`number` INT,
`type` NVARCHAR(3),
`low` INT,
`high` INT,
`status` INT
)
The "table" name is going to be the worksheet name, or precisely, the named data set in the worksheet. I made mine Sheet1 and clicked OK. Now that the sheet exists, select it in the drop down. I went with the Sheet1$ as the target sheet name. Not sure if it makes a difference.
Click the Mappings tab and things should auto-map just fine so click OK.
Finally
At this point, if we ran the package it would overwrite the template file every time. The secret is we need to tell that Excel Connection Manager we just made that it needs to not have a hard coded name.
Click once on the Excel Connection Manager in the Connection Managers tab. In the Properties window, find the Expressions section and click the ellipses ... Here we will configure the Property ExcelFilePath and the Expression we will use is
#[User::OutputFileName]
If your icons and such look different, that's to be expected. This was documented using SSIS 2012. Your work flow will be the same in 2005 and 2008/2008R2 just the skin is different.
If you run this package and it doesn't even start and there is an error about the ACE 12 or Jet 4.0 something not available, then you are on a 64bit machine and need to tell BIDS/SSDT that you want to run in 32 bit mode.
Ensure the Run64BitRuntime value is False. This project setting can be found by right clicking on the project, expand the Configuration Properties and it will be an option under Debugging.
Further reading
A different example of shredding a recordset object can be found on How to automate the execution of a stored procedure with an SSIS package?
I am new with SSIS packages, There is a specific process I want to accomplish.
1st) I want to get a record set from a SQL Server DB and Mark that I have retrieved the specific rows.
2nd) Loop through each row
3rd) for each row looped through call a web service and pass two fields to the web service as input parameters.
Any help will be much appreciated.
The first step to do this is to create a few SSIS variables. Select View > Other Windows > Variables to display the variables window. Add a variable that will contain your result set from the SQL query. Set the data type to Object. Create additional variables for each column selected in the query.
In the Connection Managers tab, add a new Connection Manger. Edit and configure the new Connection Manager with the SQL Server connection information (e.g., Server name, authentication and database, etc.)
Add an Execute SQL Task to the Control Flow. Edit the task. In the Connection property, select your new Connection Manager. Enter the SQL SELECT statement in the SQLStatement property. Set the ResultSet property to Full result set.
Select the Result Set tab in the Execute SQL Task. Add a new result set. Set the Result Name to 0 and the Variable Name to the object variable you created earlier. Close the editor.
Add a Foreach Loop Container to Control Flow. Create a Precedence Constraint (green line) from the Execute SQL Task to the Foreach Loop Container. This will ensure that the SQL task must succeed before the Foreach Loop starts.
Edit the Forach Loop Container and select the Collection tab. Set the Enumerator property to Foreach ADO Enumerator and select your object variable in the ADO object source variable. Select the Variable Mappings tab. Select your variables that will retrieve the column value for each row. Associate the first variable with the first column by selecting index 0. Index 1 will be the second column, etc.
Add a Web Service Task in the Foreach Loop Container. This task will run once for each row in the result set. You can use the variable values to configure the Web Service Task. You may need to work with expressions if the property doesn't directly accept variables.
If you want to mark your rows one at a time, you can add another Execute SQL Task and place it into the Foreach Loop Container. Configure it similarly to the first Execute SQL Task, but create an SQL UPDATE statement. You might consider updating all rows the rows at once to get better performance. If so, then move the Execute SQL Task out of the Foreach Loop Container and configure the UPDATE statement so that it updates the set of rows.