I have a question in SSIS.
For an instance, I have 100 flat files of the same metadata columns to be loaded using an incremental load, but my question is how can we find which flat contains error data while loading using for each loop container. Any solution can be appreciated
Simplest solution: Since you are using a Foreach Loop container, then the file path is mapped to a variable. You can simply add a Derived Column Transformation and use this variable within the expression as following (assuming the variable name is FilePath):
#[User::FilePath]
Then insert it with the erroneous rows.
Related
I have an SSIS package that has this Foreach Loop Container(with File Enumerator) that reads from a folder with multiple CSVs file and then upload the data into a flat table.
This is working fine but my problem is trying to also extract the filenames of the file and then populate the last column in the flat table after inserting a row.
I have also added an execute SQL task after the Data Flow task(within the ForEach Loop Container) hoping that it would execute straight away before the loop goes to the next file, but unfortunately this is not the behavior.
The execute SQL task will only execute after all the data in all the files is read. Is there a way to do this filename update row by row, such as read a row from the CSV file, insert this row into the table, update the row in the filename column in the same table, and then read the next row? Continue this way until the CSV is read completely then move to the next CSV file and do the same.
I have a programming background and slightly feel that nested for loops could be a way but not sure how to achieve this in SSIS. The setup of my ForEach loop container is shown below:
Why using an Execute SQL Task to add the file name?!
You can simply add the file name into the data pipeline using one of the following methods:
(1) Using the FileNameColumnName property
In the Data Flow Task, you can simply right-click on the Flat File Source, and click on the Show Advanced Editor option.
In the Flat File Source Advanced Editor, there is a property called FileNameColumnName. This property is used to add a column to the flat file source where the File Name is added.
You should only write the value of the column name and it will be added to the flat file source.
Flat File Custom Properties
Extract the File Name in SSIS Data Flows using the FileNameColumnName Property
(2) Using a Derived Column Transformation
Your issue can be solved by adding a Derived Column Transformation within the Data Flow Task. Then, add a column to the data pipeline using the variable that contains the File Name. (The variable used in the ForEach Loop Container's variables mappings tab)
You can learn more about Derived Column Transformation in the following article:
SSIS Derived Columns with Multiple Expressions vs Multiple Transformations
Similar questions:
How to find which flat file contains data errors while loading multiple flat files using Foreach File enumerator in SSIS
I need to import salary data from multiple excel files where filename of each file is a date.
I used SSIS and with success followed typical tutorials for importing multiple excel files. The thing is none of them show simple method how to add one extra column (with the name of the file) to the result. There are some tutorials with huge code scripts, that are too complicated for me.
What I did was to add 'Derived column' module between typical 'excel source' and 'OLE BD destination' where I added a new column 'date' with expression #[User::FileName] - a variable that is used for 'foreach loop container' but as a result I received corectly combined data from all files but the extra column contains the same data - the filename of the first imported file.
I wonder if there is any simple to make the variable I used to change with every loop ? So, as a result, I receive combined data plus one extra column containing the corresponding date, which is the name of each file. Many Thanks
If you are using a Foreach File Enumerator, select "Name only" in the Collection pane as in the image below (I'm assuming that "Name only" will give you the date that you are looking for).
This allows you to map this into a variable on each iteration. To do this, navigate to the Variable Mappings pane, and select the variable you want to use in your Data Flow Task, with 0 as the Index.
You can then add this variable as a Derived Column, and it will give you the name of the file you are importing.
I have an SSIS package which checks for the unprocessed file present in a tracking table and then processed it. Till date only one file would come in and we would process it and as such the process was designed accordingly.
However now multiple files can come in one go and we store those multiple files in the tracking table and we have a column which keeps a track of the unprocessed file.
I am trying to use the For Each loop to process all the unprocessed file. So I get the count of the unprocessed files and would like to simply tun the Point 1 by passing a parameter to the step 1 but I have not been successful in doing it using Foreach From Variable Enumerator. Am I missing something ?
You can do this using the following steps:
Add an Execute SQL Task to get unprocessed files and store the resultset inside a variable of type System.Object
Add a Foreach loop container, change the type to ADO enumerator and select the variable as source
In the variable mapping tab map the result (each file path) to a variable of type string
Inside the foreach loop container add a dataflow task that contains the Flat File source and implement the processing logic you need
Add a flat file connection manager define the columns
Click on the flat file connection manager, press F4 to show the property tab, go to expression.
Select the connectionstring property and use the variable that holds the filepath as expression
Detailed articles
Implementing Foreach Looping Logic in SSIS
Looping Through a Result Set with the ForEach Loop
Using SSIS to Loop Over Result Set and Dynamically Generate Output Files
How to loop through full result set using foreach container in SSIS
I've chased my tail for a full 12 hours. Haven't found the right solution.
I'm locked into using SSIS. I have a SQL Server table with full paths and filenames already concatenated. Examples:
\\MydevServer1\C$\ABC\App_Data\Sample.pdf
\\MydevServer2\E$\Garth\App_Data\Morefiles.txt
\\MydevServer3\D$\Paths\App_Data\MySS.xlsx
etc.
I need to read each row of the table, get the path and filename and move that file to a new static destination directory.
The rows in the table will remain unchanged. I only use it as a source to locate the file to be moved.
I've tried:
1) Feeding a resultset from an ole db source to a recordset destination then to an Object variable that connects via variable to a foreach loop container holding a files system task. (Very problematic.)
2) Sending the table rows to a .csv file and reading each line of the csv file using a foreach loop container holding a file system task.
3) Reading directly from the table rows using a foreach loop container holding a file system task. (preferred).
and many other scenarios.
I have viewed a hundred examples online, but most of them involve loading a table, or sending results to flat files, or moving files from one folder to another based on extension type, etc. I haven't found anything on configuring a file system task to read a table supplied path and move the file based on the table value as the source.
I'm rambling. :-)
Any insight or help will be appreciated. I'm not new to SSIS, but I sure feel like it right now.
Create two string variables to store source and destination paths
Use an Execute SQL Task to populate a Full Recordset (Variable with Object data type)
Use For Loop container to go through each row of recordset and set those two variables.
Inside For Loop container, use File System Task. You need to specify IsSourcePathVariable = True, IsDestinationPathVariable = True, path variables - DestinationVariable / SourceVariable, and set operation (copy, move, etc.)
It appears I've been tail chasing due to the error, "Source is empty error".
This was caused by a blank first row in my recordset. I was searching for a fix to the Object variable is empty issue, when in reality the issue was that the Object variable couldn't find data right off the bat.
Insert shameful smug here.
Thanks to Anton for the help.
I'm little new to SSIS and I have a need to import some flat files into SQL tables in the same structure.
(Assume the table is already exist in the same structure and table name and flat file name is same)
I thought to create a generic package (sql 2014) to import all those file by looping through a folder.
I try to create a data flow task in a foreach loop container in the data flow task I dropped a flat file source and ADO.Net destination .
I have set the file source to a variable so that every time it loops through it get the new file. similarly for the ADO.net table name I set it to a the variable so that each time it select a different table according to the file name.
since both source column names and destination column names are same I assume it will map the columns automatically.
but with a simple map it didn't let me to run the package so added a column on the source and selected a table and mapped it.
when I run the package I assumed it will automatically re map everything.
but for the first file it ran but second file it failed complaining with map issues.
can some one let me know whether this is achievable by doing some dynamic mapping?? or using any other way.
any help would be much appreciated.
thanks
Ned