The Foreach Loop container in my process isn't pushing the new filename into the established variable. It loops through the process as many times as there are files that meet the criteria, I just need the file name to be dynamic.
I have created a variable name that contains the full filepath of the first file in my desired directory. Looks something like C:\Somepath\ExcelFile.xlsx
I have also created a variable name ExtProperties to be used in the ConnectionString with the value "Excel 12.0;HDR=Yes"
The Foreach Loop Container has the following settings:
:
The Enumerator is set to the Foreach File Enumerator
The Folder is the directory location of my files
The Files is currently set to *.xlsx
Retrieve file name is set to Fully Qualified
The ExcelFileName variable I mentioned previously has been set at Index 0
I've created an Excel connection manager pointing to the initial file with the following relevant properties:
DelayValidation: True
Expression: I have tried both setting the ExcelFilePath to the
ExcelFileName variable and using the following for the
ConnectionString:
"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + #[User::ExcelFileName] + ";Extended Properties=\"" + #[User::ExtProperties] + "\""
Right now it is using only the ConnectionString.
Retain Same Connection: False
The data flow is using an excel source using the excel connection manager. The purpose of the dataflow is to pull the number of records from each excel file, get the name of the file and the user performing the load, and push the information into the DB. When pushed out to the DB however, the filename and record count is constantly the first file used, the same number of times as however many files meet the criteria.
I get no error messages or warnings. I have used the following script in my control flow to see if the value of the variable has been changing, but the message box popping up shows that I still get the initial value.
MessageBox.Show(Dts.Variables["User::ExcelFile"].Value.ToString());
Dts.TaskResult = (int)ScriptResults.Success;
I've been reading threads about this for days and these were the settings that were proposed to work, but this is still an issue for me. Any help would be appreciated.
From the first image, it looks like you have set the Variable ExcelFileName to be evaluated as Expression since the expression mark (fx) is shown on the variable icon:
Just remove the expression from the variable and check that the EvaluateAsExpression property is set to False
In the foreach loop editor select name and extension instead or fully qualified.
and check traverse subfolders if you have subfolders.
Related
I am new to SSIS. I am trying to create a separate excel file dynamically in data flow task for each iteration of the for-each loop? Please guide
You can utilize the following approach.
Create an excel file template on the folder where you want to drop the new files.
Connect your excel file destination to the template file created in the folder.
Create two variables:
variable: IterationCount Data Type Int default value 1.
Variable: FileName Data Type: string
Expression = "Mybasefilename_" + (DT_STR, 4,1252)[User::IterationCount] + ".xlsx"
On your excel file connection hit right click and hit properties go to expression and hit three ellipses and look for filename property.
Set the property value choosing #[User::Filename] variable. If the Name property is not available use the connection string property, however, you should add the folder path as part of your filename variable to create the entire file destination and name.
Last step in your FELC you need to update the IterationCount variable in each iteration.
So, we cannot catch the index of the iteration then you need to use an expression in the FELC, expression task, or a script task to update the IterationCount variable.
Expression task example:
#[User::IterationCount] = #[User::IterationCount] + 1
Helpful Links:
Microsoft - SSIS ForEach Loop Container
SSIS Expression Task
SSIS - Updating variables using Script Task
I have been trying to create an SSIS task which picks the MS Access file from a specific folder
and then export to SQL Server ( if that file/table found in server then skip else export).
I am new to SSIS, i have used script task to select the file names dynamically and then trying to move, but I end up getting unsatisfied results . Even I have googled and got few ideas, but still not able to get it the way I wanted. Any detailed help would be very helpful.
Note : Here, am not always sure about the filename from that folder(i.e dynamic)
There are many options for dynamically selecting files. Since you're unsure about the filename, I'm assuming this is a parameter or variable. The following is an example of checking a folder from a variable for the given file name and loading it to an SSIS object variable. These files are then loaded into a SQL Server table using the Foreach Loop. You mentioned files as opposed to a single file, so this example assumes that only part of the file name is passed in, such as would be the case if the date/UID was appended to the beginning or end of the file name.
Add a Script Task, with the parameters/variables holding the file and folder name as ReadOnlyVariables and the object variable which will store the file names during execution as a ReadWriteVariable. The code for this is at the end of this post.
The string.IndexOf method is used to check for files containing the given text, with the StringComparison.CurrentCultureIgnoreCase parameter used to make this search case-insensitive. This example uses a variable for the file path and a parameter for the file name (denoted by $Package in the parameter name).
Add a Foreach Loop of the Foreach From Variable Enumerator Enumerator type. Add the object variable that was populated in the Script Task as the Variable on the Collection page. On the Variable Mappings pane, add a string variable at index 0. This will need to be an empty string variable that will hold the name of each file.
Create a Flat File Connection Manager from an example data file. Make sure that the column names and data types are appropriately configured. To set the file name dynamically, choose the ConnectionString expression (click the ellipsis of the Expression property in the Properties window of the connection manager) and add the same string variable from the Mappings Pane of the Foreach Loop.
Inside the Foreach Loop, add a Data Flow Task with a Flat File Source using the same connection manager. Then add either an OLE DB or SQL Server Destination with your destination connection and connect the flat file source to this. I've found SQL Server Destinations to perform better, but you'll want to verify this in your own environment before making the choice. Choose the necessary table and map the columns from the flat file source accordingly.
List<string> fileList = new List<string>();
//get files from input directory
DirectoryInfo di = new DirectoryInfo(Dts.Variables["User::FilePathVariable"].Value.ToString());
foreach (FileInfo f in di.GetFiles())
{
//check for files with name containing text
if (f.Name.IndexOf(Dts.Variables["$Package::FileNameParameter"].Value.ToString(), 0, StringComparison.CurrentCultureIgnoreCase) >= 0)
{
fileList.Add(f.FullName);
}
}
//populate object variable
Dts.Variables["User::YourObjectVariable"].Value = fileList;
I am using SSDT and working on a simple SSIS package.
The Control flow:
1. A Foreach Loop Container and seek a folder exist a "importdata{}.csv" file or not.
2. If found, a script task will set variables:
- User::FullPath = (e.g C:\importdata{}.csv)
- User::varFileNameNoExt = (importdata{}) without extension.
The {} is possible in "toy","game","food".
3. Go to dataflow
The Data Flow:
1. Flat File Source with a flat file connection, the connection string is varible and mapped connection string expression.
2.ADO.NET Destination , insert data.
My question is how can i set the ADO.NET Destination [TableOrViewName] Property in variable?
Assume the table : importdatatoy,importdatagame and importdatafood is created on SQL Server.
I try to set as "dbo"."[User::varFileNameNoExt]" ,but it cannot resolve the table name on runtime.
ADO.NET Destination [TableOrViewName] parametrization can be done at Data flow level. In data flow properties, you can specify "ADO.NET Destination [TableOrViewName]".
Also specify the quotes while assigning value to variable
Eg: varFileNameNoExt = "dbo"."tableName"
But first you will need to create mapping with an existent table.
Can you post your error message? I'm thinking you won't be able to combine static text and a variable like that inside of the TableOrViewName field. Instead do the combination in a new [User::varTableName] SSIS variable and use the Advanced Properties Expression editor to set the TableOrViewName to this new SSIS variable. Have a look here.
I have an SSIS package that has a Foreach Loop container loading all .txt files in a static folder. I'm passing the fully qualified file name as a variable which I'm using in the connection string.
I now need to pass just the filename to a variable to use to execute a stored procedure, trouble is if I change the Foreach Loop collection to just retrieve the filename it prevents the rest from working.
Is there a way to pass both the fully qualified and name only file name to a variable in ssis?
Nope, you get to pick one: either fully qualified or file name only. Personally, if everything except the stored procedure expects fully qualified name, I'd create a second variable FileNameOnly and either populate it with an expression like
RIGHT(#[User::CurrentFileName], FINDSTRING(REVERSE(#[User::CurrentFileName]), "\\", 1 )-1)
(this assumes you've stored the value in a variable called CurrentFileName)
Or you can use a script Task to assign the value to FileNameOnly and leverage the .NET System.IO.Path.GetFileName method.
You can add another variable #[User::Filename] and inside the Foreach loop container add an Expression Task with the following expression:
#[User::Filename] = TOKEN(#[User::FilePath],"\\", TOKENCOUNT(#[User::FilePath],"\\"))
I have static table that has the has something like this:
xy_Jan10
yz_Feb11
xx_March14
by_Aug09
etc.
these names are static and they are stored in a table. I am using ForEachLoop container, so first i am reading and saving all the static names that i mentioned above into an system object variable. Next i am using the ForeachLoop containter and looping through for each of the file name and saving it into another string variable called strFileName. So in my for each loop container, i have script task that checks first if the file exists and here is where i have the problem, for each file name that comes to the variable i want to check if that file name exist firs, if exists i want to load it into my table, if not exist then i want to check the next file name, if next file name does not exist then i want to check the next variable name inline and so on. I only want to load if the variable file name matches the files on the network drive, if it is not found then i want to check next one until i go through each one in my static list names. My issue now script task stops when there is no match with the file names but i want it to go to the next variable name in the list and load it because there are a lot of other matches that are not loaded. the script task stops at the first one where it finds non much. Here is my script task:
please not the files i am loading are SAS files.
Public Sub Main()
'
' Add your code here
'
Dim directory As DirectoryInfo = New DirectoryInfo("\\840KLM\DataMart\CPTT\Cannon")
Dim file As FileInfo() = directory.GetFiles("*.sas7bdat")
If file.Length > 0 Then
Dts.Variables("User::var_IsFileExist").Value = True
Else
Dts.Variables("User::var_IsFileExist").Value = False
End If
Dts.TaskResult = ScriptResults.Success
End Sub
It looks like you need to wrap the script task inside a ForEach loop container. There's plenty of information about how to do this on the web, or even on Stack Overflow: How to loop through Excel files and load them into a database using SSIS package?
or
http://www.sqlis.com/sqlis/post/Looping-over-files-with-the-Foreach-Loop.aspx