I am new to SSIS. I am trying to create a separate excel file dynamically in data flow task for each iteration of the for-each loop? Please guide
You can utilize the following approach.
Create an excel file template on the folder where you want to drop the new files.
Connect your excel file destination to the template file created in the folder.
Create two variables:
variable: IterationCount Data Type Int default value 1.
Variable: FileName Data Type: string
Expression = "Mybasefilename_" + (DT_STR, 4,1252)[User::IterationCount] + ".xlsx"
On your excel file connection hit right click and hit properties go to expression and hit three ellipses and look for filename property.
Set the property value choosing #[User::Filename] variable. If the Name property is not available use the connection string property, however, you should add the folder path as part of your filename variable to create the entire file destination and name.
Last step in your FELC you need to update the IterationCount variable in each iteration.
So, we cannot catch the index of the iteration then you need to use an expression in the FELC, expression task, or a script task to update the IterationCount variable.
Expression task example:
#[User::IterationCount] = #[User::IterationCount] + 1
Helpful Links:
Microsoft - SSIS ForEach Loop Container
SSIS Expression Task
SSIS - Updating variables using Script Task
Related
I want to export some SQL Server tables to Excel but the Excel file must already exist otherwise the SSIS would throw an error. Any way to tell SSIS to create the excel file with the correct columns if the Excel file doesn't exist?
When I try to click on mapping without the proper Excel file already made it throws error below:
Create three variables in your SSIS Package
FolderPath : provide the folder path in which you want to check the file
FileName : provide the file name
FileExistsFlg : we will use this to indicate file exists or not. 1 means exists and 0 means does not exists.
After creating variables, Drag Script Task to the Control Flow Pane and Provide Variables and Finally Click on Edit Script :
In the Task Script :
public void Main()
{
String Filepath=Dts.Variables["User::FolderPath"].ValueToString()+Dts.Variables["User::FileName"].Value.ToString();
if( File.Exists(FilePath))
{ Dts.Variables["User::FileExissFlg"].Value=1; }
Dts.TaskResult= (int) ScriptResults.Success;
}
Use the variable FileExistsFlg to run the next tasks if file exists (FileExistsFlg=1).
Bring Data Flow task in Control Flow Pane and Connect Script Task to it.
Double click on Green line ( Precedence Constraint) between two tasks.
Choose Expression and Constraint from Evaluation operation drop down and in Value choose "Success". In expression write #FileExistsFlg==1.
This is the method I ended up going with. Grab a new "Execute SQL Task".
ConnectionType = EXCEL,
Connection = [YOU'R EXCEL CONNECTION MANAGER],
SQLStatement = CREATE TABLE YourTable(COL1 VARCHAR(10))
Finally set the Excel Destination DelayValidation = True and put the Execute SQL Task in front of the Excel destination somewhere in Control Flow.
This will probably throw an error if the file already exist but for my use case it's good enough.
The Foreach Loop container in my process isn't pushing the new filename into the established variable. It loops through the process as many times as there are files that meet the criteria, I just need the file name to be dynamic.
I have created a variable name that contains the full filepath of the first file in my desired directory. Looks something like C:\Somepath\ExcelFile.xlsx
I have also created a variable name ExtProperties to be used in the ConnectionString with the value "Excel 12.0;HDR=Yes"
The Foreach Loop Container has the following settings:
:
The Enumerator is set to the Foreach File Enumerator
The Folder is the directory location of my files
The Files is currently set to *.xlsx
Retrieve file name is set to Fully Qualified
The ExcelFileName variable I mentioned previously has been set at Index 0
I've created an Excel connection manager pointing to the initial file with the following relevant properties:
DelayValidation: True
Expression: I have tried both setting the ExcelFilePath to the
ExcelFileName variable and using the following for the
ConnectionString:
"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + #[User::ExcelFileName] + ";Extended Properties=\"" + #[User::ExtProperties] + "\""
Right now it is using only the ConnectionString.
Retain Same Connection: False
The data flow is using an excel source using the excel connection manager. The purpose of the dataflow is to pull the number of records from each excel file, get the name of the file and the user performing the load, and push the information into the DB. When pushed out to the DB however, the filename and record count is constantly the first file used, the same number of times as however many files meet the criteria.
I get no error messages or warnings. I have used the following script in my control flow to see if the value of the variable has been changing, but the message box popping up shows that I still get the initial value.
MessageBox.Show(Dts.Variables["User::ExcelFile"].Value.ToString());
Dts.TaskResult = (int)ScriptResults.Success;
I've been reading threads about this for days and these were the settings that were proposed to work, but this is still an issue for me. Any help would be appreciated.
From the first image, it looks like you have set the Variable ExcelFileName to be evaluated as Expression since the expression mark (fx) is shown on the variable icon:
Just remove the expression from the variable and check that the EvaluateAsExpression property is set to False
In the foreach loop editor select name and extension instead or fully qualified.
and check traverse subfolders if you have subfolders.
I have been trying to create an SSIS task which picks the MS Access file from a specific folder
and then export to SQL Server ( if that file/table found in server then skip else export).
I am new to SSIS, i have used script task to select the file names dynamically and then trying to move, but I end up getting unsatisfied results . Even I have googled and got few ideas, but still not able to get it the way I wanted. Any detailed help would be very helpful.
Note : Here, am not always sure about the filename from that folder(i.e dynamic)
There are many options for dynamically selecting files. Since you're unsure about the filename, I'm assuming this is a parameter or variable. The following is an example of checking a folder from a variable for the given file name and loading it to an SSIS object variable. These files are then loaded into a SQL Server table using the Foreach Loop. You mentioned files as opposed to a single file, so this example assumes that only part of the file name is passed in, such as would be the case if the date/UID was appended to the beginning or end of the file name.
Add a Script Task, with the parameters/variables holding the file and folder name as ReadOnlyVariables and the object variable which will store the file names during execution as a ReadWriteVariable. The code for this is at the end of this post.
The string.IndexOf method is used to check for files containing the given text, with the StringComparison.CurrentCultureIgnoreCase parameter used to make this search case-insensitive. This example uses a variable for the file path and a parameter for the file name (denoted by $Package in the parameter name).
Add a Foreach Loop of the Foreach From Variable Enumerator Enumerator type. Add the object variable that was populated in the Script Task as the Variable on the Collection page. On the Variable Mappings pane, add a string variable at index 0. This will need to be an empty string variable that will hold the name of each file.
Create a Flat File Connection Manager from an example data file. Make sure that the column names and data types are appropriately configured. To set the file name dynamically, choose the ConnectionString expression (click the ellipsis of the Expression property in the Properties window of the connection manager) and add the same string variable from the Mappings Pane of the Foreach Loop.
Inside the Foreach Loop, add a Data Flow Task with a Flat File Source using the same connection manager. Then add either an OLE DB or SQL Server Destination with your destination connection and connect the flat file source to this. I've found SQL Server Destinations to perform better, but you'll want to verify this in your own environment before making the choice. Choose the necessary table and map the columns from the flat file source accordingly.
List<string> fileList = new List<string>();
//get files from input directory
DirectoryInfo di = new DirectoryInfo(Dts.Variables["User::FilePathVariable"].Value.ToString());
foreach (FileInfo f in di.GetFiles())
{
//check for files with name containing text
if (f.Name.IndexOf(Dts.Variables["$Package::FileNameParameter"].Value.ToString(), 0, StringComparison.CurrentCultureIgnoreCase) >= 0)
{
fileList.Add(f.FullName);
}
}
//populate object variable
Dts.Variables["User::YourObjectVariable"].Value = fileList;
i have a SSIS package that outputs a csv file to a location this package will be run on a daily basis.at the moment I look if a file exists and if it does I delete it before creating a new one so that I don't get any errors with creating a file that already exists - what I want to do is create a new csv and add a date/timestamp to the end of file name. I'm not sure how I go about achieving this.
You can achieve this using the following steps:
Add a variable i.e. User::Filename
Click on the variable, and press F4 to show the properties Tab
In the Expression use an expression similar to the following:
"C:\\Filename_" + (DT_WSTR,4)YEAR(GETDATE()) + "_" + (DT_WSTR,4)MONTH(GETDATE()) + "_" + (DT_WSTR,4)DAY(GETDATE()) + ".csv"
In the Destination Flat File Connection manager, in the expressions (found in the property tab) use this variable as a Connection String expression
#[User::Filename]
I am using SSDT and working on a simple SSIS package.
The Control flow:
1. A Foreach Loop Container and seek a folder exist a "importdata{}.csv" file or not.
2. If found, a script task will set variables:
- User::FullPath = (e.g C:\importdata{}.csv)
- User::varFileNameNoExt = (importdata{}) without extension.
The {} is possible in "toy","game","food".
3. Go to dataflow
The Data Flow:
1. Flat File Source with a flat file connection, the connection string is varible and mapped connection string expression.
2.ADO.NET Destination , insert data.
My question is how can i set the ADO.NET Destination [TableOrViewName] Property in variable?
Assume the table : importdatatoy,importdatagame and importdatafood is created on SQL Server.
I try to set as "dbo"."[User::varFileNameNoExt]" ,but it cannot resolve the table name on runtime.
ADO.NET Destination [TableOrViewName] parametrization can be done at Data flow level. In data flow properties, you can specify "ADO.NET Destination [TableOrViewName]".
Also specify the quotes while assigning value to variable
Eg: varFileNameNoExt = "dbo"."tableName"
But first you will need to create mapping with an existent table.
Can you post your error message? I'm thinking you won't be able to combine static text and a variable like that inside of the TableOrViewName field. Instead do the combination in a new [User::varTableName] SSIS variable and use the Advanced Properties Expression editor to set the TableOrViewName to this new SSIS variable. Have a look here.