i have a folder containing multiple excel file. Excel files name are almost same except every file name contain month and year number in last.
Example
Emp_04_2017.xlsx
Emp_05_2017.xlsx
...
I want to create a SSIS package that pick the current month file and insert it into the destination table.
One way would be to create SSIS variables to store the current month and year, and then use those to construct the name of the file in a third variable.
Use a ForEachLoopContainer.
ForEach Loop Container will pick files from the FolderPath variable and return complete CompletePath (Path+fileName). Loop will iterate through all the files in the FolderPath location.
Foreach Loop Container: Double click -> In Collection set Expression Directory = FolderPath, Enumerator Configuration -> Files : (.xlsx).
Vaiable Mappings -> Variable (CompletePath) Index 0.
EXPR_GetFirstOcrDash: Expression used to get first occurrence of dash in filename, #[User::FirstOcr] = FINDSTRING(REVERSE(#[User::CompletePath]), "_", 1).
EXPR_ExtractFileName: The expression is used to get month from the file name, #[User::FileMonth] = (REVERSE(SUBSTRING(REVERSE(#[User::CompletePath]), #[User::FirstOcr]+2, 1)) == "0" ?
REVERSE(SUBSTRING(REVERSE(#[User::CompletePath]), #[User::FirstOcr]+1, 1))
:REVERSE(SUBSTRING(REVERSE(#[User::CompletePath]), #[User::FirstOcr]+1, 2)))
EXPR_SetFileToProcess: Used to set the file which we found for processing, #[User::FileToProcess] = #[User::CompletePath]
EXPR_StopProcessing: The loop will continously check all files in the folder, when we found first file with month of current date, we will not further look for file.
A better practice could be use two directory Source and Archive, once file is processed move the processed file to Archive directory using FileSystemTask.
Precedence Constraints are added on the green arrows.
After the Foreach Loop Container gets processed, you can use FileToProcess variable and use the file in the DataFlowTask.
Related
I have bunch of XML files that I wanted to transformed them (using XSLT operations) in SSIS (SQL Server 2014). I have created my xsl and can transform the files one by one to the format that I want.
Trouble is I can't work out how to transform and move the files at once.
Up until now, I have tried a Foreach Loop Container, XML TASK and a File System Task to do the job ( off course with no luck). I have created a variable for the filenames.
Have configured the Foreach Loop Container as below:
Foreach File Enumerator for the Enumerator in the Enumerator (Collection tab)
Set the destination directory
Mapped the FileName variable
Configured the XML Task as below:
Source ==> the first file from the source folder
Destination ==> FileName variable
SecondOperandType ==> XSL file
I'm not sure what to do next:
to read the files from the source directory
transform the files one by one
move the transformed files into the destination folder
I would be really thankful if someone please help and point me to the right direction.
This below steps will answer you to loop through all xml files in folders and subfolders using for each loop container.
Take Foreach Loop Container from Toolbox
Create four different variables. (Take care that their scope must be package and not a container).
2a. FilePath - DataType String - Value (Set full XML file path like c:\data\filexml.xml)
2b. FileType - DataType String - Value (*.xml)
2c. FileFolderPath - DataType String - Value (C:\Data)
2d. DestFolderPath - DataType String - Value (C:\TransformedXMLFiles)
Right click on Foreach loop container and click edit
Foreach loop editor will be pop up
4a. Select collection in left pane.
4b. In right pane, select enumerator as Foreach File Enumerator
4c. Click on expression eclipse (...) three dot button
i. Select Directory left side and set variable as #[User::FileFolderPath]
ii. Select FileSpec left side and set variable as #[User::FileType]
4d. Select "Fully qualified" in Retrieve File Name section
4e. Select Traverse Subfolder checkbox if you want to traverse the subfolders.
4f. In left pane, select Variable Mapping
4g. Select User::FilePath variable under variable column that will be used to iterate through the files in the Folder and Subfolder specified.
4h. Put 0 (zero) value in Index column
Use your XML source container in for each loop
Use XSD or XSLT operations
Use File System Task to move file from source to destination
I have been trying to create an SSIS task which picks the MS Access file from a specific folder
and then export to SQL Server ( if that file/table found in server then skip else export).
I am new to SSIS, i have used script task to select the file names dynamically and then trying to move, but I end up getting unsatisfied results . Even I have googled and got few ideas, but still not able to get it the way I wanted. Any detailed help would be very helpful.
Note : Here, am not always sure about the filename from that folder(i.e dynamic)
There are many options for dynamically selecting files. Since you're unsure about the filename, I'm assuming this is a parameter or variable. The following is an example of checking a folder from a variable for the given file name and loading it to an SSIS object variable. These files are then loaded into a SQL Server table using the Foreach Loop. You mentioned files as opposed to a single file, so this example assumes that only part of the file name is passed in, such as would be the case if the date/UID was appended to the beginning or end of the file name.
Add a Script Task, with the parameters/variables holding the file and folder name as ReadOnlyVariables and the object variable which will store the file names during execution as a ReadWriteVariable. The code for this is at the end of this post.
The string.IndexOf method is used to check for files containing the given text, with the StringComparison.CurrentCultureIgnoreCase parameter used to make this search case-insensitive. This example uses a variable for the file path and a parameter for the file name (denoted by $Package in the parameter name).
Add a Foreach Loop of the Foreach From Variable Enumerator Enumerator type. Add the object variable that was populated in the Script Task as the Variable on the Collection page. On the Variable Mappings pane, add a string variable at index 0. This will need to be an empty string variable that will hold the name of each file.
Create a Flat File Connection Manager from an example data file. Make sure that the column names and data types are appropriately configured. To set the file name dynamically, choose the ConnectionString expression (click the ellipsis of the Expression property in the Properties window of the connection manager) and add the same string variable from the Mappings Pane of the Foreach Loop.
Inside the Foreach Loop, add a Data Flow Task with a Flat File Source using the same connection manager. Then add either an OLE DB or SQL Server Destination with your destination connection and connect the flat file source to this. I've found SQL Server Destinations to perform better, but you'll want to verify this in your own environment before making the choice. Choose the necessary table and map the columns from the flat file source accordingly.
List<string> fileList = new List<string>();
//get files from input directory
DirectoryInfo di = new DirectoryInfo(Dts.Variables["User::FilePathVariable"].Value.ToString());
foreach (FileInfo f in di.GetFiles())
{
//check for files with name containing text
if (f.Name.IndexOf(Dts.Variables["$Package::FileNameParameter"].Value.ToString(), 0, StringComparison.CurrentCultureIgnoreCase) >= 0)
{
fileList.Add(f.FullName);
}
}
//populate object variable
Dts.Variables["User::YourObjectVariable"].Value = fileList;
I have static table that has the has something like this:
xy_Jan10
yz_Feb11
xx_March14
by_Aug09
etc.
these names are static and they are stored in a table. I am using ForEachLoop container, so first i am reading and saving all the static names that i mentioned above into an system object variable. Next i am using the ForeachLoop containter and looping through for each of the file name and saving it into another string variable called strFileName. So in my for each loop container, i have script task that checks first if the file exists and here is where i have the problem, for each file name that comes to the variable i want to check if that file name exist firs, if exists i want to load it into my table, if not exist then i want to check the next file name, if next file name does not exist then i want to check the next variable name inline and so on. I only want to load if the variable file name matches the files on the network drive, if it is not found then i want to check next one until i go through each one in my static list names. My issue now script task stops when there is no match with the file names but i want it to go to the next variable name in the list and load it because there are a lot of other matches that are not loaded. the script task stops at the first one where it finds non much. Here is my script task:
please not the files i am loading are SAS files.
Public Sub Main()
'
' Add your code here
'
Dim directory As DirectoryInfo = New DirectoryInfo("\\840KLM\DataMart\CPTT\Cannon")
Dim file As FileInfo() = directory.GetFiles("*.sas7bdat")
If file.Length > 0 Then
Dts.Variables("User::var_IsFileExist").Value = True
Else
Dts.Variables("User::var_IsFileExist").Value = False
End If
Dts.TaskResult = ScriptResults.Success
End Sub
It looks like you need to wrap the script task inside a ForEach loop container. There's plenty of information about how to do this on the web, or even on Stack Overflow: How to loop through Excel files and load them into a database using SSIS package?
or
http://www.sqlis.com/sqlis/post/Looping-over-files-with-the-Foreach-Loop.aspx
I have four files xxxxxxCd999, xxxxCf999, xxxxC999 , xxxxD999 ... I need to move these files to their respective folders based on file name , for example file xxxxxCd999 should be moved to folder Cd999 , file xxxxCf999 should be moved to folder Cf999 ,file xxxC999 should ne moved to folder C999 so on ...
How do I achieve this in ssis ?
I have used a for each loop container, assigned some variables for sourcepath, destinationpath , and a file system task to use these variables , but im lost now n have no idea how to proceed ,
Kindly help me
Try this :-
The Foreach Loop will enumerate the source folder and the path will be stored in a variable. In the script task write a code to get the folder Name using regular expression .The script task value will be stored in another variable which will be used in File System Task
The package design will be
Create 3 variable
Name DataType Expression
FolderName string
DestLoc string "D:\\"+ #[User::FolderName]
LoopFiles string
In the above expression for DestLoc variable ,change it as per your location
ForEach Loop configuration
Change the source folder location as per the need
Script task -Add the 2 variable as below
You need to extract the folder name from the variable LoopFiles
Example
LoopFiles variable will have D:\ForLoop\SampleFolder1.txt at runtime
So in order to extract folder name from the above variable use regular expression
Open Edit Script and write the following code
List<string> filePatterns = null;
public void Main()
{
filePatterns = new List<string>();
filePatterns.Add("Folder1");
filePatterns.Add("Folder2");
string fileName = Path.GetFileNameWithoutExtension(Dts.Variables["User::LoopFiles"].Value.ToString());
Match match = Regex.Match(fileName, string.Join("|", filePatterns.ToArray()));
Dts.Variables["User::FolderName"].Value = match.Value;
Dts.TaskResult = (int)ScriptResults.Success;
}
In the above code ,you are extracting the folder name and storing it in the variable FolderName.If you have multiple folders ,then just add the folder names to the filePatterns collection variable.
File System Task Configuration
I use SSIS to read .txt files in input and execute my business logic over them saving the output results in a file whose name is the same as the current inpout file (file name dynamically stored in a variable).
When all the files are stored in the same folder, I have no problem accessing them since I use the following expression for the flat file connection string in the data flow: "path" + #[User::inputFileName] + ".txt"
Now I have to process a folder with subfolders (I set traverse subfolders in the foreach loop) and I have some issues with the flat file connection string since I cannot use a wildcard like: my path\\subfolder*" + #[User::inputFileName] + ".txt" where every subfolder has same name and changes only the last portion of the name.
How can I save the current subfolder name in a variable so that I can use it in the following way? "path\\"+ #[User::currentSubFolder] +"\\" + #[User::inputFileName] + ".txt"
I was able to solve my issue, therefore I write here my solution in the case someone else would be in the same situation.
I used a script transformation block before my foreach loop. From it I can retrieve the current full path (used afterwards in the Flat File connection string) and the input file name without extension to be used as output file name containing the results of the SSIS scripts.
In order to keep the values of interests I used 2 variables: one for the file name and one for the path.
Here the script code:
Public Sub Main()
'Variable Index 0 => FileName
'Variable Index 1 => filePath
Dim fullPath As String = Dts.Variables.Item(1).Value.ToString
Dim fileName As String = Path.GetFileName(fullPath)
fileName = fileName.Substring(0, fileName.Length - 4)
Dts.Variables.Item(0).Value = fileName
Dim x As String = Dts.Variables.Item(0).Value.ToString
Dts.TaskResult = Dts.Results.Success
End Sub