I have an ssis package that loops through files and import the files to a SQL DB. In the loop the file is picked up regardless if it is a duplicate file. In my data flow i create a key that ignores the duplicates and imports new records.
After importing the file and ignoring duplicate i have a script task that check if the file exists in a destination folder.
string filepath;
filepath = Dts.Variables["User::FILE_PATH_VAR"].Value.ToString();
Dts.Variables["User::FILE_EXISTS"].Value = File.Exists(filepath);
Dts.TaskResult = (int) ScriptResults.Success
Up to this point the package is working. What i want to do is, if the file exists in the destination folder move the file to a duplicate destination folder. If the file does not exist move it to the destination folder.
I have tried two file system tasks with constraint editor to the one on File exist = true and the other = False. But it only diverts to the one and moves both files to the duplicate folder.
Related
I have a dataflow that is used to do transformation of multiple flat files from given folder using for each loop container. I have a flat file again as output file. The problem is that every time I execute the the job only the last file that got transformed will be stored in destination file.
Is there a way in SSIS I can create individual transformed output file instead on overwriting on same one over and over again?
For. eg. I have 5 flat files ,test_1.txt,test_2.txt,test_3.txt ,test4_.txt
and test_5.txt in a folder.
After the job ran I can only see the data from last file test_5.txt being
transformed in my destination file.
Here's steps on a working example I tested.
Variables
I have 3 variables defined:
FileName - To be used in the foreach loop
DestinationDir - where are the files going
SourceDir - where are the files I want to process
Foreach Loop Setup
I have a foreach loop configured as:
Expression for "Directory" set to #[User::SourceDir]
Retrieve file name set to "Name and extension"
Then under the "Variable Mappings":
That means as the foreach loop is iterating over the files in the directory it will be setting the "Name and extension" of the file its on to the variable #[User:FileName]
Data Flow Task
The I add a Data Flow Task inside the foreach loop:
Then inside the DFT I have a simple Flat File Source to Flat File Destination. We'll just pass the contents of each file to new files:
During initial development I'll manually pick one file to walk through setting each of the source and destinations. Then come back and change the connection managers and set an expression on the ConnectionString.
Connection Manager Expressions
SourceFile Connection Manager:
ConnectionString gets an expression as: #[User::SourceDir] + #[User::FileName]
DestinationFile Connection Manager:
ConnectionString gets an expression as: #[User::DestinationDir] + #[User::FileName]
Testing
I have 2 test files in my source directory and no files in my destination:
After I execute my package I get success and also get new files in my destination:
There are ways to do what you are asking in SSIS with variables and expressions but there is an easier way to accomplish it using command line.
Since you are just consolidating a text files into 1 you can use a command prompt to better handle your issue:
copy *.txt output.txt
I created a ssis package to move backup files from c: location to E location:
source file path: C:\Backup\A\xxx.bak
destination file path: E:\DB Backups\A\xxx.bak
I used foreachloop container and created two variables: sourcefileV and destinationV and used file system task to rename file. The package works however after moved the file to destination, the file is not in subfolder E:\DB Backups\A, it's in E:\DB Backups. I have to manually cut and move it to the subfolder. Is there any way to directly move file to the destination subfolder in ssis package?
Thanks,
You can create two variables #[User::DestinationFolder] and #[User::SourceFolder].
#[User::SourceFolder] = C:\Backup\
#[User::DestinationFolder] = E:\DB Backups\
And you can use the following expression to create the destination folder path.
#[User::DestinationFolder] + SUBSTRING(REPLACE(#[User::SourceFileV],#[User::SourceFolder],""),1, LEN(REPLACE(#[User::SourceFileV],#[User::SourceFolder],"")) - FINDSTRING(REVERSE(#[User::SourceFileV]),"\\",1))
And you can use the following expression to get the destination file path
#[User::DestinationFolder] + REPLACE(#[User::SourceFileV],#[User::SourceFolder],"")
First, you have to add an Execute File System Task to create the destination directory. Then you have to add a second Execute File System Task to copy the file.
I have a package which save multiple(14) csv files with region name to its respective Region (14) folders. Now i would like to copy all the newly created 14 csv files to a single folder . How can i achieve it through SSIS and can anyone help me achieving it please.
Any help much appreciated.
There are 3 ways to copy files between directories in SSIS:
Using Execute Process Task
you can use a similar command to copy files:
COPY c:\Source\*.txt c:\Destination
Using a Script Task
You can write a small script that loop over files in a directory and copy them to a destination
string fileName = string.Empty;
string destFile = string.Empty;
string sourcePath = #"C:\test1";
string targetPath = #"C:\test2";
// Create a new target folder, if necessary.
if (!System.IO.Directory.Exists(targetPath))
{
System.IO.Directory.CreateDirectory(targetPath);
}
if (System.IO.Directory.Exists(sourcePath))
{
string wildcard = "*.txt";
string[] files = System.IO.Directory.GetFiles(sourcePath, wildcard);
// Copy the files and overwrite destination files if they already exist.
foreach (string s in files)
{
fileName = System.IO.Path.GetFileName(s);
destFile = System.IO.Path.Combine(targetPath, fileName);
System.IO.File.Copy(s, destFile, true);
}
}
else
{
throw new Exception("Source path does not exist!");
}
Reference: Copying all files in SSIS without Foreach
Using a File System Task
You can refer to one of the following links, to get an idea on how File System Task works
SSIS: File System Task Move and rename files in one step
Copy Files Using File System Task in SSIS
Copy Files from One Location to Another and Rename Them after Putting Time Stamp
Inside a script component:
DirectoryInfo ParentInfo = new DirectoryInfo(ParentDirectoryPath);
FileInfo[] FileArray = ParentInfo.GetFiles("*",SearchOption.AllDirectories);
foreach (FileInfo Fil in FileArray)
{
Fil.CopyTo(Destination Path);
}
Not tested, but something like this
You can get way more specific by using C# however you can use a foreach file enumerator.
Map path to parent folder and select traverse subfolders.
Add *.csv as search string.
Map full path to a variable.
Inside foreach add a file system task to copy file.
I am trying to archive a file. included in the image is the settings of my system file task. I'm trying to send a file whose name is stored in a variable within the loop, and send that file into the requisite folder.
The file is supposed to land in F:\DATA\ARCHIVE\WELLBORE\ using the filename it was given. However, it keeps landing in F:\DATA\ARCHIVE with the NAME WELLBORE, which is not what I am looking for.
What do I have missing?
Thanks
Problem
The issue is when you select Move File so you have to select the source full file path and the destination full file path, so the destination variable should contains the destination full path (with file name) and if the variable contains a folder path, the folder name will be considered as a filename and it will throw an exception.
Solution
Create a new variable #[User::DestinationFile] and assign the following expression to it:
#[User::ArchiveFolder] + RIGHT( #[User::WellBoreFile] , FINDSTRING(REVERSE( #[User::WellBoreFile] ) , "\\", 1) - 1)
This expression will add the filename to the destination path. And use this new variable as a Destination
References
SSIS EXPRESSION TO GET FILE NAME FROM FULL PATH
SSIS Expression to get filename from FilePath
Microsoft Docs article*
How to move multiple excel files to different folders based on file name in ssis? means based on the file name it will move to respective folder.
Have you tried this?
In this you can see that you have to create a foreach loop, a script task a and a file system task to move the files to destination folder.
how to move files to different folders , based on matching filename and foldername in ssis
Using Foreach loop container
You have to add a for-each loop container to loop over files in a specific directory.
Choose the follow expression as a filename:
*takeme*
Map the filename to a variable
Add a dataflow task inside the for each loop to transfer files
use the filename variable as a source
you can follow the detailed article at:
http://www.sqlis.com/sqlis/post/Looping-over-files-with-the-Foreach-Loop.aspx
if you want to add multiple filter follow my answer at:
How to add multiple file extensions to the Files: input field in the Foreach loop container SSIS
Using a script task
or you can achieve this using a script task with a similar code: (i used VB.Net)
Public Sub Main()
For Each strFile As String In IO.Directory.GetFiles("C:\New Folder\", "*takeme*", IO.SearchOption.AllDirectories)
Dim filename As String = IO.Path.GetFileName(strFile)
IO.File.Copy(strFile, "D:\New Folder\" & filename)
Next
Dts.TaskResult = ScriptResults.Success
End Sub