Currently, I have a file path value hard coded inside the Script task of an SSIS package.
I have a string variable sPath. How should I use this variable sPath inside Script Task?
string strPath = "C:\\File.xls";
if( File.Exists(strPath))
{
File.Delete(strPath);
}
Here is one possible way of using variables inside Script Task. Assuming that you have a variable named FilePath declared on your package as shown in screenshot #1 then you can use the following code to use the variable inside the Script Task. this is one of the possible ways to use the variable. Here the variable is used only to read the value using the method LockForRead. You can also write values to the variable if the variable is declared with LockForWrite method.
By the way, the functionality described in the Scrip Task code can also be carried out using the File System Task available in SSIS Control Flow task list.
Hope that helps.
Using package variables inside Script Task:
C# code that can be used only in SSIS 2008 and above.
.
public void Main()
{
Variables varCollection = null;
string FilePath = string.Empty;
Dts.VariableDispenser.LockForRead("User::FilePath");
Dts.VariableDispenser.GetVariables(ref varCollection);
FilePath = varCollection["User::FilePath"].Value.ToString();
if (File.Exists(FilePath))
{
File.Delete(FilePath);
}
varCollection.Unlock();
Dts.TaskResult = (int)ScriptResults.Success;
}
Screenshot #1:
Related
I have a Variable which uses another variable to get its value. I had set the EvaluateAsExpression property to True. Now when I use this variable in a Script task it is taking 2-3 Minutes to evaluate that expression. But if I Hard Code the variable value instead of reading it from another variable then it is finishing in less than a Second. Anyone let me know if I am missing anything ? I am using BIDS/SSIS 2008
Variable1: D:\app\srikar.mogaliraju
Variable2: #[User::Variable1]+"\\Product"
Script Task Code:
public void Main()
{
String SourceFolderPath, DestinationFolderpath;
SourceFolderPath = Dts.Variables["User::Variable2"].Value.ToString();
//Random Code
Dts.TaskResult = (int)ScriptResults.Success;
}
Variable 2 Expression Builder:
I haven't got any solution for this. So posting the answer with the solution I implemented to bypass this problem.
Use ExecuteSqlTask to read Variable1 and Append "\Product" to Variable1 value.Set the result set property to Single Row.
Assign the Result Set to Variable2 and use this Variable in Script task.
I have a process built in SSIS that loops through Excel files and Import data only from those that include name Report.
My UserVariable used as Expression is: *Report*.xlsx
and it works perfectly fine. Now I am trying to build similar loop but only for files that DOES NOT include Report in file name.
Something like *<>Report*.xlsx
Is it possible?
Thanks for help!
Matt
In your loop, put a Script task before your first task. Connect those two with a line. Right click that line and set Constraint Options to expression. Your expression would look like this...
FINDSTRING(#var, "Report", 1) == 0
Where #var is the loop iterable.
Only files without "Report" inside will proceed to the next step.
Referencing this exact answer. SSIS Exclude certain files in Foreach Loop Container
Unfortunately, you cannot achieve this using SSIS expression (something like *[^...]*.xlsx) you have to search for some workarounds:
Workarounds
First
Get List of - filtered - files using an Execute Script Task before entering Loop and loop over then using ForEach Loop container (Ado enumerator)
You have to a a SSIS variable (ex: User::FilesList) with type System.Object (Scope: Package)
Add an Execute Script Task befor the for each Loop container and add User::FilesList as a ReadWrite Variable
In the Script Write The following Code:
Imports System.Linq
Imports System.IO
Imports System.Collections.Generic
Public Sub Main()
Dim lstFiles As New List(Of String)
lstFiles.AddRange(Directory.GetFiles("C:\Temp", "*.xlsx", SearchOption.TopDirectoryOnly).Where(Function(x) Not x.Contains("Report")).ToList)
Dts.Variables.Item("FilesList").Value = lstFiles
Dts.TaskResult = ScriptResults.Success
End Sub
In the For each Loop Container Choose the Enumertaion Type as 'From variable Enumerator' and choose FilesList variable as a source
ScreenShots
Second
Inside the for each loop add an Expression Task to check if the file contains Report string or not
Add a variable of type System.Boolean (Name: ExcludeFile)
Inside the ForEach Loop Container add an Expression Task component before the DataFlowTask you that imports the Excel File
Inside The Expression Task write the following:
#[User::ExcludeFile] = (FINDSTRING(#[User::XlsxFile], "Report", 1 ) == 0)
Double Click on the connector between the expression task and the DataFlowTask and write the following expression
#[User::ExcludeFile] == False
Note: It is not necessary to use an Expression Task to validate this you can use a Dummy DataFlowTask or a Script Task to check if the filename contains the Keyword you want to exclude or not
In Script Component [Input0_ProcessInputRow], i am trying to fetch "ReadWrite" global variable value and it throws below error.
ERROR:
The collection of variables locked for read and write access is not available outside of PostExecute.
Below is my code
If Row.Column13 = "C" Then
Variables.mTotalCreditCount = Variables.mTotalCreditCount - 1
Variables.mTotalCreditAmount = Variables.mTotalCreditAmount - CDbl(Row.Column14)
ElseIf Row.Column13 = "D" Then
Variables.mTotalDebitCount = Variables.mTotalDebitCount - 1
Variables.mTotalDebitAmount = Variables.mTotalDebitAmount - CDbl(Row.Column14)
End If
I also tried to read the value in local variable and then assign to global variable in the PostExecute() as below. No Luck
If Row.Column13 = "C" Then
mTotalCrCnt = Variables.mTotalCreditCount - 1
mTotalCrAmt = Variables.mTotalCreditAmount - CDbl(Row.Column14)
ElseIf Row.Column13 = "D" Then
mTotalDbCnt = Variables.mTotalDebitCount
mTotalDbCnt = mTotalDbCnt - 1
mTotalDbAmt = Variables.mTotalDebitAmount
mTotalDbAmt = mTotalDbAmt - CDbl(Row.Column14)
End If
Public Overrides Sub PostExecute()
MyBase.PostExecute()
Variables.ProcessCount = intProcessCount
Variables.mTotalCreditCount = mTotalCrCnt
Variables.mTotalCreditAmount = mTotalCrAmt
Variables.mTotalDebitCount = mTotalDbCnt
Variables.mTotalDebitAmount = mTotalDbAmt
End Sub
Any assistance please?
Looking to your comment it looks that you have solved the issue, but i am posting this answer to give information about how working with variables in a SSIS script and how to solve a similar issue, so it can helps other users
SSIS Variables
Variables Stores values that can be used in all SSIS components and containers.
Integration Services supports two types of variables: user-defined variables and system variables. User-defined variables are defined by package developers, and system variables are defined by Integration Services. You can create as many user-defined variables as a package requires, but you cannot create additional system variables. More info in this MSDN article
Using Variables in Script components
Every script has a ReadOnlyVariables and ReadWriteVariables Lists that can be defined on the Script page.
ReadOnlyVariables
ReadOnlyVariables can be accessed from all script Sub's and they are Read-Only as they are named.
ReadWriteVariables
The collection of ReadWriteVariables is only available in the PostExecute method to maximize performance and minimize the risk of locking conflicts. Therefore you cannot directly increment the value of a package variable as you process each row of data. Increment the value of a local variable instead, and set the value of the package variable to the value of the local variable in the PostExecute method after all data has been processed. You can also use the VariableDispenser property to work around this limitation. However, writing directly to a package variable as each row is processed will negatively impact performance and increase the risk of locking conflicts. More in this MSDN article
Methods to Work with Variables
There are 3 Methods to work with variables:
Accessing them directly after having selected them as ReadOnlyVariables or ReadWriteVariables in the script page
Using a Variables Dispenser (LockForRead and LockForWrite methods)
IDTSVariables100 vars = null;
VariableDispenser.LockForRead("User::MyVariable");
VariableDispenser.GetVariables(out vars);
string TaskName = vars["User::MyVariable"].Value.ToString();
vars.Unlock();
Using SSIS Logging Task: to read variable and log them to Execution Log, Message Box or File
There are many articles Talking about this methods and you can refer to them to learn more
VariableDispenser.GetVariables Method (Variables)
VariableDispenser.LockForWrite Method (String)
3 Ways -SSIS Read Write Variables – Script Task C# / VB.net
Read and Write variables in a Script Component in SSIS (SQL Server Integration Services) using C#
Use SSIS Variables and Parameters in a Script Task
I've done a few searches here, and while some issues are similar, they don't seem to be exactly what I need.
What I'm trying to do is import an Excel file into a SQL table via SSIS, but the problem is that I will never know the exact filename. We get files at no steady interval, and the file usually has a date/month in the name. For instance, our current file is "Census Data - May 2013.xls". We will only ever load ONE file at a time, so I don't need to loop through a directory for multiple Excel files.
My concept is that I can take this file, copy it to a "Loading" directory, and load it from there. At the start of the package, I will first clear out the loading directory, then scan the original directory for an Excel file, copy it to the loading directory and then load it into SQL. I suppose I may have to store the file names somewhere so I don't copy the same file into the loading directory in subsequent months, but I'm not really sure of the best way to handle that.
I've pretty much got everything down except the part that scans the directory for the Excel file and copies it to the loading directory. I've taken the majority of my info from this page, which (again) is close to what I want to do but not quite exactly the solution I need.
Can anyone get me over the finish line? I can't seem to get the Excel Connection Manager right (this is my first time using variables), and I can't figure out how to get the file into the Loading directory.
Problem statement
How do I dynamically identify a file name?
You will require some mechanism to inspect the contents of a folder and see what exists. Specifically, you are looking for an Excel file in your "Loading" directory. You know the file extension and that is it.
Resolution A
Use a ForEach File Enumerator.
Configure the Enumerator with an Expression on FileSpec of *.xls or *.xlsx depending on which flavor of Excel you're dealing with.
Add another Expression on Directory to be your Loading directory.
I typically create SSIS Variables named FolderInput and FileMask and assign those in the Enumerator.
Now when you run your package, the Enumerator is going to look in Diretory and find all the files that match the FileSpec.
Something needs to be done with what is found. You need to use that file name that the Enumerator returns. That's done through the Variable Mappings tab. I created a third Variable called CurrentFileName and assign it the results of the enumerator.
If you put a Script Task inside the ForEach Enumerator, you should be able to see that the value in the "Locals" window for #[User::CurrentFileName] has updated from the Design time value of whatever to the "real" file name.
Resolution B
Use a Script Task.
You will still need to create a Variable to hold the current file name and it probably won't hurt to also have the FolderInput and FileMask Variables available. Set the former as ReadWrite and the latter as ReadOnly variables.
Chose the .NET language of your choice. I'm using C#. The method System.IO.Directory.EnumerateFiles
using System;
using System.Data;
using System.IO;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
namespace ST_fe2ea536a97842b1a760b271f190721e
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
string folderInput = Dts.Variables["User::FolderInput"].Value.ToString();
string fileMask = Dts.Variables["User::FileMask"].Value.ToString();
try
{
var files = Directory.EnumerateFiles(folderInput, fileMask, SearchOption.AllDirectories);
foreach (string currentFile in files)
{
Dts.Variables["User::CurrentFileName"].Value = currentFile;
break;
}
}
catch (Exception e)
{
Dts.Events.FireError(0, "Script overkill", e.ToString(), string.Empty, 0);
}
Dts.TaskResult = (int)ScriptResults.Success;
}
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
}
}
Decision tree
Given the two resolutions to the above problem, how do you chose? Normally, people say "It Depends" but there only possible time it would depend is if the process should stop/error out in the case that more than one file did exist in the Loading folder. That's a case that the ForEach enumerator would be more cumbersome than a script task. Otherwise, as I stated in my original response that adds cost to your project for Development, Testing and Maintenance for no appreciable gain.
Bits and bobs
Further addressing nuances in the question: Configuring Excel - you'll need to be more specific in what isn't working. Both Siva's SO answer and the linked blogspot article show how to use the value of the Variable I call CurrentFileName to ensure the Excel File is pointing to the "right" file.
You will need to set the DelayValidation to True for both the Connection Manager and the Data Flow as the design-time value for the Variable will not be valid when the package begins execution. See this answer for a longer explanation but again, Siva called that out in their SO answer.
I've got an interesting challenge with SSIS. Using a for-each file enumerator, I need to pick the subfolder which has been most recently created, and then iterate through each of the files.
Perhaps an example would explain better. The folders look something like this:
c:\data\2011-0703
c:\data\2011-0626
c:\data\2011-0619
How could you get a for each file enumerator to pick the most recent folder? This could either be by looking at the creation date, or comparing the file names.
I'm guessing it would be done with an expression in the enumerator, just can't work out how! Couldn't find anything on the net either.
Thanks
Here is one possible option that you can achieve this with the help of Script Task. Following example shows how this can be done. The example was created in SSIS 2008 R2.
Step-by-step process:
Create three folders named 2011-0619, 2011-0626 and 2011-0703 in the folder path C:\temp\ as shown in screenshot #1. Make note of the Date created value of each of the folders.
Place few files in each of the folders as shown in screenshots #2 - #4.
On the SSIS package, create four variables as shown in screenshot #5. Set the variable RootFolder with value C:\temp\ (in your case this will be c:\data). Set the variable FilePattern with value *.*. Variable RecentFolder will be assigned with the recent folder path in the Script Task. To avoid design time errors, assign the variable RecentFolder with a valid file path. Variable FilePath will be assigned with values when the files are looped through in the recent folder.
On the SSIS package, place a Script Task. Replace the Main() method within the Script Task with the script task code give under section Script task code (Get recent folder):. This script gets the list of folders in the root folder and loops through to check the creation datetime to pick the most recently created folder. The recently created folder path is then stored in the variable RecentFolder.
On the SSIS package, place a Foreach Loop container and configure it as shown in screenshots #6 and #7.
Place a Script Task inside the Foreach Loop container. Replace the Main() method within the Script Task with the script task code give under section Script task code (Display file names):. This script simply displays the names of files within the recently created folder.
Once all tasks are configured, the package should look like as shown in screenshot #8.
Screenshots #9 - #11 show that the package displays the file names in the recently created folder 2011-0703.
Hope that helps.
Script task code (Get recent folder):
C# code that can be used only in SSIS 2008 and above.
public void Main()
{
Variables varCollection = null;
Dts.VariableDispenser.LockForRead("User::RootFolder");
Dts.VariableDispenser.LockForWrite("User::RecentFolder");
Dts.VariableDispenser.GetVariables(ref varCollection);
string rootFolder = varCollection["User::RootFolder"].Value.ToString();
DateTime previousFolderTime = DateTime.MinValue;
string recentFolder = string.Empty;
foreach (string subFolder in System.IO.Directory.GetDirectories(rootFolder))
{
DateTime currentFolderTime = System.IO.Directory.GetCreationTime(subFolder);
if (previousFolderTime == DateTime.MinValue || previousFolderTime <= currentFolderTime)
{
previousFolderTime = currentFolderTime;
recentFolder = subFolder;
}
}
varCollection["User::RecentFolder"].Value = recentFolder;
Dts.TaskResult = (int)ScriptResults.Success;
}
Script task code (Display file names):
C# code that can be used only in SSIS 2008 and above.
public void Main()
{
Variables varCollection = null;
Dts.VariableDispenser.LockForRead("User::FilePath");
Dts.VariableDispenser.GetVariables(ref varCollection);
MessageBox.Show(varCollection["User::FilePath"].Value.ToString(), "File Path");
Dts.TaskResult = (int)ScriptResults.Success;
}
Screenshot #1:
Screenshot #2:
Screenshot #3:
Screenshot #4:
Screenshot #5:
Screenshot #6:
Screenshot #7:
Screenshot #8:
Screenshot #9:
Screenshot #10:
Screenshot #11:
Iterate through the folders. Save the name of the first one. Compare that saved value to the name of each subsequent folder. If the next folder is more recent, swap that name in and keep going. At the end, your saved value will be the name of the most recent folder (if you're comparing creation dates, you'll need to save both the folder name and the creation date).
You can then use the saved value as an argument to your second iteration loop.