In Script Component [Input0_ProcessInputRow], i am trying to fetch "ReadWrite" global variable value and it throws below error.
ERROR:
The collection of variables locked for read and write access is not available outside of PostExecute.
Below is my code
If Row.Column13 = "C" Then
Variables.mTotalCreditCount = Variables.mTotalCreditCount - 1
Variables.mTotalCreditAmount = Variables.mTotalCreditAmount - CDbl(Row.Column14)
ElseIf Row.Column13 = "D" Then
Variables.mTotalDebitCount = Variables.mTotalDebitCount - 1
Variables.mTotalDebitAmount = Variables.mTotalDebitAmount - CDbl(Row.Column14)
End If
I also tried to read the value in local variable and then assign to global variable in the PostExecute() as below. No Luck
If Row.Column13 = "C" Then
mTotalCrCnt = Variables.mTotalCreditCount - 1
mTotalCrAmt = Variables.mTotalCreditAmount - CDbl(Row.Column14)
ElseIf Row.Column13 = "D" Then
mTotalDbCnt = Variables.mTotalDebitCount
mTotalDbCnt = mTotalDbCnt - 1
mTotalDbAmt = Variables.mTotalDebitAmount
mTotalDbAmt = mTotalDbAmt - CDbl(Row.Column14)
End If
Public Overrides Sub PostExecute()
MyBase.PostExecute()
Variables.ProcessCount = intProcessCount
Variables.mTotalCreditCount = mTotalCrCnt
Variables.mTotalCreditAmount = mTotalCrAmt
Variables.mTotalDebitCount = mTotalDbCnt
Variables.mTotalDebitAmount = mTotalDbAmt
End Sub
Any assistance please?
Looking to your comment it looks that you have solved the issue, but i am posting this answer to give information about how working with variables in a SSIS script and how to solve a similar issue, so it can helps other users
SSIS Variables
Variables Stores values that can be used in all SSIS components and containers.
Integration Services supports two types of variables: user-defined variables and system variables. User-defined variables are defined by package developers, and system variables are defined by Integration Services. You can create as many user-defined variables as a package requires, but you cannot create additional system variables. More info in this MSDN article
Using Variables in Script components
Every script has a ReadOnlyVariables and ReadWriteVariables Lists that can be defined on the Script page.
ReadOnlyVariables
ReadOnlyVariables can be accessed from all script Sub's and they are Read-Only as they are named.
ReadWriteVariables
The collection of ReadWriteVariables is only available in the PostExecute method to maximize performance and minimize the risk of locking conflicts. Therefore you cannot directly increment the value of a package variable as you process each row of data. Increment the value of a local variable instead, and set the value of the package variable to the value of the local variable in the PostExecute method after all data has been processed. You can also use the VariableDispenser property to work around this limitation. However, writing directly to a package variable as each row is processed will negatively impact performance and increase the risk of locking conflicts. More in this MSDN article
Methods to Work with Variables
There are 3 Methods to work with variables:
Accessing them directly after having selected them as ReadOnlyVariables or ReadWriteVariables in the script page
Using a Variables Dispenser (LockForRead and LockForWrite methods)
IDTSVariables100 vars = null;
VariableDispenser.LockForRead("User::MyVariable");
VariableDispenser.GetVariables(out vars);
string TaskName = vars["User::MyVariable"].Value.ToString();
vars.Unlock();
Using SSIS Logging Task: to read variable and log them to Execution Log, Message Box or File
There are many articles Talking about this methods and you can refer to them to learn more
VariableDispenser.GetVariables Method (Variables)
VariableDispenser.LockForWrite Method (String)
3 Ways -SSIS Read Write Variables – Script Task C# / VB.net
Read and Write variables in a Script Component in SSIS (SQL Server Integration Services) using C#
Use SSIS Variables and Parameters in a Script Task
Related
I have created one Snowflake storage integration as follow:
create or replace storage integration s3_int
type = external_stage
storage_provider = s3
enabled = TRUE
storage_aws_role_arn = 'arn:aws:iam::xxx'
storage_allowed_locations = ('S3://location1',
's3://location2',
's3://location4',
's3://location5',
's3://location6',
's3://location7',
's3://location8');
Now I like to add 2 more locations, I know that I can use SET storage_allowed_locations like following:
ALTER STORAGE INTEGRATION s3_int
SET storage_allowed_locations = ('S3://location1',
's3://location2',
's3://location4',
's3://location5',
's3://location6',
's3://location7',
's3://location8',
's3://location9',
's3://location10');
But the problem is, along the time I will have more and more locations to add in this storage_allowed_location list. Do I always have to give the whole list of locations and add the new locations in the end of it? It will become super long and annoying in the end. Do you know if there is other ways to add/append the new location in existing location list?
At this point, yes, the storage locations have to be added along with the existing ones.
I've done a few searches here, and while some issues are similar, they don't seem to be exactly what I need.
What I'm trying to do is import an Excel file into a SQL table via SSIS, but the problem is that I will never know the exact filename. We get files at no steady interval, and the file usually has a date/month in the name. For instance, our current file is "Census Data - May 2013.xls". We will only ever load ONE file at a time, so I don't need to loop through a directory for multiple Excel files.
My concept is that I can take this file, copy it to a "Loading" directory, and load it from there. At the start of the package, I will first clear out the loading directory, then scan the original directory for an Excel file, copy it to the loading directory and then load it into SQL. I suppose I may have to store the file names somewhere so I don't copy the same file into the loading directory in subsequent months, but I'm not really sure of the best way to handle that.
I've pretty much got everything down except the part that scans the directory for the Excel file and copies it to the loading directory. I've taken the majority of my info from this page, which (again) is close to what I want to do but not quite exactly the solution I need.
Can anyone get me over the finish line? I can't seem to get the Excel Connection Manager right (this is my first time using variables), and I can't figure out how to get the file into the Loading directory.
Problem statement
How do I dynamically identify a file name?
You will require some mechanism to inspect the contents of a folder and see what exists. Specifically, you are looking for an Excel file in your "Loading" directory. You know the file extension and that is it.
Resolution A
Use a ForEach File Enumerator.
Configure the Enumerator with an Expression on FileSpec of *.xls or *.xlsx depending on which flavor of Excel you're dealing with.
Add another Expression on Directory to be your Loading directory.
I typically create SSIS Variables named FolderInput and FileMask and assign those in the Enumerator.
Now when you run your package, the Enumerator is going to look in Diretory and find all the files that match the FileSpec.
Something needs to be done with what is found. You need to use that file name that the Enumerator returns. That's done through the Variable Mappings tab. I created a third Variable called CurrentFileName and assign it the results of the enumerator.
If you put a Script Task inside the ForEach Enumerator, you should be able to see that the value in the "Locals" window for #[User::CurrentFileName] has updated from the Design time value of whatever to the "real" file name.
Resolution B
Use a Script Task.
You will still need to create a Variable to hold the current file name and it probably won't hurt to also have the FolderInput and FileMask Variables available. Set the former as ReadWrite and the latter as ReadOnly variables.
Chose the .NET language of your choice. I'm using C#. The method System.IO.Directory.EnumerateFiles
using System;
using System.Data;
using System.IO;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
namespace ST_fe2ea536a97842b1a760b271f190721e
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
string folderInput = Dts.Variables["User::FolderInput"].Value.ToString();
string fileMask = Dts.Variables["User::FileMask"].Value.ToString();
try
{
var files = Directory.EnumerateFiles(folderInput, fileMask, SearchOption.AllDirectories);
foreach (string currentFile in files)
{
Dts.Variables["User::CurrentFileName"].Value = currentFile;
break;
}
}
catch (Exception e)
{
Dts.Events.FireError(0, "Script overkill", e.ToString(), string.Empty, 0);
}
Dts.TaskResult = (int)ScriptResults.Success;
}
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
}
}
Decision tree
Given the two resolutions to the above problem, how do you chose? Normally, people say "It Depends" but there only possible time it would depend is if the process should stop/error out in the case that more than one file did exist in the Loading folder. That's a case that the ForEach enumerator would be more cumbersome than a script task. Otherwise, as I stated in my original response that adds cost to your project for Development, Testing and Maintenance for no appreciable gain.
Bits and bobs
Further addressing nuances in the question: Configuring Excel - you'll need to be more specific in what isn't working. Both Siva's SO answer and the linked blogspot article show how to use the value of the Variable I call CurrentFileName to ensure the Excel File is pointing to the "right" file.
You will need to set the DelayValidation to True for both the Connection Manager and the Data Flow as the design-time value for the Variable will not be valid when the package begins execution. See this answer for a longer explanation but again, Siva called that out in their SO answer.
I am banging my head into a wall over this and hoping you can tell me the very simple thing I have overlooked in my sleep deprived/noob state.
Very simply I am doing a query and the type of object returned is different on my local machine than what gets returned once I deploy the application.
match = MatchRealTimeStatsModel.queryMatch(ancestor_key)[0]
On my local machine the above produces a MatchRealTimeStatsModel object. So I can run the following to lines without a problem:
logging.info(match) # outputs a MatchRealTimeStatsModel object
logging.info(match.match) # outputs a dictionary from json data
When the above two lines are run on Goggles machines I get the following though:
logging.info(match) # outputs a dictionary from json data
logging.info(match.match) # AttributeError: 'dict' object has no attribute 'match'
Any suggestions as to what might be causing this? I cleared the data store and did everything I could think of to clean the GAE environment.
Edit #1: Adding MatchRealTimeStatsModel code:
class MatchRealTimeStatsModel(ndb.Model):
match = ndb.JsonProperty()
#classmethod
def queryMatch(cls, ancestor_key):
return cls.query(ancestor=ancestor_key).fetch()
And here is the actual call:
ancestor_key = ndb.Key('MatchRealTimeStatsModel', matchUniqueUrl)
match = MatchRealTimeStatsModel.queryMatch(ancestor_key)[0]
Perhaps you are using different versions of your code locally than in prod? Try to reset your copy of the source code in both places.
Currently, I have a file path value hard coded inside the Script task of an SSIS package.
I have a string variable sPath. How should I use this variable sPath inside Script Task?
string strPath = "C:\\File.xls";
if( File.Exists(strPath))
{
File.Delete(strPath);
}
Here is one possible way of using variables inside Script Task. Assuming that you have a variable named FilePath declared on your package as shown in screenshot #1 then you can use the following code to use the variable inside the Script Task. this is one of the possible ways to use the variable. Here the variable is used only to read the value using the method LockForRead. You can also write values to the variable if the variable is declared with LockForWrite method.
By the way, the functionality described in the Scrip Task code can also be carried out using the File System Task available in SSIS Control Flow task list.
Hope that helps.
Using package variables inside Script Task:
C# code that can be used only in SSIS 2008 and above.
.
public void Main()
{
Variables varCollection = null;
string FilePath = string.Empty;
Dts.VariableDispenser.LockForRead("User::FilePath");
Dts.VariableDispenser.GetVariables(ref varCollection);
FilePath = varCollection["User::FilePath"].Value.ToString();
if (File.Exists(FilePath))
{
File.Delete(FilePath);
}
varCollection.Unlock();
Dts.TaskResult = (int)ScriptResults.Success;
}
Screenshot #1:
I have the following lua script :
module("modoo",package.seeall)
foo=1
bar={12,34}
Which works fine using the cli, such as :
> dofile "mod_modoo.lua"
> =modoo.foo
1
> =modoo
table: 0x86ce058
As far as I understood, it works like a table, but whenever I try loading it as a table, a nil value is pushed onto the stack. Every other table works normally.
I thought using lua_getglobal wouldn't work with modules, but I couldn't find the proper way to load it either ; how should I do it ?
Load Lua modules with require like lua.c does. See http://www.lua.org/source/5.1/lua.c.html#dolibrary
Instead of using module and dofile, it is better practice in Lua 5.1 to simply return a table representing your module when the script is run. All functions and variables should be declared as local within the module script so that the returned table provides the only access point and no globals from other modules are trampled. The module should be imported using require as follows.
mod_modoo.lua:
return { foo = 1, bar = { 12, 34 } }
modoo_test.lua:
> local modoo = require "mod_modoo"
However, it is often nice to bring the module table (i.e. modoo) in as a global without explicitly assigning it. To do so, assign the module table a global reference in addition to returning it from the script. In that case, the module can be used as follows:
mod_modoo.lua:
modoo = { foo = 1, bar = { 12, 34 } }
return modoo
modoo_test.lua:
> require "mod_modoo"
> print(modoo.foo)
1
See the Lua Module Function Critiqued article for more information on the matter.