I'm using schemachange (https://github.com/Snowflake-Labs/schemachange) to manage creating resources in snowflake. I have an initial script that sets up, among other things, a stage. In schemachange, you can use variables with {{ ENV }} (as example). In the case below, I want to provide a url that uses {{ ENV }} as art of the string. i.e.
create stage if not exists test_{{ ENV }}
file_format = (type=json)
copy_options = (match_by_column_name=case_insensitive)
url='s3://test-bucket-{{ ENV }}-xxxxxxx/'
storage_integration = s3_int_{{ DBVERSION }};
The URL line is where I want to reference a bucket name specific to the ENV variable that's being passed in. I'm not sure how to do it in this case, tried just using the above but it doesn't work. Any suggestions? Thanks!
Here's how I did it:
CREATE STAGE IF NOT EXISTS Test
URL = 's3://test-bucket-{{env}}'
In your schemachange command, you need to make sure you include -- vars command:
--vars '{"env":"MyEnvironment"}'
We have this running thru Azure DevOps pipelines and works great so far.
Related
I am trying to place my snowflakes into git repository. Some of the part are environment specific, like
use schema analytics_dev ### I will have to change this while deploying to qa and stuff.
I was thinking if could replace these with variable like this
set env='dev'
use schema analytics+$env
But this an error "SQL compilation error: syntax error line 1 at position 20 unexpected '+'."
How do I achieve this functionality?
Here's a way, to append vars, and use as an identifier:
set env = 'dev';
set analytics_schema = 'analytics' || $env;
use schema identifier($analytics_schema);
A couple of options here:
set (min, max)=(40, 70);
select $min;
And this:
set var_artist_name ='Jackson Browne';
select getvariable('var_artist_name');
and more here:
https://docs.snowflake.com/en/sql-reference/session-variables.html
I try to create configuration file for my go app what loops through some jobs.
My .yaml file looks like that (array):
jobToRun:
- name: This is my first job to run
sqlToRun: select 1 from some_table
someVariable: 1
- name: Other job to run
sqlToRun: select 2 from some_table
someVariable: 2
I have successfully imported the YAML file and created also structure.
type Service struct {
JobToRun []struct {
Name string `yaml:"name"`
SQLToRun string `yaml:"SqlToRun"`
SomeVariable int `yaml:"someVariable"`
} `yaml:"jobToRun"`
}
But I have no idea how to assign them to variable.
I tried some stuff what work with Json array-s but without any luck.
So I tried to print it to console without any luck:
println(service.JobToRun.name[0])
before that I tried to assign that SQL to my variable (which works if it is not an array item.
var sqlQuery = service.JobToRun.name[0]
And here is what I try to accomplish:
I take the Job parameters from .yaml array and run it.
I am using that kind of array in YAML because it is easiest way to add new jobs.
ah as soon as i posted it i facepalmed.
println(service.JobToRun[0].Name)
so the reason was that "JobToRun is an array not "Name"
In Script Component [Input0_ProcessInputRow], i am trying to fetch "ReadWrite" global variable value and it throws below error.
ERROR:
The collection of variables locked for read and write access is not available outside of PostExecute.
Below is my code
If Row.Column13 = "C" Then
Variables.mTotalCreditCount = Variables.mTotalCreditCount - 1
Variables.mTotalCreditAmount = Variables.mTotalCreditAmount - CDbl(Row.Column14)
ElseIf Row.Column13 = "D" Then
Variables.mTotalDebitCount = Variables.mTotalDebitCount - 1
Variables.mTotalDebitAmount = Variables.mTotalDebitAmount - CDbl(Row.Column14)
End If
I also tried to read the value in local variable and then assign to global variable in the PostExecute() as below. No Luck
If Row.Column13 = "C" Then
mTotalCrCnt = Variables.mTotalCreditCount - 1
mTotalCrAmt = Variables.mTotalCreditAmount - CDbl(Row.Column14)
ElseIf Row.Column13 = "D" Then
mTotalDbCnt = Variables.mTotalDebitCount
mTotalDbCnt = mTotalDbCnt - 1
mTotalDbAmt = Variables.mTotalDebitAmount
mTotalDbAmt = mTotalDbAmt - CDbl(Row.Column14)
End If
Public Overrides Sub PostExecute()
MyBase.PostExecute()
Variables.ProcessCount = intProcessCount
Variables.mTotalCreditCount = mTotalCrCnt
Variables.mTotalCreditAmount = mTotalCrAmt
Variables.mTotalDebitCount = mTotalDbCnt
Variables.mTotalDebitAmount = mTotalDbAmt
End Sub
Any assistance please?
Looking to your comment it looks that you have solved the issue, but i am posting this answer to give information about how working with variables in a SSIS script and how to solve a similar issue, so it can helps other users
SSIS Variables
Variables Stores values that can be used in all SSIS components and containers.
Integration Services supports two types of variables: user-defined variables and system variables. User-defined variables are defined by package developers, and system variables are defined by Integration Services. You can create as many user-defined variables as a package requires, but you cannot create additional system variables. More info in this MSDN article
Using Variables in Script components
Every script has a ReadOnlyVariables and ReadWriteVariables Lists that can be defined on the Script page.
ReadOnlyVariables
ReadOnlyVariables can be accessed from all script Sub's and they are Read-Only as they are named.
ReadWriteVariables
The collection of ReadWriteVariables is only available in the PostExecute method to maximize performance and minimize the risk of locking conflicts. Therefore you cannot directly increment the value of a package variable as you process each row of data. Increment the value of a local variable instead, and set the value of the package variable to the value of the local variable in the PostExecute method after all data has been processed. You can also use the VariableDispenser property to work around this limitation. However, writing directly to a package variable as each row is processed will negatively impact performance and increase the risk of locking conflicts. More in this MSDN article
Methods to Work with Variables
There are 3 Methods to work with variables:
Accessing them directly after having selected them as ReadOnlyVariables or ReadWriteVariables in the script page
Using a Variables Dispenser (LockForRead and LockForWrite methods)
IDTSVariables100 vars = null;
VariableDispenser.LockForRead("User::MyVariable");
VariableDispenser.GetVariables(out vars);
string TaskName = vars["User::MyVariable"].Value.ToString();
vars.Unlock();
Using SSIS Logging Task: to read variable and log them to Execution Log, Message Box or File
There are many articles Talking about this methods and you can refer to them to learn more
VariableDispenser.GetVariables Method (Variables)
VariableDispenser.LockForWrite Method (String)
3 Ways -SSIS Read Write Variables – Script Task C# / VB.net
Read and Write variables in a Script Component in SSIS (SQL Server Integration Services) using C#
Use SSIS Variables and Parameters in a Script Task
I want to create a copy of a database to a folder not under the data either locally or on a server I have code that looks like this:
var arcName:String = "C:\Archive\MyArchives\SomeName.nsf"
var arcDB:NotesDatabase = appDB.createCopy("", arcName);
When the action finishes (it does not generate any errors) I can't find the database anywhere. if I change the arcName to "Archives\Myarchives\SomeName,nsf" the process works correctly. But I don't want these Archives under Data.
Using the full path does not seem to make it move out from under the Data folder.
This may be a case of string escaping - in SSJS, like most C-lineage languages, \ is the escape character. Give it a shot with \\ in place of each. In my testing, it works as database.createCopy("", "C:\\Archive\\MyArchives\\SomeName.nsf").
I am banging my head into a wall over this and hoping you can tell me the very simple thing I have overlooked in my sleep deprived/noob state.
Very simply I am doing a query and the type of object returned is different on my local machine than what gets returned once I deploy the application.
match = MatchRealTimeStatsModel.queryMatch(ancestor_key)[0]
On my local machine the above produces a MatchRealTimeStatsModel object. So I can run the following to lines without a problem:
logging.info(match) # outputs a MatchRealTimeStatsModel object
logging.info(match.match) # outputs a dictionary from json data
When the above two lines are run on Goggles machines I get the following though:
logging.info(match) # outputs a dictionary from json data
logging.info(match.match) # AttributeError: 'dict' object has no attribute 'match'
Any suggestions as to what might be causing this? I cleared the data store and did everything I could think of to clean the GAE environment.
Edit #1: Adding MatchRealTimeStatsModel code:
class MatchRealTimeStatsModel(ndb.Model):
match = ndb.JsonProperty()
#classmethod
def queryMatch(cls, ancestor_key):
return cls.query(ancestor=ancestor_key).fetch()
And here is the actual call:
ancestor_key = ndb.Key('MatchRealTimeStatsModel', matchUniqueUrl)
match = MatchRealTimeStatsModel.queryMatch(ancestor_key)[0]
Perhaps you are using different versions of your code locally than in prod? Try to reset your copy of the source code in both places.