I have created a Snowflake Java UDF function snowflake_email_validation and trying to call that function from the snowpipe.
It is throwing the error as:
User-defined function 'SNOWFLAKE_EMAIL_VALIDATION' with language 'JAVA' is not allowed in pipe definition" .
This is my command
create or replace pipe emailpipe auto_ingest=true as
copy into TGT_EMAIL_TABLE(EMAIL, IS_VALID)
from (select $1,snowflake_email_validation($1) from #s3_stage)
on_error=continue;
Please help to fix this
COPY INTO has several limitations like this, and Snowpipe can work only with COPY INTO.
Try as an alternative the different query below. It can be eventually encapsulated in a stored procedure, and called periodically on a schedule by a task.
create or replace table TGT_EMAIL_TABLE(EMAIL, IS_VALID) as
select $1, snowflake_email_validation($1)
from #s3_stage
Related
I utilized the Snowpark Python (private preview) to do some data engineering tasks (transform the data from a raw state to a clean state). How do I upload the Python code to Snowflake to then run it?
I think it needs to be a stored procedure but I can not find any documentation on how to create a stored procedure in Python.
Snowpark Stored Procedures for Python — Preview was relased in June 2022.
Documentation is avaiable at Writing Stored Procedures in Snowpark (Python)
Example:
In an in-line stored procedure, you write your Python code in the AS clause of the CREATE PROCEDURE statement. For example:
CREATE OR REPLACE PROCEDURE MYPROC(from_table STRING, to_table STRING, count INT)
RETURNS STRING
LANGUAGE PYTHON
RUNTIME_VERSION = '3.8'
PACKAGES = ('snowflake-snowpark-python')
HANDLER = 'run'
AS
$$
def run(session, from_table, to_table, count):
session.table(from_table).limit(count).write.save_as_table(to_table)
return "SUCCESS"
$$;
Adding to Lukasz Szozda's answer, we could write the python code in a file and upload it to a Snowflake stage. Then write a procedure to invoke the code from the stage. Attaching link for reference: Code uploaded from stage
I am trying to call Liquibase Stored Procedure Through Tag
it gives me error as sql code
DB2 SQL Error: SQLCODE=-440, SQLSTATE=42884
I Am Tring to call as follows : :
<sql> CALL TestProcedure('abc','xyz') </sql>
it executed well from outside command line Client and not using liquibase
i also tried calling using schema name no luck Open to suggestions
SQL440N is routine not found. The error message should give the text of the rountine name. Does it match what you have in your database. Is is a case sensitive problem? (i.e. try in upper case?)
SQL0440N No authorized routine named "<routine-name>" of type
"<routine-type>" having compatible arguments was found.
I am trying to build a scenario that would allow me to expose additional data from a server (for case-demo purposes). The server calls a vulnerable SQL code:
EXEC my_storeProc '12345'
where "12345" (not the single quotes) is the parameter. This performs a SELECT statement. I would like to eliminate this execution and instead call my own select statement, however the server side code will only accept the first select statement called, contained within the aforementioned EXEC call. Calling the second statement is easy:
EXEC my_storeProc '12345 ' select * from MySecondTable--
(the -- at the end will block the closing single quote added by the server to prevent errors). My problem is that although there are 2 select statements, the server will only parse the first one. Is there a way to cancel the first EXEC call without throwing an error so that the second one would be taken instead? Perhaps even a UNION but there isn't much I can do with only one variable open to exploit (variable being 12345 in this case).
You have to think of how it will be executed, specifically you want it called so it doesn't raise an exception and put the kabosh on the whole statement. You can't set the result to always true with a proc call, so there is no real way escape the proc. Instead, you'll want to slip a second command in, Your desired code looks like;
exec my_Storeproc '1234'; select * from mysecondtable
So we need to close the quotes, and make a new statement. That would mean the string with the insert needs to be;
1234'; select * from mysecondtable where 1 = '1
There is a flaw in this, whatever command you are executing is not being returned to the UI. To get the data you'll have to add a server connection to the second command.
To make the second command unnecessary you would have to inject code into the proc, which is a non starter since the proc is already complied and sql injection relies on confusing the compiler as to what is data and what is commands. For a more verbose explanation of that check out this answer:
https://security.stackexchange.com/a/25710
I have a batch file which has many steps in it that will get executed one by one.
However, to be able to make it more flexible, I want to include a condition check in SQLPlus.
Something like, to get the conditional variables' value from a query first and store is in say v_variable. Then use it for some checks like
IF v_variable = 'Y' THEN
--DO SOME DDL
ELSE
--DO OTHER DDL
END IF
I have to repeat this block in many places in the batch file and I can't do it through a PL/SQL somehow.
I am trying to use this COLUMN command in SQLPlus but somehow not getting the variable value to get saved.
COLUMN VARIABLE1 NEW_VALUE V_VARIABLE1
SELECT PARAM_VAL AS VARIABLE1 FROM TABLE_T WHERE PARAM_TYPE = 'XYZ';
-- This query will only throw one record.
DEFINE V_VARIABLE1
Is that absolutely wrong? What do we do to see if the V_VARIABLE1 is getting the value coming from the query?
And even after I get it right I am clueless on the IF-ELSE part. Can anybody help here? I am interested in solution that works for SQLPlus.
sql*plus doesn't support flow control natively, so anything you can do here would fall somewhere between "workaround" and "hack". Some of possible options are:
Use PL/SQL block and run your DDL as dynamic SQL via execute
immediate or dbms_utility.execute_ddl_statement. Full access to
PL/SQL features, so most flexible in terms of flow control and
statement building, however harder to manage if you're deploying
something large.
Write a script file per if/else branch, get its name with something
like column/query trick you provided in your post, run it with
something like #&scriptname.
Use substitution variables that, when used properly, will comment
out some parts of your script. You can use Tanel Poder's snapper
utility script as example; notice &_IF_ substitution
variables there.
You can embed child script into parent script's pl/sql block. Like
this:
--
21:21:23 SQL> ho cat run.sql
begin
case '&1.' when 'A' then
#script_a
when 'B' then
#script_b
else
null;
end case;
end;
/
21:21:26 SQL> ho cat script_a.sql
dbms_output.put_line('this is a');
21:21:30 SQL> ho cat script_b.sql
dbms_output.put_line('this is b');
21:21:34 SQL> #run
Enter value for 1: A
this is a
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.02
21:21:37 SQL> #run B
this is b
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.03
I don't have any experience with SQLPlus. I am going to assume that you can redirect the output of your select command to a temp file. Assuming you only selected one column in 1 record you can then do something like this:
FOR /F "tokens=*" %%A IN ('FINDSTR /v "(" YourTempFile.txt) DO SET YourVariable=%%A
"tokens=*" is optional. It will remove leading spaces and save the rest of the string including spaces. May not be relevant depending on your data.
You may not need the 'FINDSTR /V "(" and the trailing ' either. I am assuming the output of SQLPlus will be similar to SQL and this would exclude the number of rows processed that gets reported by SQL. Anything is possible. If you can't make this work, post the contents of the temp file and we can make the necessary modifications.
I'm tring to create an SSIS package to import some dataset files, however given that I seem to be hitting a brick
wall everytime I achieve a small part of the task I need to take a step back and perform a sanity check on what I'm
trying to achieve, and if you good people can advise whether SSIS is the way to go about this then I would
appreciate it.
These are my questions from this morning :-
debugging SSIS packages - debug.writeline
Changing an SSIS dts variables
What I'm trying to do is have a For..Each container enumerate over the files in a share on the SQL Server. For each
file it finds a script task runs to check various attributes of the filename, such as looking for a three letter
code, a date in CCYYMM, the name of the data contained therein, and optionally some comments. For example:-
ABC_201007_SalesData_[optional comment goes here].csv
I'm looking to parse the name using a regular expression and put the values of 'ABC', '201007', and
'SalesData' in variables.
I then want to move the file to an error folder if it doesn't meet certain criteria :-
Three character code
Six character date
Dataset name (e.g. SalesData, in this example)
CSV extension
I then want to lookup the Character code, the date (or part thereof), and the Dataset name against a lookup table
to mark off a 'checklist' of received files from each client.
Then, based on the entry in the checklist, I want to kick off another SSIS package.
So, for example I may have a table called 'Checklist' with these columns :-
Client code Dataset SSIS_Package
ABC SalesData NorthSalesData.dtsx
DEF SalesData SouthSalesData.dtsx
If anyone has a better way of achieving this I am interested in hearing about it.
Thanks in advance
That's an interesting scenario, and should be relatively easy to handle.
First, your choice of the Foreach Loop is a good one. You'll be using the Foreach File Enumerator. You can restrict the files you iterate over to be just CSVs so that you don't have to "filter" for those later.
The Foreach File Enumerator puts the filename (full path or just file name) into a variable - let's call that "FileName". There's (at least) two ways you can parse that - expressions or a Script Task. Depends which one you're more comfortable with. Either way, you'll need to create three variables to hold the "parts" of the filename - I'll call them "FileCode", "FileDate", and "FileDataset".
To do this with expressions, you need to set the EvaluateAsExpression property on FileCode, FileDate, and FileDataset to true. Then in the expressions, you need to use FINDSTRING and SUBSTRING to carve up FileName as you see fit. Expressions don't have Regex capability.
To do this in a Script Task, pass the FileName variable in as a ReadOnly variable, and the other three as ReadWrite. You can use the Regex capabilities of .Net, or just manually use IndexOf and Substring to get what you need.
Unfortunately, you have just missed the SQLLunch livemeeting on the ForEach loop: http://www.bidn.com/blogs/BradSchacht/ssis/812/sql-lunch-tomorrow
They are recording the session, however.