I utilized the Snowpark Python (private preview) to do some data engineering tasks (transform the data from a raw state to a clean state). How do I upload the Python code to Snowflake to then run it?
I think it needs to be a stored procedure but I can not find any documentation on how to create a stored procedure in Python.
Snowpark Stored Procedures for Python — Preview was relased in June 2022.
Documentation is avaiable at Writing Stored Procedures in Snowpark (Python)
Example:
In an in-line stored procedure, you write your Python code in the AS clause of the CREATE PROCEDURE statement. For example:
CREATE OR REPLACE PROCEDURE MYPROC(from_table STRING, to_table STRING, count INT)
RETURNS STRING
LANGUAGE PYTHON
RUNTIME_VERSION = '3.8'
PACKAGES = ('snowflake-snowpark-python')
HANDLER = 'run'
AS
$$
def run(session, from_table, to_table, count):
session.table(from_table).limit(count).write.save_as_table(to_table)
return "SUCCESS"
$$;
Adding to Lukasz Szozda's answer, we could write the python code in a file and upload it to a Snowflake stage. Then write a procedure to invoke the code from the stage. Attaching link for reference: Code uploaded from stage
Related
I have created a Snowflake Java UDF function snowflake_email_validation and trying to call that function from the snowpipe.
It is throwing the error as:
User-defined function 'SNOWFLAKE_EMAIL_VALIDATION' with language 'JAVA' is not allowed in pipe definition" .
This is my command
create or replace pipe emailpipe auto_ingest=true as
copy into TGT_EMAIL_TABLE(EMAIL, IS_VALID)
from (select $1,snowflake_email_validation($1) from #s3_stage)
on_error=continue;
Please help to fix this
COPY INTO has several limitations like this, and Snowpipe can work only with COPY INTO.
Try as an alternative the different query below. It can be eventually encapsulated in a stored procedure, and called periodically on a schedule by a task.
create or replace table TGT_EMAIL_TABLE(EMAIL, IS_VALID) as
select $1, snowflake_email_validation($1)
from #s3_stage
I'm dealing with an Oracle DB, connecting from go via InstantClient (version 11) (https://github.com/mattn/go-oci8). I need to be able to load this object and browse results... t_cursor output parameter.
I have tried many strategies, I know how to map function parameters to go structures but I don't know how to work with t_cursor type since it seems not being implemented in InstantClient
Example of stored procedure
create or replace procedure EXAMPLE(a IN NUMBER, b IN NUMBER, c OUT T_CURSOR) AS BEGIN
[Edit] We have also tried to execute SQL blocks from code to try to handle this third parameter.
i.e.
If you add something like
declare
c t_cursor;
begin
EXAMPLE(:1, :2, c)
end
then I don't know how you can get the block to return a result set that contains the cursor.
declare
c t_cursor;
begin
EXAMPLE(:1, :2, c)
select 1, c from dual
end
The whole block returning the result of that select would be ideal but oracle blocks do not return result sets afaik.
Anyone who can bear a hand on this?
Thank you very much
It can be done with the driver https://github.com/rana/ora instead.
*Rset may be passed to Stmt.Exe when prepared with a stored procedure accepting an OUT SYS_REFCURSOR
The README.me even has that exact example.
Caveats:
It's unclear whether the database/sql interface may be used or you are limited to the lib specific API.
Instant Client gets restricted to versions from 12.1.0.1.0 on.
I have an Execute SQL Task which tries to execute a stored procedure, like this:
EXEC usp_stored_proc ?, ?, ? OUTPUT, ? OUTPUT;
I have 4 variables mapped to parameters. Ignoring the output parameters, these are both strings mapped to NVARCHAR params (as expected by the stored procedure).
When I run the package, an error tells me that execution failed with the message input string is not in the correct format. However, when I use a breakpoint to find the runtime values of the input parameters (or at least the variables mapped to them) and execute the same line of SQL in SSMS using the runtime values, it works fine.
Can anyone help? I'm at the end of my tether with this. I can't even find out the exact parameter causing the issue although it's probably both as the values follow the same format.
More details:
Connection type: OLE DB
Input Variable: String = schema.table
Mapped Param: NVARCHAR, ParamName = 0, ParamSize = -1
UPDATE
Solved the issue by making a new execute sql component that calls a stripped down procedure. I then slowly added lines of code to the procedure and additional parameters until arriving at the same component I started with and now it works. Comparing the original and rebuilt tasks, I see absolutely no differences (same with the procedure), so I don't know why this issue was occuring.
Try changing the parameter size (ParamSize) to match the parameter size within the stored procedure; if nvarchar(50) then set it to 50.
Solved the issue by making a new execute sql component that calls a stripped down procedure. I then slowly added lines of code to the procedure and additional parameters until arriving at the same component I started with and now it works. Comparing the original and rebuilt tasks, I see absolutely no differences (same with the procedure), so I don't know why this issue was occurring.
My query in SQL Developer is executing fine, but when I implement in APEX, the application won't even run. So just wondering if Developer version and APEX version are compatible.
There is no relationship between SQL Developer and your APEX application, at least as far as the scenario you're describing.
It would help to show your query, show your application not running, i.e. error messages encountered when running the query.
If you wanted, you could test the query in the APEX SQL Workshop.
SQL Developer runs its queries through a jdbc driver as a client application. APEX SQL runs inside the database via stored procedures.
Based on a comment you posted: what you executed in Apex' SQL Workshop is an anonymous PL/SQL block.
By the way, what is a? You never declared it.
How to transform it to a function? For example:
create or replace function f_test (a number)
return varchar2 is
begin
return case when a < 10 or a > 57 then 'Value is not in valid range'
else null
end;
end;
/
Call it as e.g. (in function's body)
return f_test(:P1_DEPTNO);
Aha, yet another comment: that is supposed to be a validation. Its type should be "PL/SQL function returning error text":
return case when :P1_DEPTNO < 10 or :P1_DEPTNO > 57
then 'Value is not in valid range'
else null
end;
Script export.database in Openbravo changes the return type of procedures that return RECORD (e.g. m_reservation_reallocate). In the xml:
function name="M_RESERVATION_REALLOCATE" type="NULL"
Are this normal?
Thanks in advance.
Openbravo's DB Source Manager does not support return type "RECORD" You can find out more about the return types which openbravo supports click here.
You can write your own HQL functions using DAL instead of writing any new functions that is specific to either postgreSQL or Oracle.