How do you parameterize Limit and Offset in a Snowflake Procedure? When calling this procedure like so: CALL GETJOBS ('test1', 3, 3);
The response is: "Failed: SQL compilation error:\nInvalid row count ':2' in limit clause"
It doesnt appear to be accepting limit and offset parameters.
CREATE OR REPLACE PROCEDURE "GETJOBS"("CODES" VARCHAR(16777216), "LIMIT" FLOAT, "OFFSET" FLOAT)
RETURNS VARIANT
LANGUAGE JAVASCRIPT
EXECUTE AS OWNER
AS '
var sql = snowflake.createStatement({
sqlText: `select jobnbr
from jobs
where islocked = 1
order by jobnbr
limit :2 offset :3;`,
binds: [CODES, LIMIT, OFFSET].map(function(x){return x === undefined ? null : x})
});
try {
var results = sql.execute();
var columns = sql.getColumnCount();
var array = [];
while (results.next()) {
let object = new Object;
var current = 1;
while (current <= columns) {
object[results.getColumnName(current).toString()] = results.getColumnValue(current);
current++;
}
array.push(object);
}
return array;
}
catch (err) {
return "Failed: " + err;
}
return null;
';
Your solution can be to change the following:
var sql = snowflake.createStatement({
sqlText: `select jobnbr
from jobs
where islocked = 1
order by jobnbr
limit :2 offset :3;`,
binds: [CODES, LIMIT, OFFSET].map(function(x){return x === undefined ? null : x})
});
to something like:
var sql_cmd = "select jobnbr from jobs where islocked = 1 order by jobnbr limit " + LIMIT + " offset " + OFFSET;
var sql = snowflake.createStatement({
sqlText: sql_cmd,
binds: [CODES].map(function(x){return x === undefined ? null : x})
});
but this would not use bindings for LIMIT and OFFSET. It would still work when calling stored proc like:
CALL GETJOBS ('test1', 3, 3);
Alternative approach using QUALIFY and ROW_NUMBER:
var sql = snowflake.createStatement({
sqlText: `select jobnbr
from jobs
where islocked = :1
qualify row_number() over(order by jobnbr) between :2 and :3
order by jobnbr`,
binds: [CODES, OFFSET + 1, LIMIT + OFFSET + 1].map(function(x){return x === undefined ? null : x})
});
Related
i'm working with Snowflake,
i created this Stored Procedure, it's purpose is to do the following steps:
extract the relevant 'application_id' values from the SQL query
use a FOR loop over a SQL query, with the 'application_id' values that i extracted
in step 1, and delete the relevant rows from the target table.
when i call the Stored Procedure, it runs without errors, but it doesn't do anything ( the relevant records are not been deleted).
i added my SP code,
please let me know if you see any syntax / logic errors,
thanks
CREATE OR REPLACE PROCEDURE DWH.sp_testing()
RETURNS string
LANGUAGE javascript
strict
EXECUTE AS owner
AS
$$
try
{
var application_list = ` SELECT application_id
FROM PUBLIC.my_source_table_name
WHERE my_flag = 1
`
var query_statement = snowflake.createStatement({sqlText: application_list});
var application_list_result = query_statement.execute();
for (i = 1; i <= query_statement.getRowCount(); i++)
{
application_list_result.next()
application_id = application_list_result.getColumnValue(1)
var delete_daily_records = `
DELETE FROM PUBLIC.my_target_table_name AS target
WHERE target.application_id = ${application_id}
`
snowflake.execute({sqlText: delete_daily_records});
}
}
catch (err)
{
throw "Failed: " + err;
}
$$
;
CALL DWH.sp_testing();
Are you sure your query is returning data? Also are you sure the target table has data matching your DELETE statement?
The following test works for me using your stored procedure:
select count(*) from citibike_trips where end_station_id=6215;
returns: 14565
Now, I created your stored proc:
CREATE OR REPLACE PROCEDURE sp_testing()
RETURNS string
LANGUAGE javascript
strict
EXECUTE AS owner
AS
$$
try
{
var application_list = `SELECT end_station_id
FROM citibike_trips
WHERE end_station_id=6215 limit 10
`
var query_statement = snowflake.createStatement({sqlText: application_list});
var application_list_result = query_statement.execute();
for (i = 1; i <= query_statement.getRowCount(); i++)
{
application_list_result.next()
end_station_id = application_list_result.getColumnValue(1)
var delete_daily_records = `
DELETE FROM citibike_trips AS target
WHERE target.end_station_id = ${end_station_id}
`
snowflake.execute({sqlText: delete_daily_records});
}
}
catch (err)
{
throw "Failed: " + err;
}
$$
;
Run it:
CALL SP_TESTING();
Shows NULL as result (expected since it is not returning anything).
But then when I check the table again:
select count(*) from citibike_trips where end_station_id=6215;
returns: 0
I am trying to do error handling for snowflake stored procedure. The stored procedure has 4 SQL statement inside. I need check errors for all for statement. When error pops, i need to know which Statement error. How to achieve this? Thanks in advance. I appreciate it.
CREATE OR REPLACE PROCEDURE dw.sp_fidctp_audit_trail_load(SYSDATE varchar)
RETURNS STRING
LANGUAGE javascript
AS
'
var sql_command = "truncate table STG.tb_fidctp_audit_trail_input";
var result = snowflake.execute ( {sqlText: sql_command, binds: [SYSDATE]});
sql_command =`COPY INTO STG.tb_fidctp_audit_trail_input (record_id,ORDER_ID,PARENT_ORDER_ID,EVENT_TYPE,EVENT_DATETIME,PRIMARY_STATE,SECONDARY_STATE,CURRENT_SERVICE_ID,CURRENT_EXECUTOR_ID,TRADING_QUANTITY,LIMIT_PRICE,EVENT_TEXT,ORDER_NOTES,REASON_TEXT,ROUTED_ORDER_CODE,ROOT_ORDER_ID,INSTRUMENT_CODE,ORDER_SOURCE,LEAVES,ACCT_ID,CUST_CROSS_ID,ORDER_FLAGS)
from #STG.CTP_STAGE/AUDIT_TRAIL.TOP.`;
sql_command += SYSDATE;
sql_command += ".psv.gz file_format = (FORMAT_NAME = ''STG.CTP_AUDIT_TRAIL_FF'', ERROR_ON_COLUMN_COUNT_MISMATCH = TRUE, encoding = ''iso-8859-1'') ";
try{
result = snowflake.execute (
{sqlText: sql_command, binds: [SYSDATE]}
);
return "Succeeded."; // Return a success/error indicator.
}
catch (err) {
return "Failed: " + err + "ERROR ON COPY STATEMENT"; // Return a success/error indicator.
}
sql_command = `delete from DW.tb_fidctp_audit_trail
where SYSDATE = to_date((:1), ''YYYYMMDD'')`;
result = snowflake.execute ( {sqlText: sql_command, binds: [SYSDATE]});
sql_command = ` insert into DW.tb_fidctp_audit_trail(sysdate, order_id, parent_order_id, root_order_id, event_datetime, instrument_code, event_text,reason_text, current_service_id, current_executor_id, trading_quantity, leaves )
select to_date(left(event_datetime, 8), ''YYYYMMDD''), order_id, parent_order_id, root_order_id, to_timestamp(left(event_datetime, 24), ''YYYYMMDD HH24:MI:SS.FF''), instrument_code, event_text, reason_text, current_service_id, current_executor_id, cast(trading_quantity as NUMBER(18,0)), cast(leaves as NUMBER(18,0))
from STG.tb_fidctp_audit_trail_input
`;
try{
snowflake.execute (
{sqlText: sql_command, binds: [SYSDATE]}
);
return "Succeeded."; // Return a success/error indicator.
}
catch (err) {
return "Failed: " + err + "ERROR ON INSERT STATEMENT"; // Return a success/error indicator.
}
'
;
There are a few things to do here. Let's start with the create header:
CREATE OR REPLACE PROCEDURE dw.sp_fidctp_audit_trail_load(SYSDATE varchar)
RETURNS STRING
LANGUAGE javascript
AS
'
When you use a single quote to enclose the body of a JavaScript procedure, it makes using single quotes in SQL statements inside the body very cumbersome, having to double the single quotes. To avoid that, you can use an alternate string terminator in Snowflake, $$ like this:
CREATE OR REPLACE PROCEDURE dw.sp_fidctp_audit_trail_load(SYSDATE varchar)
RETURNS STRING
LANGUAGE javascript
AS
$$
// Body goes here. There's no longer a need to escape single quotes
$$;
The issue with the return values is in this part:
return "Succeeded."; // Return a success/error indicator.
}
catch (err) {
return "Failed: " + err + "ERROR ON COPY STATEMENT"; // Return a success/error indicator.
}
In JavaScript, when the return statement is used in the main function, it leaves the main function, which terminates the stored procedure. This means if the copy statement succeeds, the procedure will exit and return "Succeeded." If the copy statement fails, it will hit a different return statement and exit with the error message.
Either way, it will terminate execution after that code block.
Take the return statements out of the successful branches of the try/catch blocks except for the last one.
You can just print the statement in the CATCH statement, either by printing the sql_command variable, or using getSqlText() function from the Statement object.
CREATE OR REPLACE PROCEDURE test_catch_error_sp(SYSDATE varchar)
RETURNS STRING
LANGUAGE javascript
AS
$$
var commands = [];
var params = [];
var results = {"success": [], "failed": []};
commands[0] = "create or replace table test1 (a int)";
params[0] = [];
commands[1] = "INSERT INTO test1 values (?)";
params[1] = [SYSDATE];
commands[2] = "drop table test1";
params[2] = [];
var stmt = null;
for (i = 0; i < commands.length; i++) {
try {
stmt = snowflake.createStatement({sqlText: commands[i], binds: params[i]});
var result = stmt.execute();
results['success'].push(stmt.getSqlText());
stmt = null; // reset it
} catch (err) {
// if createStatement failed, then stmt will still be null
results['failed'].push(
"Failed: " + err + ". ERROR ON STATEMENT: '" + stmt ? stmt.getSqlText() : commands[i] + "'"
);
}
}
return JSON.stringify(results);
$$
;
call test_catch_error_sp('2020-01-01');
-- returns:
-- {"success":["create or replace table test1 (a int)","drop table test1"],"failed":["INSERT INTO test1 values (?)"]}
I'm working through a stored procedure and wondering if there's a way to retrieve the anticipated result column list from a sql statement before fully executing.
Scenarios:
dynamic SQL
a UDF that might vary the columns outside of our control
EX:
//inbound parameter
SET QUERY_DEFINITION_ID = 12345;
//Initial statement pulls query text from bank of queries
var sqlText = getQueryFromQueryBank(QUERY_DEFINITION_ID);
//now we run our query
var cmd = {sqlText: sqlText };
stmt = snowflake.createStatement(cmd);
What I'd like to be able to do is say "right - before you run this, give me the anticipated column list" so I can compare it to what's expected.
EX:
Expected: [col1, col2, col3, col4]
Got: [col1]
Result: Oops. Don't run.
Rationale here is that I want to short-circuit the execution if something is missing - before it potentially runs for a while. I can validate all of this after the fact, but it would be really helpful to stop early.
Any ideas very much appreciated!
This sample SP code shows how to get a list of columns that a query will project into the result before you run the query. It should only be used for large, long running queries because it will take a few seconds to get the column list.
There are a couple of caveats. 1) It will only return the names of the columns. It won't tell you how they were built, that is, whether they're aliased, direct from a table, calculated, etc. 2) The example query I used is straight from the Snowflake documentation here https://docs.snowflake.com/en/user-guide/sample-data-tpcds.html#functional-query-definition. For convenience, I minimized the query to a single line. The output of the columns includes object qualifiers in addition to the column names, so V1.I_CATEGORY, V1.D_YEAR, V1.D_MOY, etc. If you don't want them to make it easier to compare names, you can strip off the qualifiers using the JavaScript split function on the dot and take index 1 of the resulting array.
create or replace procedure EXPLAIN_BEFORE_RUNNING()
returns string
language javascript
execute as caller
as
$$
// Set the context for the session to the TPC-H sample data:
executeNonQuery("use schema snowflake_sample_data.tpcds_sf10tcl;");
// Here's a complex query from the Snowflake docs (minimized to one line for convienience):
var sql = `with v1 as( select i_category, i_brand, cc_name, d_year, d_moy, sum(cs_sales_price) sum_sales, avg(sum(cs_sales_price)) over(partition by i_category, i_brand, cc_name, d_year) avg_monthly_sales, rank() over (partition by i_category, i_brand, cc_name order by d_year, d_moy) rn from item, catalog_sales, date_dim, call_center where cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and cc_call_center_sk= cs_call_center_sk and ( d_year = 1999 or ( d_year = 1999-1 and d_moy =12) or ( d_year = 1999+1 and d_moy =1)) group by i_category, i_brand, cc_name , d_year, d_moy), v2 as( select v1.i_category ,v1.d_year, v1.d_moy ,v1.avg_monthly_sales ,v1.sum_sales, v1_lag.sum_sales psum, v1_lead.sum_sales nsum from v1, v1 v1_lag, v1 v1_lead where v1.i_category = v1_lag.i_category and v1.i_category = v1_lead.i_category and v1.i_brand = v1_lag.i_brand and v1.i_brand = v1_lead.i_brand and v1.cc_name = v1_lag.cc_name and v1.cc_name = v1_lead.cc_name and v1.rn = v1_lag.rn + 1 and v1.rn = v1_lead.rn - 1) select * from v2 where d_year = 1999 and avg_monthly_sales > 0 and case when avg_monthly_sales > 0 then abs(sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1 order by sum_sales - avg_monthly_sales, 3 limit 100;`;
// Before actually running the query, generate an explain plan.
executeNonQuery("explain " + sql);
// Now read the column list from the explain plan from the result set.
var columnList = executeSingleValueQuery("COLUMN_LIST", `select "expressions" as COLUMN_LIST from table(result_scan(last_query_id())) where "operation" = 'Result';`);
// For now, just exit with the column list as the output...
return columnList;
// Your code here...
// Helper functions:
function executeNonQuery(queryString) {
var out = '';
cmd = {sqlText: queryString};
stmt = snowflake.createStatement(cmd);
var rs;
rs = stmt.execute();
}
function executeSingleValueQuery(columnName, queryString) {
var out;
cmd1 = {sqlText: queryString};
stmt = snowflake.createStatement(cmd1);
var rs;
try{
rs = stmt.execute();
rs.next();
return rs.getColumnValue(columnName);
}
catch(err) {
if (err.message.substring(0, 18) == "ResultSet is empty"){
throw "ERROR: No rows returned in query.";
} else {
throw "ERROR: " + err.message.replace(/\n/g, " ");
}
}
return out;
}
$$;
call Explain_Before_Running();
I have various streams but some of the streams are becoming stale. To avoid them becoming stale, I want to put some process in place which can read the 'show stream' property 'stale after', if it is only 1 day left, run a process to refresh the stream.
To achieve your goal you have to capture the output of SHOW STREAMS https://docs.snowflake.com/en/sql-reference/sql/show-streams.html. You can start building a stored procedure that runs it and returns its output as a resultset using TABLE(RESULT_SCAN(LAST_QUERY_ID())) similar to the following one, which could be enriched by a parameter for the time window you want to check (your "1 day left) and a subsequent CREATE OR REPLACE STREAM.
Please note this is not a full solution to your problem but only half of it as it does not include the action required to re-create the stale streams.
CREATE OR REPLACE PROCEDURE sp_show_stream_stale()
RETURNS VARIANT NOT NULL
LANGUAGE Javascript
EXECUTE AS Caller
AS
$$
var sql_command0 = snowflake.createStatement({ sqlText:`show streams in database`});
var sql_command1 = snowflake.createStatement({ sqlText:`SELECT "created_on"
, "name"
, "database_name"
, "schema_name"
, "owner"
, "comment"
, "table_name"
, "type"
, "stale"
, "mode"
, "stale_after"
FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))`});
try {
sql_command0.execute();
var db = sql_command1.execute();
var json_rows = {};
var array_of_rows = [];
var COLUMNS = ["created_on","name","database_name","schema_name","owner", "comment", "table_name", "type", "stale", "mode", "stale_after"];
var row_num = 1;
while (db.next()) {
json_rows = {};
for (var col_num = 0; col_num < COLUMNS.length; col_num = col_num + 1) {
var col_name = COLUMNS[col_num];
json_rows[col_name] = db.getColumnValue(col_num + 1);
}
array_of_rows.push(json_rows);
++row_num;
}
return array_of_rows;
}
catch (err) {
return "Failed: " + err;
}
$$;
As the resultset is a single JSON, you can run the stored procedure and soon after the following SELECT statement to get the resultset in tabular format.
CALL sp_show_stream_stale();
SELECT value:created_on::datetime as "created_on",
value:name::string as "name",
value:database_name::string as "database_name",
value:schema_name::string as "schema_name",
value:owner::string as "owner",
value:comment::string as "comment",
value:table_name::string as "table_name",
value:type::string as "type",
value:stale::string as "stale",
value:mode::string as "mode",
value:stale_after::datetime as "stale_after"
FROM (SELECT * FROM TABLE(RESULT_SCAN(LAST_QUERY_ID())))
, LATERAL FLATTEN(Input => sp_show_stream_stale)
WHERE DATEDIFF(Day, current_timestamp, value:stale_after::datetime) <= 1 ;
I am trying to assign a simple sql query result in stored procedure to a variable which I will use it later in another sql statement. If I execute below statement with out putting in procedure it works but not with stored procedure, I get an error while calling sp , Can someone please help me here ?
Procedure code:
$$
VAR NAME = 'ABC'
SET (COUNT_VALUE) = (SELECT COUNT(*) FROM COUNT_TABLE)
With out Procedure below code works
SET (COUNT_VALUE) = (SELECT COUNT(*) FROM COUNT_TABLE)
select $COUNT_VALUE
Thanks
Here's an example that uses a simple helper function to return a result set from a query. You can read the Snowflake docs to see what the result set API has in it. The getResultSet function returns a variable that has the results of the query. Since the SQL is a count, you need to use rs.next() once to get to the first row and then read the value in the aliased count(*). You could also use rs.getColumnValue(1) to get a column by ordinal position, but I recommend using column names with SQL aliases is necessary.
create or replace procedure COUNT_EXAMPLE()
returns string
language javascript
as
$$
var rowCount = 0;
var sql = "select count(*) as ROW_COUNT from SNOWFLAKE_SAMPLE_DATA.TPCH_SF10000.ORDERS;";
try {
var rs = getResultSet(sql);
if (rs.next()) {
var rowCount = rs.getColumnValue("ROW_COUNT");
} else {
return "Error: Count query failed.";
}
}
catch(err) {
return "Error: " + err.message;
}
return "The table has " + rowCount + " rows.";
//--------------------------- End of main function ---------------------------
function getResultSet(sql){
cmd1 = {sqlText: sql};
stmt = snowflake.createStatement(cmd1);
var rs;
rs = stmt.execute();
return rs;
}
$$;
call COUNT_EXAMPLE();