I read in the document that select count(*) is a Metadata operation in snowflake. SO no computation is warehouse used. But without any warehouse assigned the query select count(*) cannot be run. And once warehouse start it will start using credits. Can any one please explain this.
The count(*) does not require a running warehouse. You can see this behavior using this script:
-- Shut down a warehouse and do not allow auto-resume to test this
alter warehouse test suspend;
alter warehouse test set auto_resume = false;
use warehouse test;
-- This fails because it needs a running warehouse
select * from "SNOWFLAKE_SAMPLE_DATA"."TPCH_SF1"."ORDERS" limit 1;
-- This works because it's a metadata query
select count(*) from "SNOWFLAKE_SAMPLE_DATA"."TPCH_SF1"."ORDERS";
-- Simple aritemetic math on metadata queries is okay
select count(*) + 1 from "SNOWFLAKE_SAMPLE_DATA"."TPCH_SF1"."ORDERS";
-- Running functions requires a warehouse
select sqrt(count(*)) from "SNOWFLAKE_SAMPLE_DATA"."TPCH_SF1"."ORDERS";
--Remember to alter your warehouse back to auto-resume:
alter warehouse test set auto_resume = true;
The query can be run without any warehouse assigned - easy enough to test using the WebUI
Related
Problem: To solve data consistency on one of most important tables in our system, all writes will be covered with explicit transactions (appropriate to what given stored procedure / business application is doing). However, since there are many sources of writes, it would be best if we had numerical measure of progress.
Ratio of explicit to autocommited transactions on that particular table would be one such measure. If after all the rework, there are still autocommit transactions in stats, we have proof that we missed some procedure or application (or manual process).
How can SQL server be queried for such information?
As mentioned by #lptr you can use sys.dm_tran_session_transactions to check if a user transaction was opened.
CREATE OR ALTER TRIGGER dbo.check_tran
ON dbo.YourTable
AFTER INSERT, UPDATE, DELETE
AS
SET NOCOUNT ON;
IF (SELECT is_user_transaction FROM sys.dm_tran_session_transactions WHERE session_id = ##SPID) = 1
BEGIN
INSERT Audit_transactions (host_name, user_name, initial_batch)
SELECT HOST_NAME(), SUSER_SNAME(), CONCAT(b.parameters, ' ', b.event_info)
FROM sys.dm_exec_requests r
OUTER APPLY sys.dm_exec_input_buffer (##SPID, r.request_id) b
WHERE r.session_id = ##SPID
END;
sys.dm_tran_session_transactions requires VIEW SERVER STATE permissions, so if the user does not have that then you may need to sign the trigger with a certificate of a user that has that permission.
I am loading data through ODI into snowflake temp tables created with c$ needs to be dropped after load successful,how to drop those temp tables appreciate your suggestion
If you still need this, I wrote a stored procedure that will take a list of SQL generated dynamically and execute the lines one at a time. You can use it to run any list of generated SQL statements resulting from a select query, including dropping all tables matching a pattern such as c$%. First, here's the stored procedure:
create or replace procedure RunBatchSQL(sqlCommand String)
returns string
language JavaScript
as
$$
/**
* Stored procedure to execute multiple SQL statements generated from a SQL query
* Note that this procedure will always use the column named "SQL_COMMAND"
*
* #param {String} sqlCommand: The SQL query to run to generate one or more SQL commands
* #return {String}: A string containing all the SQL commands executed, each separated by a newline.
*/
cmd1_dict = {sqlText: SQLCOMMAND};
stmt = snowflake.createStatement(cmd1_dict);
rs = stmt.execute();
var s = '';
while (rs.next()) {
cmd2_dict = {sqlText: rs.getColumnValue("SQL_COMMAND")};
stmtEx = snowflake.createStatement(cmd2_dict);
stmtEx.execute();
s += rs.getColumnValue(1) + "\n";
}
return s;
$$
You can use this stored procedure to run any dynamically generated SQL statements in batch using the following script. Run the topmost query and it will be obvious what running the stored procedure with that query test as the parameter will do:
-- This is a select query that will generate a list of SQL commands to execute in batch.
-- This SQL will generate rows to drop all tables starting with c$. With minor edits
-- you could limit it to a specific database or schema.
select 'drop table ' || TABLE_CATALOG || '.' || TABLE_SCHEMA || '.' || "TABLE_NAME" as SQL_COMMAND
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME like 'c$%';
-- As a convenience, this grabs the last SQL run so that it's easier to insert into
-- the parameter used to call the stored procedure.
set query_text = ( select QUERY_TEXT
from table(information_schema.query_history(result_limit => 2))
where SESSION_ID = Current_Session() and QUERY_TYPE = 'SELECT' order by START_TIME desc);
-- Confirm that the query_text variable has the correct SQL query to generate our SQL commands (grants in this case) to run.
select $query_text;
-- Run the stored procedure. Note that to view its output better, double click on the output to see it in multi-line format,
Call RunBatchSQL($query_text);
--Check the last several queries run to make sure it worked.
select QUERY_TEXT
from table(information_schema.query_history(result_limit => 100))
where SESSION_ID = Current_Session() order by START_TIME desc;
The C$ prefixed work tables are a product of ODI use, but they are not created as actual Snowflake temporary tables so they do not benefit from an automatic deletion at JDBC session termination.
The ODI publishers note this about their C$ and I$ work tables:
When a scenario successfully completes, it will automatically delete these tables, as they're transitory and are no longer required. However, where a scenario does not complete successfully, it is possible these tables get left behind and from time to time it maybe desirable to clean up these tables to reclaim space.
For unsuccessful scenarios in your use of ODI that is likely leading to leftover tables on Snowflake, following the link above should help you run a procedure that deletes the leftover work tables (manually or on a schedule). Copying over the relevant procedure here for convenience:
To run the procedure:
Open ODI Studio and connect to the BI Apps ODI Repository.
Navigate to the Designer tab and use the navigator to navigate to: BI Apps Project -> Components -> DW -> Oracle -> Clean Work and Flow Tables folder
In the folder find the Clean Work and Flow Tables package and in this package is the UTILITIES_CLEAN_WORK_AND_FLOW_TABLES scenario.
Right click the scenario and select the 'Execute' option. At the prompt, provide the desired number of days to go back before deleting tables
When I run a simple alter non clustered column stored index query on table dbo.SampleDataTable (>50 million records):
ALTER INDEX CCI_SampleDataTable
ON dbo.SampleDataTable REBUILD
it blocks system tables INFORMATION_SCHEMA.ROUTINES and INFORMATION_SCHEMA.PARAMETERS.
The problem is our Reporting Server - Tableau Server. When user runs Tableau Report which uses a stored procedure in the database as a data source, Tableau Server sends this query to the database:
SELECT *
FROM INFORMATION_SCHEMA.ROUTINES R
JOIN INFORMATION_SCHEMA.PARAMETERS P ON R.SPECIFIC_SCHEMA = P.SPECIFIC_SCHEMA
AND R.SPECIFIC_NAME = P.SPECIFIC_NAME
AND R.ROUTINE_TYPE = 'PROCEDURE'
AND R.ROUTINE_SCHEMA <>'SYS'
and this query will be blocked when the ETL Process run ALTER INDEX QUERY.
Because that is the internal query from Tableau, I can't modify it with NOLOCK.
As far as I know, when ALTER INDEX runs, the system tables will be updated in real time therefore it will be blocked.
This problem of the system tables is very bad for us because the user can't access to report sometime.
Can you please help me with this problem? What should I do?
The stored procedure which is used by Tableau Report is very simple:
CREATE PROCEDURE SelectAllCustomers
#CustName VARCHAR(MAX)
AS
SELECT *
FROM dbo.SampleDataTable
WHERE CustName =#CustName
GO;
Is it possible to create a table across every schema in your database?
Specifically in Oracle.
I do not want to run the exact same query for all existing schemas.
Solution i have been using is,
Use below query to get all schema names,
select distinct owner FROM all_tables;
Get the result and use regular expression to append/prepend your table creation query.
^ - create table
$ - .tablename \( column1 varchar2(10)\);
run all the resulting queries in oracle work sheet.
You can use a bit of PL/SQL and execute immediate to do this.
For example you can create a table in all schemas if you connect as SYS User and execute the following Script:
begin
for cUsers in (select * from dba_users where account_status ='OPEN')
loop
execute immediate 'create table '||cUsers.username||'.myTable ( id number )';
end loop;
end;
My Application issues queries i.e.:
(#P1 bigint)SELECT * FROM MYTABLE WHERE MYTABLE_ID = #P1
(#P1 bigint)SELECT * FROM MYTABLE (NOLOCK) WHERE MYTABLE_ID = #P1
But in Plan Cache I also see queries like this (note the brackets):
(#1 int)SELECT * FROM [MYTABLE] WHERE [MYTABLE_ID]=#1
This queries are not issued by any application. Where do they come from? Are they created by MS SQL Simple Parameterization? (Parameterization set to simple)
This queries have refcount > 2 in sys.dm_exec_cached_plans DMV. What does refcount mean? Can someone elaborate?