How to drop temp tables created in snowflake - database

I am loading data through ODI into snowflake temp tables created with c$ needs to be dropped after load successful,how to drop those temp tables appreciate your suggestion

If you still need this, I wrote a stored procedure that will take a list of SQL generated dynamically and execute the lines one at a time. You can use it to run any list of generated SQL statements resulting from a select query, including dropping all tables matching a pattern such as c$%. First, here's the stored procedure:
create or replace procedure RunBatchSQL(sqlCommand String)
returns string
language JavaScript
as
$$
/**
* Stored procedure to execute multiple SQL statements generated from a SQL query
* Note that this procedure will always use the column named "SQL_COMMAND"
*
* #param {String} sqlCommand: The SQL query to run to generate one or more SQL commands
* #return {String}: A string containing all the SQL commands executed, each separated by a newline.
*/
cmd1_dict = {sqlText: SQLCOMMAND};
stmt = snowflake.createStatement(cmd1_dict);
rs = stmt.execute();
var s = '';
while (rs.next()) {
cmd2_dict = {sqlText: rs.getColumnValue("SQL_COMMAND")};
stmtEx = snowflake.createStatement(cmd2_dict);
stmtEx.execute();
s += rs.getColumnValue(1) + "\n";
}
return s;
$$
You can use this stored procedure to run any dynamically generated SQL statements in batch using the following script. Run the topmost query and it will be obvious what running the stored procedure with that query test as the parameter will do:
-- This is a select query that will generate a list of SQL commands to execute in batch.
-- This SQL will generate rows to drop all tables starting with c$. With minor edits
-- you could limit it to a specific database or schema.
select 'drop table ' || TABLE_CATALOG || '.' || TABLE_SCHEMA || '.' || "TABLE_NAME" as SQL_COMMAND
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME like 'c$%';
-- As a convenience, this grabs the last SQL run so that it's easier to insert into
-- the parameter used to call the stored procedure.
set query_text = ( select QUERY_TEXT
from table(information_schema.query_history(result_limit => 2))
where SESSION_ID = Current_Session() and QUERY_TYPE = 'SELECT' order by START_TIME desc);
-- Confirm that the query_text variable has the correct SQL query to generate our SQL commands (grants in this case) to run.
select $query_text;
-- Run the stored procedure. Note that to view its output better, double click on the output to see it in multi-line format,
Call RunBatchSQL($query_text);
--Check the last several queries run to make sure it worked.
select QUERY_TEXT
from table(information_schema.query_history(result_limit => 100))
where SESSION_ID = Current_Session() order by START_TIME desc;

The C$ prefixed work tables are a product of ODI use, but they are not created as actual Snowflake temporary tables so they do not benefit from an automatic deletion at JDBC session termination.
The ODI publishers note this about their C$ and I$ work tables:
When a scenario successfully completes, it will automatically delete these tables, as they're transitory and are no longer required. However, where a scenario does not complete successfully, it is possible these tables get left behind and from time to time it maybe desirable to clean up these tables to reclaim space.
For unsuccessful scenarios in your use of ODI that is likely leading to leftover tables on Snowflake, following the link above should help you run a procedure that deletes the leftover work tables (manually or on a schedule). Copying over the relevant procedure here for convenience:
To run the procedure:
Open ODI Studio and connect to the BI Apps ODI Repository.
Navigate to the Designer tab and use the navigator to navigate to: BI Apps Project -> Components -> DW -> Oracle -> Clean Work and Flow Tables folder
In the folder find the Clean Work and Flow Tables package and in this package is the UTILITIES_CLEAN_WORK_AND_FLOW_TABLES scenario.
Right click the scenario and select the 'Execute' option. At the prompt, provide the desired number of days to go back before deleting tables

Related

Remote Call Compiles SPROC differently to local, causes error

I have a stored proc that returns details of SQL Agent Jobs on the local server. There is a master script that calls this proc, using OPENQUERY, against every SQL server in the ecosystem. In pseudocode, the master script looks something like this:
FOR EACH #LinkedServer in the list
SET #SQL = 'INSERT #Results SELECT * FROM OPENQUERY(' + #LinkedServer + ',''EXEC ScriptToGetAgentJobInfo'')'
EXEC sp_executesql #SQL
NEXT #LinkedServer
Some of the agent jobs are created from SSRS report subscriptions, so they have horrible looking names. In order to replace them with the name of the report that is the target of the subscription, I appeal to the ReportServer database on the #LinkedServer, as part of the ScriptToGetAgentJobInfo.
However, not every server contains a ReportServer database, so sometimes this appeal would fail. To get round that failure, I have the following lines of script:
DECLARE #Reports TABLE
( AgentJob SYSNAME
,Reportname NVARCHAR(128));
IF EXISTS(SELECT 1 FROM [master].[dbo].sysdatabases WHERE [name] = 'ReportServer')
BEGIN;
INSERT #Reports(AgentJob,Reportname)
SELECT Job.job_id, Report.[Name]
FROM
ReportServer.dbo.ReportSchedule AS ReportSched
INNER JOIN dbo.sysjobs AS Job ON CONVERT(SYSNAME,ReportSched.ScheduleID) = Job.[name]
INNER JOIN ReportServer.dbo.Subscriptions AS Subscription ON ReportSched.SubscriptionID = Subscription.SubscriptionID
INNER JOIN ReportServer.dbo.[Catalog] AS Report ON Subscription.report_oid = Report.itemid;
END;
The idea is that if the reportserver database doesn't exist, I can avoid any calls to it, which would error, but if it does, I can get data from it. I then join the #Reports table to my SQL Agent job query with a LEFT JOIN to show the name of the relevant report if there is one.
All this works fine when I run the script locally, but when it is called through the master procedure, I get an error saying Invalid object name 'ReportServer.dbo.ReportSchedule'.
I can get round this problem by making the reportserver select statement "dynamic" (although it is totally static) and calling it with another sp_executesql call, but I really hate doing this!
So my question is this: Why does the error only occur when calling the script remotely and how can I avoid it without recourse to dynamic sql?
The master script is written and run in SQL Server 14.0, while the linked server that is causing the problem is only on SQL Server 10.50.

Create table across all schemas at once

Is it possible to create a table across every schema in your database?
Specifically in Oracle.
I do not want to run the exact same query for all existing schemas.
Solution i have been using is,
Use below query to get all schema names,
select distinct owner FROM all_tables;
Get the result and use regular expression to append/prepend your table creation query.
^ - create table
$ - .tablename \( column1 varchar2(10)\);
run all the resulting queries in oracle work sheet.
You can use a bit of PL/SQL and execute immediate to do this.
For example you can create a table in all schemas if you connect as SYS User and execute the following Script:
begin
for cUsers in (select * from dba_users where account_status ='OPEN')
loop
execute immediate 'create table '||cUsers.username||'.myTable ( id number )';
end loop;
end;

Temporary table from stored procedure result in SQL Server

Can we create a temporary table from the stored procedure results dynamically?
I do not want to declare the temporary table columns manually. It shud take the schema of table from the stored procedure results.
use one of the Rowset Functions:
SELECT *
INTO #Temp
FROM OPENQUERY(SERVERNAME, 'EXEC pr_StorProcName')
Use following syntax template to create temp table on basis of result set.
Select * into #temptable from mytable
Select column1,column2,..columnn into #temptable from mytable
Notes:
The SELECT INTO statement is very fast, for one reason: the command isn't logged for backup purposes. More precisely, the command can be inside a transaction and any rollback command will correctly undo its effects. However, the new values aren't permanently stored in the log file, therefore after this command you can only perform a complete database backup (incremental backup raise errors). This explains why you have to explicitly enable this functionality for non-temporary tables (temporary tables are never included in backup, so you don't need to use the sp_dboption command before using SELECT INTO with a temporary table).

INSERT in Stored Procedure called via JDBC

I have an MSSQL Server 2008 (Express) set up.
In my database I have a set of tables and a stored procedure.
What I want to achieve is get any changes that have been made to an existing table, and return them at the end of the procedure. The stored procedure I have created works fine when I run it locally within MSSQL Management Studio.
However, when I call the procedure through a JDBC connection certain parts of the procedure seem to have not completed.
The summary of what I'm trying to do is as follows:
1) Put a snapshot of the data contained in CurrentTableA into #CurrentShotA (temporary table)
2) Compare #CurrentShotA with PreviousTableA
3) Insert differences into #TempTableB
(this equates to new rows or altered rows in #CurrentShotA)
4) Empty PreviousTableA
5) Insert contents of #CurrentShotA into PreviousTableA
6) Select * from #TempTableB (return all new rows and changes)
Step 6 returns the data correctly the first time it is called via JDBC.
When the procedure is called the second and subsequent times it is clear that step 5 has not completed as expected. PreviousTableA is always empty when it should contain a snapshot of the old data.
Question is why does the procedure work properly when called with in MSSQL Management studio but not when I call it via JDBC?
CREATE PROCEDURE [dbo].[getUpdatedSchedules]
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- Check of the temporary table exists, delete if it does
IF OBJECT_ID('#TempTableB','U')IS NOT NULL
begin
drop table #TempTableB
end
-- Force the creation of the temporary tables quickly
select * into #TempTableB from dbo.CurrentTableA where 1=0
select * into #CurrentShotA from dbo.CurrentTableA where 1=0
-- Get the differences between schedules and put into #TempTableB
insert #CurrentShotA select * from dbo.CurrentTableA
insert #TempTableB select * from #CurrentShotA
except select * from dbo.PreviousTableA
TRUNCATE TABLE dbo.PreviousTableA
insert dbo.PreviousTableA select * from #CurrentShotA
select * from #TempTableB
END
GO
I'm new enough to stored procedures and MSSQL configuration so I have considered that it might be a permissions issue. I login to MSSQL Studio using SQL authentication that is not linked to a windows account and the procedure runs as normal so I don't think it's permissions.
I hope my explanation and question is clear enough. I'd appreciate any thoughts or suggestions as to what I am doing wrong.
There's no need to check if #TempTableB exists since it will be dropped automatically once the procedure finishes executing, just as the other temp tables are dropped automatically; perhaps you expect these tables to have data on subsequent calls to the proc and that's why you think it doesn't work when you call it from Java?
When you execute these SQL statements from SSMS they may have data in them since it's all the same database session, but that's not the case when called using JDBC.

Errors: "INSERT EXEC statement cannot be nested." and "Cannot use the ROLLBACK statement within an INSERT-EXEC statement." How to solve this?

I have three stored procedures Sp1, Sp2 and Sp3.
The first one (Sp1) will execute the second one (Sp2) and save returned data into #tempTB1 and the second one will execute the third one (Sp3) and save data into #tempTB2.
If I execute the Sp2 it will work and it will return me all my data from the Sp3, but the problem is in the Sp1, when I execute it it will display this error:
INSERT EXEC statement cannot be nested
I tried to change the place of execute Sp2 and it display me another error:
Cannot use the ROLLBACK statement
within an INSERT-EXEC statement.
This is a common issue when attempting to 'bubble' up data from a chain of stored procedures. A restriction in SQL Server is you can only have one INSERT-EXEC active at a time. I recommend looking at How to Share Data Between Stored Procedures which is a very thorough article on patterns to work around this type of problem.
For example a work around could be to turn Sp3 into a Table-valued function.
This is the only "simple" way to do this in SQL Server without some giant convoluted created function or executed sql string call, both of which are terrible solutions:
create a temp table
openrowset your stored procedure data into it
EXAMPLE:
INSERT INTO #YOUR_TEMP_TABLE
SELECT * FROM OPENROWSET ('SQLOLEDB','Server=(local);TRUSTED_CONNECTION=YES;','set fmtonly off EXEC [ServerName].dbo.[StoredProcedureName] 1,2,3')
Note: You MUST use 'set fmtonly off', AND you CANNOT add dynamic sql to this either inside the openrowset call, either for the string containing your stored procedure parameters or for the table name. Thats why you have to use a temp table rather than table variables, which would have been better, as it out performs temp table in most cases.
OK, encouraged by jimhark here is an example of the old single hash table approach: -
CREATE PROCEDURE SP3 as
BEGIN
SELECT 1, 'Data1'
UNION ALL
SELECT 2, 'Data2'
END
go
CREATE PROCEDURE SP2 as
BEGIN
if exists (select * from tempdb.dbo.sysobjects o where o.xtype in ('U') and o.id = object_id(N'tempdb..#tmp1'))
INSERT INTO #tmp1
EXEC SP3
else
EXEC SP3
END
go
CREATE PROCEDURE SP1 as
BEGIN
EXEC SP2
END
GO
/*
--I want some data back from SP3
-- Just run the SP1
EXEC SP1
*/
/*
--I want some data back from SP3 into a table to do something useful
--Try run this - get an error - can't nest Execs
if exists (select * from tempdb.dbo.sysobjects o where o.xtype in ('U') and o.id = object_id(N'tempdb..#tmp1'))
DROP TABLE #tmp1
CREATE TABLE #tmp1 (ID INT, Data VARCHAR(20))
INSERT INTO #tmp1
EXEC SP1
*/
/*
--I want some data back from SP3 into a table to do something useful
--However, if we run this single hash temp table it is in scope anyway so
--no need for the exec insert
if exists (select * from tempdb.dbo.sysobjects o where o.xtype in ('U') and o.id = object_id(N'tempdb..#tmp1'))
DROP TABLE #tmp1
CREATE TABLE #tmp1 (ID INT, Data VARCHAR(20))
EXEC SP1
SELECT * FROM #tmp1
*/
My work around for this problem has always been to use the principle that single hash temp tables are in scope to any called procs. So, I have an option switch in the proc parameters (default set to off). If this is switched on, the called proc will insert the results into the temp table created in the calling proc. I think in the past I have taken it a step further and put some code in the called proc to check if the single hash table exists in scope, if it does then insert the code, otherwise return the result set. Seems to work well - best way of passing large data sets between procs.
This trick works for me.
You don't have this problem on remote server, because on remote server, the last insert command waits for the result of previous command to execute. It's not the case on same server.
Profit that situation for a workaround.
If you have the right permission to create a Linked Server, do it.
Create the same server as linked server.
in SSMS, log into your server
go to "Server Object
Right Click on "Linked Servers", then "New Linked Server"
on the dialog, give any name of your linked server : eg: THISSERVER
server type is "Other data source"
Provider : Microsoft OLE DB Provider for SQL server
Data source: your IP, it can be also just a dot (.), because it's localhost
Go to the tab "Security" and choose the 3rd one "Be made using the login's current security context"
You can edit the server options (3rd tab) if you want
Press OK, your linked server is created
now your Sql command in the SP1 is
insert into #myTempTable
exec THISSERVER.MY_DATABASE_NAME.MY_SCHEMA.SP2
Believe me, it works even you have dynamic insert in SP2
I found a work around is to convert one of the prods into a table valued function. I realize that is not always possible, and introduces its own limitations. However, I have been able to always find at least one of the procedures a good candidate for this. I like this solution, because it doesn't introduce any "hacks" to the solution.
I encountered this issue when trying to import the results of a Stored Proc into a temp table, and that Stored Proc inserted into a temp table as part of its own operation. The issue being that SQL Server does not allow the same process to write to two different temp tables at the same time.
The accepted OPENROWSET answer works fine, but I needed to avoid using any Dynamic SQL or an external OLE provider in my process, so I went a different route.
One easy workaround I found was to change the temporary table in my stored procedure to a table variable. It works exactly the same as it did with a temp table, but no longer conflicts with my other temp table insert.
Just to head off the comment I know that a few of you are about to write, warning me off Table Variables as performance killers... All I can say to you is that in 2020 it pays dividends not to be afraid of Table Variables. If this was 2008 and my Database was hosted on a server with 16GB RAM and running off 5400RPM HDDs, I might agree with you. But it's 2020 and I have an SSD array as my primary storage and hundreds of gigs of RAM. I could load my entire company's database to a table variable and still have plenty of RAM to spare.
Table Variables are back on the menu!
I recommend to read this entire article. Below is the most relevant section of that article that addresses your question:
Rollback and Error Handling is Difficult
In my articles on Error and Transaction Handling in SQL Server, I suggest that you should always have an error handler like
BEGIN CATCH
IF ##trancount > 0 ROLLBACK TRANSACTION
EXEC error_handler_sp
RETURN 55555
END CATCH
The idea is that even if you do not start a transaction in the procedure, you should always include a ROLLBACK, because if you were not able to fulfil your contract, the transaction is not valid.
Unfortunately, this does not work well with INSERT-EXEC. If the called procedure executes a ROLLBACK statement, this happens:
Msg 3915, Level 16, State 0, Procedure SalesByStore, Line 9 Cannot use the ROLLBACK statement within an INSERT-EXEC statement.
The execution of the stored procedure is aborted. If there is no CATCH handler anywhere, the entire batch is aborted, and the transaction is rolled back. If the INSERT-EXEC is inside TRY-CATCH, that CATCH handler will fire, but the transaction is doomed, that is, you must roll it back. The net effect is that the rollback is achieved as requested, but the original error message that triggered the rollback is lost. That may seem like a small thing, but it makes troubleshooting much more difficult, because when you see this error, all you know is that something went wrong, but you don't know what.
I had the same issue and concern over duplicate code in two or more sprocs. I ended up adding an additional attribute for "mode". This allowed common code to exist inside one sproc and the mode directed flow and result set of the sproc.
what about just store the output to the static table ? Like
-- SubProcedure: subProcedureName
---------------------------------
-- Save the value
DELETE lastValue_subProcedureName
INSERT INTO lastValue_subProcedureName (Value)
SELECT #Value
-- Return the value
SELECT #Value
-- Procedure
--------------------------------------------
-- get last value of subProcedureName
SELECT Value FROM lastValue_subProcedureName
its not ideal, but its so simple and you don't need to rewrite everything.
UPDATE:
the previous solution does not work well with parallel queries (async and multiuser accessing) therefore now Iam using temp tables
-- A local temporary table created in a stored procedure is dropped automatically when the stored procedure is finished.
-- The table can be referenced by any nested stored procedures executed by the stored procedure that created the table.
-- The table cannot be referenced by the process that called the stored procedure that created the table.
IF OBJECT_ID('tempdb..#lastValue_spGetData') IS NULL
CREATE TABLE #lastValue_spGetData (Value INT)
-- trigger stored procedure with special silent parameter
EXEC dbo.spGetData 1 --silent mode parameter
nested spGetData stored procedure content
-- Save the output if temporary table exists.
IF OBJECT_ID('tempdb..#lastValue_spGetData') IS NOT NULL
BEGIN
DELETE #lastValue_spGetData
INSERT INTO #lastValue_spGetData(Value)
SELECT Col1 FROM dbo.Table1
END
-- stored procedure return
IF #silentMode = 0
SELECT Col1 FROM dbo.Table1
Declare an output cursor variable to the inner sp :
#c CURSOR VARYING OUTPUT
Then declare a cursor c to the select you want to return.
Then open the cursor.
Then set the reference:
DECLARE c CURSOR LOCAL FAST_FORWARD READ_ONLY FOR
SELECT ...
OPEN c
SET #c = c
DO NOT close or reallocate.
Now call the inner sp from the outer one supplying a cursor parameter like:
exec sp_abc a,b,c,, #cOUT OUTPUT
Once the inner sp executes, your #cOUT is ready to fetch. Loop and then close and deallocate.
If you are able to use other associated technologies such as C#, I suggest using the built in SQL command with Transaction parameter.
var sqlCommand = new SqlCommand(commandText, null, transaction);
I've created a simple Console App that demonstrates this ability which can be found here:
https://github.com/hecked12/SQL-Transaction-Using-C-Sharp
In short, C# allows you to overcome this limitation where you can inspect the output of each stored procedure and use that output however you like, for example you can feed it to another stored procedure. If the output is ok, you can commit the transaction, otherwise, you can revert the changes using rollback.
On SQL Server 2008 R2, I had a mismatch in table columns that caused the Rollback error. It went away when I fixed my sqlcmd table variable populated by the insert-exec statement to match that returned by the stored proc. It was missing org_code. In a windows cmd file, it loads result of stored procedure and selects it.
set SQLTXT= declare #resets as table (org_id nvarchar(9), org_code char(4), ^
tin(char9), old_strt_dt char(10), strt_dt char(10)); ^
insert #resets exec rsp_reset; ^
select * from #resets;
sqlcmd -U user -P pass -d database -S server -Q "%SQLTXT%" -o "OrgReport.txt"

Resources