(Submitting on behalf of a Snowflake client)
.........................
I would like to flexibly name tables I create.
For example
Set name = April
and then
Create table customer_data_$name as
I've found two recommended options thus far:
1 - Using Snowsql:
snowsql -c myconn -w trainingwh --variable NAME=April -f test.sql -o variable_substitution=True
script test.sql:
create table mytab_&NAME as
select current_timestamp ts;
2 - Using JavaScript Stored procedures:
create or replace procedure Proc_CT(NAME varchar)
RETURNS varchar(22)
LANGUAGE JAVASCRIPT
Execute as OWNER
as
$$
var ct_qry = `create or replace table mytab_`+NAME+`(i int);`
var ct_stmt = snowflake.createStatement({ sqlText: ct_qry });
ct_stmt.execute();
return 'Done.';
$$
;
CALL Proc_CT('April');
Two Questions:
A. Out of these two recommendations, is there any reason to leverage one more than the other?
B. Are there any other recommended options that can be leveraged in this situation?
.........................
Any advice or additional recommendations would be GREATLY APPRECIATED. Thank you!
Of the 2 options, I'd go with the stored procedure over Snowsql, because it's a more portable solution. Snowsql needs to be executed from a host machine, while stored procedures can be executed from anywhere, since they run inside Snowflake. This way, if you want to do this within an ELT/ETL process using a third-party tool, python, java, etc., you could simply call the SP to create your table.
As a note, I'd probably create a SP that renames a table for me, rather than doing the full CTAS statement. Your process could then create a table without an SP being involved, and then you could pass the table name + $name values into the SP and have it rename it for you. Either way works, but that's how I'd do it.
Related
I'm trying to add support for PostgreSQL to application which is currently written for MSSQL. In C# I'd like to have "one code for all" wherever possible.
There are bunch of stored procedures, which are rewritten in Postgres as functions.
In the code, to retrieve sp parameters metadata, there used to be ms specific sp_procedure_params_rowset.
I changed it to:
select p.data_type, p.parameter_name, p.ordinal_position, p.parameter_mode
from information_schema."routines" r
join information_schema.parameters p on r.specific_name=p.specific_name
where r.routine_schema = 'schema' and r.routine_name = 'procedure_name'
This works fine for Postgres, however, for MSSQL the parameter_mode is INOUT for parameters which were set as output.
For example CREATE PROCEDURE [sp_login] #login varchar(50), #uid int output... will return
varchar #login 1 IN
int #uid 2 INOUT
Now the code is failing for MSSQL, because it is expecting the parameter #uid.
Is there a reason why MSSQL would say in information_schema.parameters INOUT when in fact it is an OUT parameter?
Is there a "universal" way to retrieve the metadata from different databases? Or there's no way around it and I'll have to write some conditions in C#?
Understanding that side-effecting operators (like "insert") are disallowed in user-defined functions, how does one log (or otherwise track) calls to a specific user-defined function? I'd also like to capture the parameters passed into the UDF.
Ideally, the log would be a table into which information (time stamp and parameter values) about each call to the UDF is inserted. Reports and usage metrics could then be derived from that table.
I can't rewrite the UDF as a stored procedure, even of the same name, without breaking many downstream systems that are out in the wild that expect a UDF and that I have no control over.
Nor am I willing to enable any type of command shell features on our server that will diminish SQL Server's best-practice security defaults.
I found solution of your problem. It’s a little bit tricky and looks like a hack, but it seems it’s impossible to solve in another way.
The idea is to create a .NET SQL function which logs data where you need (file, Windows EventLog, db and so on), next create SQL UDF which calls this .NET function and finally call this SQL function from your functions passing all parameters needed to be logged. SQL Server doesn't check what is inside .net function and you can write there all logic you need.
The idea of how to create a .net SQL function without any security limitations is taken from this post.
So, create a .net library project with this one file
using System;
namespace SqlTest
{
public class LogEvent
{
[Microsoft.SqlServer.Server.SqlFunction]
public static int Log(string data)
{
System.IO.File.AppendAllText(#"C:\Log\LogUDF.txt", data);
return 0;
}
}
}
Sign it with some pfx certificate (project properties -> signing tab).
Next, call this query
USE [master]
CREATE ASYMMETRIC KEY LogKey FROM EXECUTABLE FILE =
'C:\Work\ConsoleApplication1\SqlTest\bin\Debug\SqlTest.dll'
CREATE LOGIN LogLogin FROM ASYMMETRIC KEY LogKey
GRANT UNSAFE ASSEMBLY TO LogLogin
GO
USE [MyDB]
CREATE ASSEMBLY SqlTest FROM
'C:\Work\ConsoleApplication1\SqlTest\bin\Debug\SqlTest.dll'
WITH PERMISSION_SET = unsafe
GO
CREATE FUNCTION dbo.Log( #data as nvarchar(200) )
RETURNS int
AS EXTERNAL NAME SqlTest.[SqlTest.LogEvent].Log
Here you need to change path to your compiled library, MyDB - your database name.
And you will create dbo.Log SQL function. Next you can call it where you need. For example like from this TestFunction
CREATE FUNCTION TestFunction
(
#p1 int
)
RETURNS int
AS
BEGIN
DECLARE #temp int
SELECT #temp = [dbo].[Log] ('fff')
RETURN 1
END
So, calling SELECT TestFunction(1) will write 'fff' text to C:\Log\LogUDF.txt file.
That’s it. A few important notes:
SQL server should have permissions (login/user) to write into file C:\Log\LogUDF.txt.
You should be SQL server admin
You can try the following:
1) Use SQL Profiler to check caught data for each of your different scenarios
Check SP:StmtCompleted to ensure that you catch the statements that execute within the stored procedure or used defined functions. Also make sure you include all required columns (TextData, LoginName, ApplicationName etc.). TextData is essential for this solution.
2) Check each scenario to see what you receive in the profiler. E.g.:
-- a mock function that is similar to what I have understood your function does
alter FUNCTION dbo.GetLoginResult(#username VARCHAR(64))
RETURNS INT
AS
BEGIN
DECLARE #l INT = LEN(#username)
IF (#l < 10)
RETURN 0
RETURN 1
-- DECLARE #Result INT
-- SELECT #Result = DATEPART(s, GETDATE()) % 3
-- RETURN #Result
END
go
select dbo.GetLoginResult('SomeGuy') --> `IF (#l < 10)` and `RETURN 0`
GO
select dbo.GetLoginResult('Some girl with a long name') --> `IF (#l < 10)` and `RETURN 1`
GO
So, if you can adapt your function to be written in a such a way that a specific instruction is executed when a particular output is about to be returned, you can recognize what is the result of the function based on the profiled information (as input and output values do not seem to be caught in the profiler)
3) Server-side tracing
As already suggested, SQL Profiler puts significant overhead, so you should use server-side tracing. Luckly, you can export just created profiling information as indicated here:
i) SQL Profiler -> File -> Export -> Script Trace Definition -> For SQL Server ..
ii) Replace the path in the generated script and run it -> remember generated id (it is trace id)
iii) you can open the file in the profiler and export its data to a table, after stopping the trace (it is locked by sqlserver process).
I've mostly worked with SQL Server so far and now I'm moving to Oracle for a new project.
I'm trying to create a proc that will return data to a .net app. The only way I got this to work is by using packages like this:
CREATE OR REPLACE PACKAGE GetAllDepartments AS
TYPE T_CURSOR IS REF CURSOR;
PROCEDURE p_GetAllDepartments
(
cur_Result OUT T_CURSOR
);
END GetAllDepartments;
CREATE OR REPLACE PACKAGE BODY GetAllDepartments AS
PROCEDURE p_GetAllDepartments
(
cur_Result OUT T_CURSOR
)
IS
BEGIN
OPEN cur_Result FOR
SELECT * FROM DEPARTMENTS;
END p_GetAllDepartments;
END GetAllDepartments;
Is this the only way to go with Oracle?, can't I just create the proc and call that directly? Thanks
Assuming you have a supported version of Oracle, you should be able to do something like
CREATE OR REPLACE PROCEDURE get_all_departments( p_result OUT SYS_REFCURSOR )
AS
BEGIN
OPEN p_result
FOR SELECT *
FROM departments;
END get_all_departments;
That said, you are normally better off from an organization standpoint using packages to collect procedures that do related things. In your case, for example, it would generally make sense to have a package that had all the procedures that dealt with departments (i.e. create_department, delete_department, etc.).
And from a general stylistic standpoint, it is more common in Oracle to create a view that implements whatever logic you would put in the stored procedure and to query that view rather than creating a stored procedure that just does a query.
You can do that without a package, e.g. by creating a function that returns result sets.
Those functions can be used like tables, e.g.: SELECT * FROM my_function()
This is especially efficient with pipelined function because the result is not buffered on the server but sent row by row to the client:
http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14251/adfns_packages.htm#i1008519
But the basic technique is still the same i.e. you have to define a type that is used for the return type of the function. You cannot have it return an "anonymous" result set like in PostgreSQL or SQL Server.
I'm migrating from SQL Server to PostgreSQL. I've seen from How to declare a variable in a PostgreSQL query that there is no such thing as temporary variables in native sql queries.
Well, I pretty badly need a few... How would I go about mixing in plpgsql? Must I create a function and then delete the function in order to get access to a language? that just seems error prone to me and I'm afraid I'm missing something.
EDIT:
cmd.CommandText="insert......" +
"declare #app int; declare #gid int;"+
"set #app=SCOPE_IDENTITY();"+ //select scope_identity will give us our RID that we just inserted
"select #gid=MAX(GROUPID) from HOUSEHOLD; set #gid=#gid+1; "+
"insert into HOUSEHOLD (APPLICANT_RID,GROUPID,ISHOH) values "+
"(#app,#gid,1);"+
"select #app";
rid=cmd.ExecuteScalar();
A direct rip from the application in which it's used. Note we are in the process of converting from SQL server to Postgre. (also, I've figured out the scope_identity() bit I think)
What is your schema for the table being inserted? I'll try and answer based on this assumption of the schema:
CREATE TABLE HOUSEHOLD (
APPLICANT_RID SERIAL, -- PostgreSQL auto-increment
GROUPID INTEGER,
ISHOH INTEGER
);
If I'm understanding your intent correctly, in PostgreSQL >= 8.2, the query would then be:
INSERT INTO HOUSEHOLD (GROUPID, ISHOH)
VALUES ((SELECT COALESCE(MAX(GROUPID)+1,1) FROM HOUSEHOLD), 1)
RETURNING APPLICANT_RID;
-- Added call to the COALESCE function to cover the case where HOUSEHOLD
-- is empty and MAX(GROUPID) returns NULL
In PostgreSQL >= 8.2, any INSERT/DELETE/UPDATE query may have a RETURNING clause that acts like a simple SELECT performed on the result set of the change query.
If you're using a language binding, you can hold the variables there.
For example with SQLAlchemy (python):
my_var = 'Reynardine'
session.query(User.name).filter(User.fullname==my_var)
If you're in psql, you have variables:
\set a 5
SELECT :a;
And if your logic is in PL/pgSQL:
tax := subtotal * 0.06;
Must I create a function and then
delete the function in order to get
access to a language?
Yes, but this shortcoming is going to be removed in PostgreSQL 8.5, with the addition of DO command. 8.5 is going to be released in 2010.
You can also declare session variables using plperl - http://www.postgresql.org/docs/8.4/static/plperl-global.html
you install a language that you want to use with the CREATE LANGUAGE command for known languages. Although you can use other languages.
Language installation docs
CREATE LANGUAGE usage doc
You will have to create a function to use it. If you do not want to make a permanent function in the db then the other choice would be to use a scrip in python or something that uses a postgresql driver to connect to the db and do queries. You can then manipulate or look through the data in the script. For instance in python you would install the pygresql library and in your script import pgdb which you can use to connect to the db.
PyGreSQL Info
I think that PostgreSQL's row-type variable would be the closest thing:
A variable of a composite type is
called a row variable (or row-type
variable). Such a variable can hold a
whole row of a SELECT or FOR query
result, so long as that query's column
set matches the declared type of the
variable.
You mentioned the post (How to declare a variable in a PostgreSQL query).
I believe there is a suitable answer farther down the chain of solutions if using psql and the \set command:
my_db=> \set myvar 5
my_db=> SELECT :myvar + 1 AS my_var_plus_1;
I'm using ADO.NET to access SQL Server 2005 and would like to be able to log from inside the T-SQL stored procedures that I'm calling. Is that somehow possible?
I'm unable to see output from the 'print'-statement when using ADO.NET and since I want to use logging just for debuging the ideal solution would be to emit messages to DebugView from SysInternals.
I think writing to a log table would be my preference.
Alternatively, as you are using 2005, you could write a simple SQLCLR procedure to wrap around the EventLog.
Or you could use xp_logevent if you wanted to write to SQL log
I solved this by writing a SQLCLR-procedure as Eric Z Beard suggested. The assembly must be signed with a strong name key file.
using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
public partial class StoredProcedures
{
[Microsoft.SqlServer.Server.SqlProcedure]
public static int Debug(string s)
{
System.Diagnostics.Debug.WriteLine(s);
return 0;
}
}
}
Created a key and a login:
USE [master]
CREATE ASYMMETRIC KEY DebugProcKey FROM EXECUTABLE FILE =
'C:\..\SqlServerProject1\bin\Debug\SqlServerProject1.dll'
CREATE LOGIN DebugProcLogin FROM ASYMMETRIC KEY DebugProcKey
GRANT UNSAFE ASSEMBLY TO DebugProcLogin
Imported it into SQL Server:
USE [mydb]
CREATE ASSEMBLY SqlServerProject1 FROM
'C:\..\SqlServerProject1\bin\Debug\SqlServerProject1.dll'
WITH PERMISSION_SET = unsafe
CREATE FUNCTION dbo.Debug( #message as nvarchar(200) )
RETURNS int
AS EXTERNAL NAME SqlServerProject1.[StoredProcedures].Debug
Then I was able to log in T-SQL procedures using
exec Debug #message = 'Hello World'
You can either log to a table, by simply inserting a new row, or you can implement a CLR stored procedure to write to a file.
Be careful with writing to a table, because if the action happens in a transaction and the transaction gets rolled back, your log entry will disappear.
Logging from inside a SQL sproc would be better done to the database itself. T-SQL can write to files but it's not really designed for it.
There's the PRINT command, but I prefer logging into a table so you can query it.
You can write rows to a log table from within a stored procedure. As others have indicated, you could go out of your way to write to some text file or other log with CLR or xp_logevent, but it seems like you need more volume than would be practical for such uses.
The tough cases occur (and it's these that you really need your log for) when transactions fail. Since any logging that occurs during these transactions will be rolled back along with the transaction that they are part of, it is best to have a logging API that your clients can use to log errors. This can be a simple DAL that either logs to the same database, or to a shared one.
For what it's worth, I've found that when I don't assign an InfoMessage handler to my SqlConnection:
sqlConnection.InfoMessage += new SqlInfoMessageEventHandler(MySqlConnectionInfoMessageHandler);
where the signature of the InfoMessageHandler looks like this:
MySqlConnectionInfoMessageHandler(object sender, SqlInfoMessageEventArgs e)
then my PRINT statements in my Stored Procs do not appear in DbgView.
You could use output variables for passing back messages, but that relies on the proc executing without errors.
create procedure usp_LoggableProc
#log varchar(max) OUTPUT
as
-- T-SQL statement here ...
select #log = #log + 'X is foo'
And then in your ADO code somehwere:
string log = (string)SqlCommand.Parameters["#log"].Value;
You could use raiserror to create your own custom errors with the information that you require and that will be available to you through the usual SqlException Errors collection in your ADO code:
RAISERROR('X is Foo', 10, 1)
Hmmm but yeah, can't help feeling just for debugging and in your situation, just insert varchar messages to an error table like the others have suggested and select * from it when you're debugging.
You may want to check Log4TSQL. It provides Database-Logging for Stored Procedures and Triggers in SQL Server 2005 - 2008. You have the possibility to set separate, independent log-levels on a per Procedure/Trigger basis.
Use cmd commands with cmdshell
I found this while searching for an answer to this question.
https://www.databasejournal.com/features/mssql/article.php/1467601/A-general-logging-t-sql-process-to-write-to-txt-files.htm
select #cmdtxt = "echo " + #logEntry + " >> drive:\path\filename.txt"
exec master..xp_cmdshell #cmdtxt
I've been searching for a way to do this, as I am trying to debug some complicated, chained, stored procedures, all that are called by an external API, and which operate in the context of a transaction.
I'd been writing diagnostic messages into a logging file, but if the transaction rolls back, the new log entries disappear with the rollback. I found a way! And it works pretty well. And it has already saved me many, many hours of debugging time.
Create a linked server to the same SQL instance, using the login's
security context. In my case, the simplest method was to use the
localhost loop address, 127.0.0.1
Set the linked server to enable RPC, and to NOT "Enable Promotion of
Distributed Transactions". This means that calls through that
server will take place outside of your transaction context.
In your logging procedure, (I have an example excerpted below) write
to the log table using the procedure through loopback linked server
if you are in a transaction. You can write to it the usual way
if your are not. Writing though the linked server is considerably
slower than direct DML.
Voila! My in-process logging survives the rollback, and I can find out what's happening internally when things are going south.
I can't claim credit for thinking of this--I found the approach after some time with Google, but I'm so pleased with the result I felt like I had to share it.
USE TX
GO
CREATE PROCEDURE dbo.LogError(#errorSource Varchar(32), #msg Varchar(400))
AS BEGIN
SET NOCOUNT ON
IF ##TRANCOUNT > 0
EXEC [127.0.0.1].TX.dbo.LogError #errorSource, #msg
ELSE
INSERT INTO TX.dbo.ErrorLog(source_module, message)
SELECT #errorSource, #msg
END
GO