I am trying write the copy statement in snowflake scripting. My copy statement is working fine out side the stored procedure. when i put the copy statement inside stored procedure it is not working.
it is so weird. Please could anyone help me on this?
create or replace procedure sp_demo_load(sysdate varchar)
returns varchar not null
language sql
as
$$
begin
copy into stg.tb_demo_input (name, id, address)
from #stg.demo_stage/DEMO.20220427.psv.gz
file_format = (format_name = 'stg.demo_ff',
error_on_column_count_mismatch = true, encoding = 'iso-8859-1') ;
end;
$$
;
Your copy statement "worked fine the first time" but if you run it again you will get 0 files loaded, because it has already loaded those file. This is because Snowflake is tracking which file have been loaded, thus there is no point to load them again.
Which means your SP is just fine.
To test truncate the destination table, and then run your SP, it should load the files, on the first run.
Related
the copy into command returns an output dataset.
CTAS can create a table from the results of a query.
combining the two, we would expect to get the list of loaded files into a new table.
CREATE TABLE MY_LOADED_FILES
AS
COPY INTO mytable
FROM #my_int_stage;
However, this returns:
SQL compilation error: syntax error line 3 at position 0 unexpected 'copy'.
What am I doing wrong?
It doesn't look like you can put a COPY INTO statement inside another statement unfortunately. There is a way to do this however by using the result_scan function to return the results of the previous query.
copy into test_database.public.test_table from #my_int_stage;
create temporary table test_database.public.test_table_results as (
select * from table(result_scan(LAST_QUERY_ID()))
);
Of course you need to make sure the second query runs in the same session as the copy statement and also that it is run directly after the copy statement. Alternatively you can use the query id with the result_scan.
If you want to see which files were loaded why don't you just look at the copy_history of the table?
I am trying to create an SSIS package for Stored Procedure Deployment and Backup for our project.
I have some .sql file, each file contains one stored procedure definition and the name of the file is the stored procedure name itself.
I am trying to do the following by using SSIS
Read all files names one by one
Find the definition of each stored procedure if it exists in the database
If exists, then save the definition with the same name in a different folder (In my case it's a ROLLBACK folder)
For all new SP it save in a same file named DropNewSp.sql with DROP STORED PROCEDURE command.
After completing the backup process, execute all files in the destination database.
I am able to generate the desire .sql files, but I am faceing the following problem
1. The package also generated unwanted blank .sql file for all new Stored Procedure
2. The execution process failed if the stored procedure has some dependency on subsequent stored proc
In this answer, I will provide the main steps with some references to get more information on how to achieve each step. Even if I agree with the comments mentioned above that this is not the job of SSIS.
Add a foreach loop container that loop over .sql files and store the file name inside a variable:
Looping over files with the Foreach Loop
Lesson 2-2: Add and configure the Foreach Loop container
Load multiple source files
Add an Expression Task to retrieve the file name from the File Full Path (variable)
#{User::FileNameWithoutExtension] = SUBSTRING (#[User::FullFilePath], LEN( #[User::FullFilePath] ) - FINDSTRING( REVERSE( #[User::FullFilePath] ), "\\", 1) + 2, LEN (RIGHT( #[User::FullFilePath], FINDSTRING( REVERSE( #[User::FullFilePath] ), "\\", 1 ) - 1 ) ) - FINDSTRING( REVERSE( #[User::FullFilePath] ), ".", 1 ) )
Add an Execute SQL Task inside the foreach loop container to check if the stored procedure is found in the database:
SELECT COUNT(*) FROM sys.objects WHERE type = 'P' AND name = ?)
Pass the procedure name as parameter to the execute sql task:
Passing Variables to and from an SSIS task
Store the count result inside a variable of type integer using Resultsets:
SSIS Basics: Using the Execute SQL Task to Generate Result Sets
Using precedence constraints with expressions add 2 paths from the execute sql task
the first using expression #[User::Count] == 0
the second using expression #[User::Count] > 0
Other references:
Working with Precedence Constraints in SQL Server Integration Services
Defining Workflow in SSIS using Precedence Constraints
On the second path add an Execute SQL Task to get the procedure definition using the same approach above:
SELECT OBJECT_DEFINITION (OBJECT_ID(N'<databasename>.<schemaname>.' + CAST(? as VARCHAR(100))));
And store the result inside a variable using a result set.
Add a Script Task to write the procedure definition into the destination file
On the first path add a File system task to move the file into the directory specified
Add another foreach loop to read new files and execute the content.
I am having the following problem with populating a dynamic file path for an XML file from a SSIS variable.
In Visual Studio 2017 I have an Execute SQL Task with a MS SQL stored procedure that returns two columns. The first column is a date (stored as a string) and the second is a URL. These two columns (single row) populate SSIS variables and are mapped on the Result Set tab of the Script Task. The next step is script task that uses the URL from the variable to download an xml file from a web service. The xml file is stored using a file Connection Manager. The connection string for the file is an expression that should be using the 1st variable (User::rateDate) from the Execute SQL Task
Connection String expression:
#[User::xmlFileLocation] + "ExchangeRates-" + #[User::rateDate] + ".xml"
This evaluates to
\server\ExchangeRates\ExchangeRates-.xml
XML file should be saved as ExchangeRates-2017-12-19.xml with 2017-12-19 being the result of the stored procedure, but instead the XML file is saved as ExchangeRates-.xml
If I manually populate the User::rateDate variable it will use that in the Connection String, but I haven't been able to get it to populate from the stored procedure result.
The date generated is part of the URL generation too so I want this both created in the same place, i.e. I don't want to assign the file name via some GETDATE() logic in the expression.
I have confirmed the variable is being populated is a Script Task C# pop up.
I have confirmed that it is not a date/string issue by changing the stored procedure result to an explicit string, like "test". It still doesn't get added to the Connection String.
Thanks, Tim
I will provide 2 solutions based on the Stored Procedure type:
Stored Procedure with a Select Statement
I will assume that you are using a Stored procedure that contains a SELECT statement that return a Table of 2 columns: ServerURL and rateDate
In this case you have to insert the result of this select statement into a temp table, then read from these temp table, as example:
CREATE TABLE #TBL(ServerURL varchar(4000), rateDate varchar(50))
INSERT INTO #TBL EXEC pr_rateDate
SELECT TOP 1 * FROM #TBL
In this way your variables mapping should work
Stored Procedure with Output Parameters
I will assume that you are using a stored procedure which require 2 output parameters to be passed, example:
EXEC sp_rateDate #ServerURL OUTPUT, #rateDate OUTPUT
So you have to use the following SQL statemment:
EXEC sp_rateDate ? OUTPUT, ? OUTPUT
and you have to add 2 output parameters in the Parameter Mapping tab
Useful Links
Parameters and Return Codes in the Execute SQL Task Read Using Parameters with Stored Procedures part
SQL Assist - SSIS Execute SQL Task
Map Result Sets to Variables in an Execute SQL Task
How to Execute Stored Procedure in SSIS Execute SQL Task in SSIS
SSIS Basics: Using the Execute SQL Task to Generate Result Sets
Change the datatype of variable User::rateDate to datetime. Then, change your connection string expression to the following:
#[User::xmlFileLocation] +
"ExchangeRates-" +
(DT_WSTR,4)DATEPART("yyyy",#[User::rateDate]) +
RIGHT("0" + (DT_WSTR,2)DATEPART("mm",#[User::rateDate]) ,2) +
RIGHT("0" + (DT_WSTR,2)DATEPART("dd",#[User::rateDate]),2) +
".xml"
I'm busy setting up an SSIS package that takes a flat file and loads its content into a staging table. From there it needs to take the file name of that file and save it into a field on another table, along with one of the lines of data. The loading of the data I do by using an execute SQL task that executes a stored procedure. This procedure takes the file name as input parameter (just so I can write it to that table)... I added a Script task before the execute SQL Task that does the following:
public void Main()
{
string inputFolder;
string fileName;
inputFolder = Dts.Variables["SourceFolder"].Value.ToString();
// MessageBox.Show(inputFolder);
string[] foundFiles = null;
foundFiles = System.IO.Directory.GetFiles(inputFolder);
if (foundFiles.Length > 0)
{
fileName = foundFiles[0];
Dts.Variables["ProcessedFileName"].Value = System.IO.Path.GetFileName(fileName);
// MessageBox.Show(fileName);
// MessageBox.Show(System.IO.Path.GetFileName(fileName));
}
Dts.TaskResult = (int)ScriptResults.Success;
}
The "ProcessedFileName" variable is a global variable in the SSIS package, as well as the "SourceFolder" variable.
The execute SQL task I set up using Direct input executing:
DECLARE #StoredProcedure varchar(100) = ?;
DECLARE #ProcessedFileName varchar(100) = ?;
EXEC ? ?;
The parameter mapping is to variables StoredProcedure and ProcessedFileName.
The problem is when I run this it tells me "No value given for one or more required parameters." Why could this be?
You cannot dynamically change a stored procedure name in an Execute SQL Task using parameters. That is what EXEC ? ?; is doing.
If you need to change the procedure name at run-time then you'll need to use string concatenation via expressions to build our your query.
Bigger picture, it seems that you are using a script task where a For Each (file) Enumerator task would suffice. There are reasons to use a script task over foreach (like if you need to process in order, etc) but it doesn't read like you're doing that.
Assorted answers that may be of help on how to accomplish expressions and foreach enumerator
SSIS: Get any flat file source from folder and cache the name as a super global variable
Passing a variable through DTEXEC with xp_cmdshell (SQL Server 2008)
Import most recent csv file to sql server in ssis
fetching data from multiple file and loading it into raw file destination(raw file should be dynamic) in ssis
How can I change the MySQL query delimiter using the C API? I tried sending DELIMITER | as a query, complained about ..the right syntax to use near 'delimiter' at line 1..
Tried DELIMITER |; too, no luck. Tried DELIMITER |; SELECT 1|, no luck. :(
The reason I ask is that I need to create a trigger through the C API such as the following:
create trigger increase_count after insert on torrent_clients for each row
begin
IF NEW.BytesLeft = 0 THEN
UPDATE torrents SET Seeders = Seeders + 1 WHERE torrents.InfoHash = NEW.InfoHash;
ELSE
UPDATE torrents SET Leechers = Leechers + 1 WHERE torrents.InfoHash = NEW.InfoHash;
END IF;
end|
but I don't think there is a way to do it without changing the delimiter, is there?
Thanks!
EDIT: Note that I do need to explicitly write the delimiter at the end, as I'm running multiple queries with only one API call (I have multi statements on)
EDIT2: the mysqlclient that uses the C api does this so there must be a way..
Changing the delimiter is only needed when using the mysql client program (because it is mysql that interpretes the semicolon as statement delimiter). You don't need to change the delimiter when using the C API:
Normally, you can execute only a single SQL command with mysql_query(). A semicolon can appear in such a command only when it is syntactically allowed, and that is normally not the case! In particular, SQL does not allow for a command to end in a semicolon. (The C API will raise an error if you attempt to do so.)
The commands CREATE PROCEDURE, CREATE FUNCTION, CREATE TRIGGER, and the like are exceptions when they define a stored procedure or trigger: In such commands the semicolon serves as a separator between the SQL instructions that are part of the stored procedure or trigger. Such commands can be executed without problem.
P.S. You probably should set multi statements off again.
You can execute multiple commands using mysql_query. You will have to set some parameters though while establishing the connection.
e.g.,
unsigned long opt_flags = CLIENT_FOUND_ROWS | CLIENT_MULTI_STATEMENTS | CLIENT_MULTI_RESULTS ;
if (0 == mysql_real_connect(mConn,mHostName,mUserName,mUserPass,mDbName,mPort,0,opt_flags))
...
/* execute multiple statements */
status = mysql_query(mysql,
"DROP TABLE IF EXISTS test_table;\
CREATE TABLE test_table(id INT);\
INSERT INTO test_table VALUES(10);\
UPDATE test_table SET id=20 WHERE id=10;\
SELECT * FROM test_table;\
DROP TABLE test_table");
Did you try
DELIMITER //
Without a ";" at the end? It might be not possible to change the delimiter within a query that has multiple queries in it.