I am creating a BQ Stored Procedure to truncate all the tables in a dataset. I have a 2 step process. Step 1 identifies all the matching tables. Step 2 is expected to iterate thru each table and truncate.
I have the following code to achieve this:
for record in
( select TABLE_NAME
from <staging_dataset>.INFORMATION_SCHEMA.TABLES
)
DO
execute immediate
"truncate table #tab" using record.TABLE_NAME as tab;
END FOR;
The Error that I am running into is in the Execute Immediate piece.
Invalid EXECUTE IMMEDIATE sql string `truncate table #tab`, Syntax error: Unexpected "#" at [8:3]
I tried replacing the #tab with ? placeholder and see a similar error. What am I doing wrong?
Is there another way to achieve the same result?
Strange; it seems DML doesn't work with USING. Queries work fine.
Try using CONCAT to build your dynamic query string:
CONCAT("truncate table ", record.TABLE_NAME);
Related
We have a tracking table used to monitor the runtime of processes. It captures data from other sources beyond Snowflake so we can't just use the history data that is automatically captured when queries run. Our idea is to have a pre statement that captures the start time as a variable, a snowflake query would then run (could be anything), and lastly a post statement which outputs the required tracking data into a different table. However, when we try the below statement or variations of it we get an error.
Error:
SQL compilation error: syntax error line 3 at position 0 unexpected 'SELECT'. syntax error line 8 at position 0 unexpected 'INSERT'.
Statement Example:
-- Pre Statement: setting the start time of the query
SET(starttime)=current_timestamp()
-- Intra Statement: query to run in Snowflake
SELECT *
FROM "Customer"
-- Post Statement: inserting output into tracking table
INSERT INTO "Tracking"
SELECT 'Example', current_timestamp(), current_warehouse(), DATEDIFF(hour,$starttime,current_timestamp()), current_user();
Tracking Table Fields:
PROCESS_TAG, RUN_DATE, WAREHOUSE, RUN_TIME, ACCOUNT
Thank you #Sergiu and #Greg Pavlik for finding my mistake of not including ";" after each statement. The query runs as expected now.
-- Pre Statement: setting the start time of the query
SET(starttime)=current_timestamp();
-- Intra Statement: query to run in Snowflake
SELECT *
FROM "Customer";
-- Post Statement: inserting output into tracking table
INSERT INTO "Tracking"
SELECT 'Example', current_timestamp(), current_warehouse(), DATEDIFF(hour,$starttime,current_timestamp()), current_user();
the copy into command returns an output dataset.
CTAS can create a table from the results of a query.
combining the two, we would expect to get the list of loaded files into a new table.
CREATE TABLE MY_LOADED_FILES
AS
COPY INTO mytable
FROM #my_int_stage;
However, this returns:
SQL compilation error: syntax error line 3 at position 0 unexpected 'copy'.
What am I doing wrong?
It doesn't look like you can put a COPY INTO statement inside another statement unfortunately. There is a way to do this however by using the result_scan function to return the results of the previous query.
copy into test_database.public.test_table from #my_int_stage;
create temporary table test_database.public.test_table_results as (
select * from table(result_scan(LAST_QUERY_ID()))
);
Of course you need to make sure the second query runs in the same session as the copy statement and also that it is run directly after the copy statement. Alternatively you can use the query id with the result_scan.
If you want to see which files were loaded why don't you just look at the copy_history of the table?
I am creating a SSIS package and trying to extract data by calling stored procedures from one database and inserting the result set values into another table of different database. I have created a Execute SQL task to extract the data, a for each loop container to loop through the result set and Execute SQL task within the for loop container to insert the result set data into another database table. I am getting an the following error while inserting the records. I guess its the issue with the mapping.
[Execute SQL Task] Error: Executing the query "insert into EmployeeCount (companyId..." failed with the following error: "Parameter name is unrecognized.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
Following the screenshot of the template design
Following is the edit window of execute sql task which in inside the foreach container
The insert statement
insert into EmployeeCount (companyId,dataItemName,dataItemvalue,fiscalYear,fiscalQuarter,PeriodTypeId) values(companyId,dataItemName,dataItemvalue,fiscalYear,fiscalQuarter,PeriodTypeId)
You will have to set "Parameter Name" in chronological order.
i.e companyID parameter must be 0, dataItemvalue to 1 ....PeriodTypeId to 5
Sample :
In the 'Parameter Mapping' for Parameter Name, instead of '?' give the values in a 0 based index i.e. 0,1,2,... till the end.
As an option:
You may be trying to insert too long values into shorter columns
For example if you create Source DB you need to be sure that each column have sufficient maximum length and you don't put largest values (too long values) into this column.
So unfortunately SSIS don't specify the problematic column, and you need to find it by yourself or enlarge maximum length of every column.
Simple Workaround
You can achieve this without using parameters, you can use expressions:
Double click on the Execute SQL Task, Go To Expressions.
add the following expression to SqlStatementSource:
"insert into EmployeeCount (companyId,dataItemName,dataItemvalue,fiscalYear,fiscalQuarter,PeriodTypeId)
values(" + (DT_WSTR,50)#[User::companyId] + ",'" + #[User::dataItemName] + "'," + (DT_WSTR,50)#[User::dataItemvalue] + "," + (DT_WSTR,50)#[User::fiscalYear] + "," + (DT_WSTR,50)#[User::fiscalQuarter] + "," + (DT_WSTR,50)#[User::PeriodTypeId] + ")"
Be aware! This approach makes your solution vulnerable to SQL injections
I have a table that contains just under a million rows. I'm building a form using SSIS that asks for user input and uses the values as parameters to build a view from source data. I'm having trouble getting SSIS to create the view from a variable.
The purpose of this 'tool' is to provide a dialogue that programmatically builds a view and later an update statement based upon parameters defined via a form that will execute an SSIS package. A number of the ppl on my team know 0 SQL. Therefore this circumvents any SQL knowledge. Creating an entirely standalone app is not ideal as it would require too much additional overhead on my side and would deviate from a number of our existing processes that currently use SSIS/SQL to achieve similar results.
With that here is what I've tried/trying.
I have an SSIS package that contains 'Execute SQL Task'
This task brings up a form with 5 inputs (variables)
var1,var2,var3,var4,var5.
some vars are strings others are doubles, ints etc... (they all vary)
You populate the fields and hit okay.
These variables are passed to an 'Execute Package Task'.
Inside this package (Package B)
the vars are used in an 'Execute SQL Task'.
This task is attempting to take the users input and create a view with a where clause containing 4 other variables.
example:
Create View ? AS Select col1,col2,col3,col4 WHERE
col1 = ?
AND col2 =?
AND col3 =?.........
First it appears that using ? in the create view is invalid.
The error being:
Error: 0x0 at Build_Query: Incorrect syntax near '#P1'.
Error: 0xC002F210 at Build_Query, Execute SQL Task: Executing the query "CREATE VIEW ? as Select * from S_t_equip_template
..." failed with the following error: "The batch could not be analyzed because of compile errors.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established
correctly.
Task failed: Build_Query
If I use the create view variable as an expression and remove the variables/paramerters for the where statements, I can create the view no problem.
However the where statements throw errors once I add them back in. I've tried evaluating these as an expression in the 'Execute SQL Task' but as these are of various types I get the error:
[Execute SQL Task] Error: Executing the query "
CREATE VIEW testing AS SELECT
P.label,P.uniq..." failed with the following error: "The metadata could not
be determined because statement 'CREATE VIEW testingagain AS SELECT
P.label,P.uniqueid,C.label as Child_Label,C.uniqueid as Child_uni' does not
support metadata discovery.". Possible failure reasons: Problems with the
query, "ResultSet" property not set correctly, parameters not set correctly,
or connection not established correctly.
No idea what is going on. Any help would be appreciated
I've googled the error and found some info but the other use-cases are so different that it's difficult to understand the actual cause, or another work around.
AS requested (simplified example):
I've created a package variable (datatype string) called: View_Name
Execute SQL Task:
CREATE VIEW #[User::View_Name] AS SELECT
* from table1
where col1 = 100;
Specifically it does not like that I use a variable here.
If I set the View name everything works until: I move on to my Where clauses that contain variables.
Create a variable called type (datatype int)
I map the variable/parameter in my sql task
Example:
CREATE VIEW tempTable AS SELECT
* from table1
where col1 = ?;
This won't work, same error.
If i attempt to do the above via an expression or expressions I get the following error
[Execute SQL Task] Error: Executing the query "CREATE VIEW test_45678 AS SELECT P.label,P.uniquei..." failed with the following error: "Must declare the scalar variable "#".". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
Generally having to due with the variables cannot be evaluated this way. I'm guessing I'd need to evaluate each piece individually and build the expression piece by piece. That's fine but very inefficient and not maintainable.
I had some fundamental misunderstanding in my attempt to evaluate my expressions. Specifically syntax issues and having to cast each variable to a string.
My SQLstatement variable
ON C.pid = P.ID
where
C.width >="+(DT_WSTR, 8)#[User::Width] +"-"+ (DT_WSTR, 8)#[User::Range]+........
The final expression looks like so:
"Create View "+#[User::View_Name] + " AS SELECT " + #[User::SQLStatement]
I encounter some strange behavior with a dynamic SQL Query.
In a stored procedure I construct an insert query string out of multiple Strings. I execute the insert query in the SP like that - due to single nvarchar length restrictions.
EXEC(#QuerySelectPT+#QueryFromPT+#QueryFromPT)
If I print each part of the query, put these parts together and execute them manually in Management Studio the query works fine and inserts the data. But, if i execute the query in the EXEC() Method in the stored procedure, I get a
Column name or number of supplied values does not match table definition.
Error Message.
Did multiple check on the amount, spelling of columns in my query and in my insert table, but I have not found any differences so far.
Any advices?
Your count of columns for insert are different from count of columns for select. Print the statement before exec and find the error.
It as shot in the dark but seen you are telling the queries are valid and if you build the final query manually and it is working, the issue could be caused by string truncation.
Could you try:
EXEC(CAST(#QuerySelectPT AS VARCHAR(MAX))+#QueryFromPT+#QueryFromPT);
Also, as the Management Studio's message tab and selects are limited to 4000 symbols I think, you can test if the whole query is assembled correctly like this:
SELECT CAST(#QuerySelectPT+#QueryFromPT+#QueryFromPT AS XML)