When you define a SQL definition for a snowflake object (i.e. VIEW, TASK ... etc/) you can get it using metadata definition (ex : SHOW VIEWS -> text) or using GET_DDL function.
In the Snowflake WEBui you can then select the output which display the original indentation of SQL code which makes it more readable.
I want to bulk recreate objects in Snowflake and I would like to keep this indentation in object's SQL definition.
Unfortunately when I bulk-generate my queries in Snowflake, the SQL definition stands as a row with SQL definition flattened/not indented.
Example :
CREATE VIEW my_view AS
SELECT
'Hello' AS col_1,
'World' AS col_2,
'!' AS col_3
;
SHOW VIEWS LIKE 'my_view';
SELECT "text" FROM table(result_scan(last_query_id()));
-> Copy paste result get me the following query instead of indented one
CREATE VIEW my_view AS SELECT 'Hello' AS col_1, 'World' AS col_2, '!' AS col_3;
Unfortunately when I bulk-generate my queries in Snowflake, the SQL definition stands as a row with SQL definition flattened/not indented
The carriage return characters are still present in resultset, but the UI rows have constant height.
There is also download button to get it as "CSV".
WebUI is not the only way of interacting with Snowflake. Here the same query using DBeaver:
Actually a long time after the best solution for me to keep indentation in bulk query generation in Snowflake UI was to replace \n by \r\n.
Like this it keeps me the original indentation when i copy / paste my query in UI.
Related
I'm testing out a trial version of Snowflake. I created a table and want to load a local CSV called "food" but I don't see any "load" data option as shown in tutorial videos.
What am I missing? Do I need to use a PUT command somewhere?
Don't think Snowsight has that option in the UI. It's available in the classic UI though. Go to Databases tab, select a database. Go to Tables tab and select a table the option will be at the top
If the classic UI is limiting you or you are already using Snowsight and don't want to switch back, then here is another way to upload a CSV file.
A preliminary is that you have installed SnowSQL on your device (https://docs.snowflake.com/en/user-guide/snowsql-install-config.html).
Start SnowSQL and perform the following steps:
Use the database where to upload the file to. You need various privileges for creating a stage, a fileformat, and a table. E.g. USE MY_TEST_DB;
Create the fileformat you want to use for uploading your CSV file. E.g.
CREATE FILE FORMAT "MY_TEST_DB"."PUBLIC".MY_FILE_FORMAT TYPE = 'CSV';
If you don't configure the RECORD_DELIMITER, the FIELD_DELIMITER, and other stuff, Snowflake uses some defaults. I suggest you have a look at https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html. Some of the auto detection stuff can make your life hard and sometimes it is better to disable it.
Create a stage using the previously created fileformat
CREATE STAGE MY_STAGE file_format = "MY_TEST_DB"."PUBLIC".MY_FILE_FORMAT;
Now you can put your file to this stage
PUT file://<file_path>/file.csv #MY_STAGE;
You can find documentation for configuring the stage at https://docs.snowflake.com/en/sql-reference/sql/create-stage.html
You can check the upload with
SELECT d.$1, ..., d.$N FROM #MY_STAGE/file.csv d;
Then, create your table.
CREATE TABLE MY_TABLE (col1 varchar, ..., colN varchar);
Personally, I prefer creating first a table with only varchar columns and then create a view or a table with the final types. I love the try_to_* functions in snowflake (e.g. https://docs.snowflake.com/en/sql-reference/functions/try_to_decimal.html).
Then, copy the content from your stage to your table. If you want to transform your data at this point, you have to use an inner select. If not then the following command is enough.
COPY INTO mycsvtable from #MY_STAGE/file.csv;
I suggest doing this without the inner SELECT because then the option ERROR_ON_COLUMN_COUNT_MISMATCH works.
Be aware that the schema of the table must match the format. As mentioned above, if you go with all columns as varchars first and then transform the columns of interest in a second step, you should be fine.
You can find documentation for copying the staged file into a table at https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html
If you can check the dropped lines as follows:
SELECT error, line, character, rejected_record FROM table(validate("MY_TEST_DB"."MY_SCHEMA"."MY_CSV_TABLE", job_id=>'xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'))
Details can be found at https://docs.snowflake.com/en/sql-reference/functions/validate.html.
If you want to add those lines to your success table you can copy the the dropped lines to a new table and transform the data until the schema matches with the schema of the success table. Then, you can UNION both tables.
You see that it is pretty much to do for loading a simple CSV file to Snowflake. It becomes even more complicated when you take into account that every step can cause some specific failures and that your file might contain erroneous lines. This is why my team and I are working at Datameer to make these types of tasks easier. We aim for a simple drag and drop solution that does most of the work for you. We would be happy if you would try it out here: https://www.datameer.com/upload-csv-to-snowflake/
I have a script that iterates over a CTE and executes a custom sp_executesql query for each line. I need to use the resulting table in an SSRS report, but datasets don't seem to allow for declare, cursor and exec statements.
Is there any way to make T-SQL work in SSRS ?
Yes, I almost always use an entire script in my datasets. It is only the query designer that does not support them.
Simply write your script in SSMS or whatever then paste it into the query editor. Remember to comment out any variable declarations that will be passed in from the report as parameters.
If you double-click the dataset name and make sure the query type is Text then you can copy the script directly in.
Notes:
DO NOT DECLARE any variables that need to be passed in from your report as parameters. I generally just comment these out in the dataset query.
e.g. If you were passing in an Employee ID from you report parameter then your dataset would NOT need that declaring so it would look something like this
SELECT * FROM myTable WHERE EmpID = #EmpID
Obvisouly you would need this declaring when you test the script in SSMS
Make sure all references to #Variables are spelled with the same case. SSRS is case sensitive when it comes to variable names.
Only the first result set your script outputs will be 'seen' my SSRS so make sure you only have one SELECT statement that outputs the final data.
T-SQL works fine in SSRS datasets. I regularly declare variables, use temp tables and other T-SQL statements, and have used cursor and exec statements in the past.
The problem is likely in what you are returning. The "Query" (entire SQL statement) should return only one table which should have a stable structure. Returning a row per iteration of a cursor, for example, will not work. Put them in a temp table and then return all at once after the cursor execution. You can't return different column types or names for different executions, either.
Also, don't declare the variables that you use as parameters.
This seems ridiculously easy, but I can't find it anywhere...
I have a VERY simple sequence container with two tasks: Truncate a SQL table, and repopulate it from production. But this container will be repeated for about 50 tables. The container's name (entered manually) = the name of both the source and destination tables.
I have two variables:
"TableName" is entered manually.
"DelTable" is an expression that uses #[User::TableName] to generate a simple SQL statement.
I'm super-lazy and would like to use an expression to set "TableName" = the name of the current scope so I only have to enter it once.
Ideas???
THANK YOU!
if you are truncating all tables in a DB and replacing with exactly the same structure, how about this approach:
Execute SQL:
select table_name
from INFORMATION_SCHEMA.TABLES --Add a where to limit the tables to the ones you want
Save results to an object variable called TABLES
Add a for each loop:
Loop through ADO Object setting value to a string variable called table
Add Execute SQL to FE LOOP: truncate table ? and map parameter.
Add a 2nd Execute SQL statement:
INSERT INTO SERVER.DB.SCHEMA.?
select * from ?
Again map the parameters.
If you are having trouble mapping parameters set up variables and use them to create the SQL statements to run.
#TomPhillips is correct, I cannot unfortunately comment or make that answer useful. hence commenting here.
There's no easy quick fix to use a loop/automate unless all the 50 tables are same structure which is rare by any stretch of imagination.
BIML is the way to go if you are lazy :)
SSIS is not dynamic. Data Flows require fixed input and output at compile time, not runtime. You cannot simply change the table name and have it work.
If you have a list of 50 tables to do the same function on, you can use BIML to dynamically generate the SSIS package(s). But the DF itself cannot be dynamic.
I am trying to see if there is an easy answer for this. I have done something similar using multiple pick dropdown parameters in SSRS but this appears to be different.
My scenario is this, so maybe there is an even better answer.
I have a production server that I do not want to make any changes to including temp tables or functions. The production server has a table of clients with about 1600 records. I have set up an SSIS package that will allow transfer of data from production to dev based on a clientid. So my sources would have a query similar to Select Field From Table Where ClientId = ?
This works fine. Now I want to load more than one client, based an data in the clients table. It may be Select ClientId From Clients where Field = A and returns multiple ClientIds.
I am able to populate a comma delimited list from an execute sql task to a SSIS variable, so it maybe 1,4,8.
If I change my source query to use ClientId in (?) I get a conversion error.
I have looked at many posts that advocate a temp table or a function which I want to avoid. Select IN using varchar string with comma delimited values
I have contemplated building the entire sql statement into a variable but this don't seem like the right path as I have many tables to query and transfer where using ClientId = ? works well without having to build each individual SQL statement to a variable.
Is there an easy fix I am missing? I will turn my research now to try to find out how I did this in SSRS but I thought that I should try a post here to see if someone has accomplished this before.
I appreciate any info on this, thank you.
EDIT: Key note is that the column on clients is on the dev server, so I cannot just use a select in the where clause as the column does not exist on the production server.
EDIT: I did not mention that I am specifically looking at OLEDB sources mapping a parameter to ? in the sql statement.
EDIT: Narrowing down on this but having trouble relating SSRS and SSIS functionality. In SSRS its called a multi-value parameter in the following link the key line is
WHERE Production.ProductInventory.ProductID IN (#ProductID)
https://msdn.microsoft.com/en-us/library/dn385719(v=sql.110).aspx
This one looks good as well
https://sqlblogcasts.com/blogs/simons/archive/2007/11/22/RS-HowTo---Pass-a-multivalue-parameter-to-a-query-using-IN.aspx
I will keep researching and thank you for the help so far.
I think this sums it up best
This functionality is limited to strictly using embedded SQL.
What SSRS does is transform your SQL column IN (#value) to column IN
(#selectedvalue1,#selectedvalue2) etc.
You need to forget anything you have about the other ways of passing
lists to SQL i.e. building strings etc. and make sure you declare the
data types are correct for the value of your parameter.
You do not need to use the Join(parameters!,",") trick UNLESS
you are passing the list to a stored procedure.
In which case you then need to use some function to turn the delimited
list into a rowset as you have done.
I hope that helps
The core question is if I can get the same functionality in SSIS as in SSRS. It reminds me of macro substitution..
If you dont want to create a function, you can use the following in your t-sql statement.
Declare #ClientIds nvarchar(50) = '123,456'; --<-- Comma delimited list of Client Ids
Select Field
From Table
Where ClientId IN (
SELECT CAST(RTRIM(LTRIM(Split.a.value('.', 'VARCHAR(100)'))) AS INT) ClientIDs
FROM (
SELECT Cast ('<X>'
+ Replace(#ClientIds, ',', '</X><X>')
+ '</X>' AS XML) AS Data
) AS t CROSS APPLY Data.nodes ('/X') AS Split(a)
)
I need to do join query to MS SQL Server 2014 DB based on a column name value. The same query runs when doing query directly to DB, but when doing query through Mule I'm getting error. The query looks something like this :
SELECT * FROM sch.emple JOIN sch.dept on sch.emple.empid = sch.dept.empid;
The above query work fine while doing query directly to MS SQL Server DB, but gives the following error through mulesoft.
Record cannot be mapped as it contains multiple columns with the same label. Define column aliases to solve this problem (java.lang.IllegalArgumentException). Message payload is of type: String
Request you to please help me out.
Specify columns list directly:
SELECT e.<col1>, e.<col2>, ...., d.<col1>,...
FROM sch.emple AS e
JOIN sch.dept AS d
ON e.empid = d.empid;
Remarks:
You could use aliases instead of schema.table_name
SELECT * in production code in 95% cases is bad practice
The column that has duplicate is empid(or more). You could add alias for it e.empid AS emple_empid and d.empid AS dept_empid or just specify e.empid once.
To avoid specifying all columns manually, you could drag and drop them from object explorer to query pane like Drag and Drop Column List into query window.
Second way is to use plugin like Redgate Prompt to expand SELECT *:
Image from: https://www.simple-talk.com/sql/sql-tools/sql-server-intellisense-vs.-red-gate-sql-prompt/
Addendum
But the same query works directly.
It works because you don't bind them. Please read carefully link I provided for SELECT * antipattern and especially:
Binding Problems
When you SELECT *, it's possible to retrieve two columns of the same name from two different tables. This can
often crash your data consumer. Imagine a query that joins two
tables, both of which contain a column called "ID". How would a
consumer know which was which? SELECT * can also confuse views (at
least in some versions SQL Server) when underlying table structures
change -- the view is not rebuilt, and the data which comes back can
be nonsense. And the worst part of it is that you can take care
to name your columns whatever you want, but the next guy who comes
along might have no way of knowing that he has to worry about adding a
column which will collide with your already-developed names.
But the same query works directly.
by Dave Markle