while executing target table in snowflake using json data as source table
merge into cust tgt using (
select parse_json(s.$1):application_num as application num
from prd_json s qualify
row_number() over(partition application
order_by application desc)=1) src
on tgt.application =src.application
when not matched and op_type='I' then
insert(application) values (src.application );
qualify commands ignores all the duplicate data present and gives only unique record but while putting joins its show only less records when compare to normal select statement.
for example :
select distinct application
from prd_json where op_type='I';
--15000 rows are there
while putting joins it shows there is not matching records in target . if it is not matched it should insert all 15000rows but 8500 rows only inserting even though it was not an duplicate record . is there any function available without using "qualify" shall we insert the record. if i ignore qualify am getting dml error duplication. pls guide me if anyone knows.
How about using SELECT DISTINCT?
You demo SQL does not compile. and you using the $1 means it's also hard to guess the names of your columns to know how the ROW_NUMBER is working.
So it's hard to nail down the problem.
But with the following SQL you can replace ROW_NUMBER with DISTINCT
CREATE TABLE cust(application INT);
CREATE OR REPLACE table prd_json as
SELECT parse_json(column1) as application, column2 as op_type
FROM VALUES
('{"application_num":1,"other":1}', 'I'),
('{"application_num":1,"other":2}', 'I'),
('{"application_num":2,"other":3}', 'I'),
('{"application_num":1,"other":1}', 'U')
;
MERGE INTO cust AS tgt
USING (
SELECT DISTINCT
parse_json(s.$1):application_num::int as application,
s.op_type
FROM prd_json AS s
) AS src
ON tgt.application = src.application
WHEN NOT MATCHED AND src.op_type = 'I' THEN
INSERT(application) VALUES (src.application );
number of rows inserted
2
SELECT * FROM cust;
APPLICATION
1
2
running the MERGE code a second time gives:
number of rows inserted
0
Now if truncate CUST and I swap to using this SQL for the inner part:
SELECT --DISTINCT
parse_json(s.$1):application_num::int as application,
s.op_type
FROM prd_json AS s
qualify row_number() over (partition by application order by application desc)=1
I get three rows inserted, because the partition by application, is effectively binding to the s.application not the output application, and there are 3 different "applications" because of the other values.
The reason I wrote my code this way is your
select distinct application
from prd_json where op_type='I';
implies there is something called application already, in the table.. and thus it runs the chance of being used in the ROW_NUMBER statement..
Anyways, there is a large possible problem is you also have "update data" I guess U in your transaction block, that you want to ORDER BY the sub-select so you never have a Inser,Update trying action in Update,Inser order. And assuming you want all update operations if there are many of them.. I will stop. But if you do not have Updates, the sub-select should have the op_type='I' to avoid the non-insert ops making it. Out, or possible worse again, in your ROW_NUMBER pattern replacing the Intserts. Which I suspect is the underlying cause of your problem.
I'm creating a data validation procedure for a database, using various models, etc. in the database.
I've created a temp table with a model, a sequence, and 3 columns.
In each of these columns I have the qualified column name (table.column) to use in my query, or a null value. Here's the temp_table structure:
create table #temp_table(model nvarchar(50), seq nvarchar(50), col nvarchar(100), col2 nvarchar(100) , col3 nvarchar(100))
In my dynamic sql I have a join something like this (exteremely simplified):
select *
from
original_table
inner join
...
#temp_table
on
original_table.models = #temp_table.models
inner join
set_model
on
original_tables.models = set_model.models
and #temp_table.col = set_model.val
and #temp_table.col2 = set_model.val2
and #temp_table.col3 = set_model.val3
What I'm working on has many more tables (hence the ... in the middle of the query), so, we'll just assume that all the tables are present and all the columns are valid.
Because #temp_table.col stores a value, when being join to set_model.val the comparison will look something like 'Buildings.year_id' = 2014.
Is there a way to force my dynamic query to use the value in #temp_table.col as part of the join condition?
For example:
If in the query above #temp_table.col = 'Buildings.year_id'
how do I make the join evaluate Buildings.year_id = set_model.val
rather than 'Buildings.year_id' = 2014?
I was trying to create a query which had a different query plan based upon the row queried.
I found a workaround (creating a cursor and looping through n different tables and appending each dynamic query with a ' union '), but I came back and thought about the problem I ran into for a little while.
I realized that I was trying to dynamically create a query based upon data from the query I was using. As a result, no effective query plan could be created, as it would need to run a unique query based upon each row.
If the query had been able to work, it would've been extremely inefficient (as each row would make its own 'select' statement).
Regardless, the question itself was based on bad/incomplete logic.
I have a stored Procedure as:
CREATE PROCEDURE [dbo].[spGetEmployeesNotInSkill]
AS
BEGIN
SELECT COUNT (*) as Total FROM
(
SELECT tblUser.EmployeeID FROM tblUser where tblUser.FirstName <> 'guest'
EXCEPT
SELECT tblSkillMetrics.EmployeeID FROM tblSkillMetrics
) r -- why is 'r' used here?
END
What I want to know is why are we using this r? If we change the r to any other letter "a/b/c/...x/y/z" it gives correct output, but if we remove it shows error.
Can anyone please explain this to me?
Whenever you introduce a subquery, CTE, or anything else the will be providing rows as part of a query, you need to provide a name by which that particular set of rows may be referred to elsewhere in the query.
In the case of a table or view, the introduction of an alias is optional, and if omitted, the name of the table or view is used. But for anything else, the name must be explicitly provided.
E.g. you could have had:
SELECT COUNT (*) as Total FROM (
SELECT tblUser.EmployeeID FROM tblUser u where u.FirstName <> 'guest'
EXCEPT
SELECT tblSkillMetrics.EmployeeID FROM tblSkillMetrics)r
Where I've now introduced an alias(u) for tblUser in the inner query. And so in your query, r is the name/alias that's being used for the subquery as a whole.
The r is being used as a table alias. The query is saying "select the number of records in this table that I'm going to call r". So you can call it anything you like.
When you select from a virtual table like this you have to give it an alias - it might be more helpful to call it something like SkilledUsers but as it's only used in one place it's quite common only to use a single character.
It's a quirk of the SQL Server implementation of SQL. The "r" is an alias for the subquery: SQL Server requires that it must be named, even if it is not otherwise referenced. You could just as easily have named it FOO or SUBQUERY or EMPLOYEES.
Other implementations of SQL don't impose this syntactic restriction.
We are using r as an table alias.When u use select * from (select cn from table_name) r you are assigning name r to derived table (select cn from table_name).It will be useful when you are using joins.
I have the below query I am trying to return distinct value from the second .value method.
Here is what I have tried. I tried adding 'distinct-values(.)' to return only distinct but it is still returning the same results as a normal '.' How can I select distinct values from just one column?
;WITH XMLNAMESPACES (default 'http://www.w3.org/2001/XMLSchema')
SELECT
a.value('.', 'NVARCHAR(50)') AS Visitor
, b.value('distinct-values(.)', 'NVARCHAR(50)') AS Sender
FROM XmlTable AS X
CROSS APPLY xmlDocument.nodes('Root/Visitors/Visitor') AS aa(a)
CROSS APPLY xmlDocument.nodes('Root/Senders/Sender') AS bb(b)
Here is the normal result
Here is whay I am trying to get
Xml Like this
<upx:Root xmlns:upx="http://www.w3.org/2001/XMLSchema">
<upx:Visitors>
<upx:Visitor>Visitor1</upx:Visitor>
<upx:Visitor>Visitor2</upx:Visitor>
</upx:Visitors>
<upx:Senders>
<upx:Sender>Sender1</upx:Sender>
</upx:Senders>
</upx:Root>
It is your cross apply with your nodes statement listed twice that is showing this problem. Do what you are doing with the 'nodes' syntax with a 'query' extension instead followed up by a 'value' extension to show what is in the xml directly from extension instead of relying on the nodes with a cross apply. The problem is you are not displaying to the audience where you get that Id from? Are you determining that at run time from the xml itself or joining yet to another table or having another part of the xml not present? What in essence that is happening with the nodes is it is cross applying and saying: "I have two vales in that node heirarchy here they are." Then you are cross applying again a different node and it is returning the same thing twice. You must be careful when using cross apply twice exactly what it is doing. I can show the differentiation but without how I know you are relating back to 1 (are you just hunting for it somehow for the int after visitor?) I don't know how to represent exactly what you are wanting.
EDIT: Okay it is what I thought then. Now my code may be longer than some and I will admit there may be an easier way to do this however I would do three things:
Keep your cross apply with nodes because nodes is useful in that it will repeat rows you need to count on. However I would add an artificial flag for the name you use for the node. Then I would union together two select statements using the nodes.
I would then use a nested select as a from statement and then determine row number with a windowed function based on the flags I just set.
I would then nest that again and then use the very same row number as the Id of the row number and then I would do some syntactic pivoting based on a max(case when) based on the flags I arbitrarily set.
I usually prefer cte's but since your XML namespace has a 'with' beginning and the first cte does as well I forgot how the syntax is to work around that. Nested Selects IMHO can get hairy when there are multiple so I choose CTE's usually but in this case I did a nested select inside of another nested select. I hope this helps:
declare #xml xml = '<upx:Root xmlns:upx="http://www.w3.org/2001/XMLSchema">
<upx:Visitors>
<upx:Visitor>Visitor1</upx:Visitor>
<upx:Visitor>Visitor2</upx:Visitor>
</upx:Visitors>
<upx:Senders>
<upx:Sender>Sender1</upx:Sender>
</upx:Senders>
</upx:Root>'
;
declare #Xmltable table ( xmlDocument xml);
insert into #XmlTable values (#xml);
WITH XMLNAMESPACES (default 'http://www.w3.org/2001/XMLSchema')
select
pos as Id
, max(case when Listing = 'Visitors' then Value end) as Visitors
, max(case when Listing = 'Senders' then Value end) as Senders
from
(
select
*
, row_number() over(partition by Listing order by Value) as pos
from
(
SELECT
'Visitors' as Listing
, a.value('.', 'NVARCHAR(50)') AS Value
FROM #XmlTable AS X
CROSS APPLY xmlDocument.nodes('Root/Visitors/Visitor') AS aa(a)
union
SELECT
'Senders'
, b.value('distinct-values(.)', 'NVARCHAR(50)') AS Sender
FROM #XmlTable AS X
CROSS APPLY xmlDocument.nodes('Root/Senders/Sender') AS bb(b)
) as u
) as listing
group by pos
I am confused with the order of execution of this query, please explain me this.
I am confused with when the join is applied, function is called, a new column is added with the Case and when the serial number is added. Please explain the order of execution of all this.
select Row_number() OVER(ORDER BY (SELECT 1)) AS 'Serial Number',
EP.FirstName,Ep.LastName,[dbo].[GetBookingRoleName](ES.UserId,EP.BookingRole) as RoleName,
(select top 1 convert(varchar(10),eventDate,103)from [3rdi_EventDates] where EventId=13) as EventDate,
(CASE [dbo].[GetBookingRoleName](ES.UserId,EP.BookingRole)
WHEN '90 Day Client' THEN 'DC'
WHEN 'Association Client' THEN 'DC'
WHEN 'Autism Whisperer' THEN 'DC'
WHEN 'CampII' THEN 'AD'
WHEN 'Captain' THEN 'AD'
WHEN 'Chiropractic Assistant' THEN 'AD'
WHEN 'Coaches' THEN 'AD'
END) as Category from [3rdi_EventParticipants] as EP
inner join [3rdi_EventSignup] as ES on EP.SignUpId = ES.SignUpId
where EP.EventId = 13
and userid in (
select distinct userid from userroles
--where roleid not in(6,7,61,64) and roleid not in(1,2))
where roleid not in(19, 20, 21, 22) and roleid not in(1,2))
This is the function which is called from the above query.
CREATE function [dbo].[GetBookingRoleName]
(
#UserId as integer,
#BookingId as integer
)
RETURNS varchar(20)
as
begin
declare #RoleName varchar(20)
if #BookingId = -1
Select Top 1 #RoleName=R.RoleName From UserRoles UR inner join Roles R on UR.RoleId=R.RoleId Where UR.UserId=#UserId and R.RoleId not in(1,2)
else
Select #RoleName= RoleName From Roles where RoleId = #BookingId
return #RoleName
end
Queries are generally processed in the follow order (SQL Server). I have no idea if other RDBMS's do it this way.
FROM [MyTable]
ON [MyCondition]
JOIN [MyJoinedTable]
WHERE [...]
GROUP BY [...]
HAVING [...]
SELECT [...]
ORDER BY [...]
SQL is a declarative language. The result of a query must be what you would get if you evaluated as follows (from Microsoft):
Logical Processing Order of the SELECT statement
The following steps show the logical
processing order, or binding order,
for a SELECT statement. This order
determines when the objects defined in
one step are made available to the
clauses in subsequent steps. For
example, if the query processor can
bind to (access) the tables or views
defined in the FROM clause, these
objects and their columns are made
available to all subsequent steps.
Conversely, because the SELECT clause
is step 8, any column aliases or
derived columns defined in that clause
cannot be referenced by preceding
clauses. However, they can be
referenced by subsequent clauses such
as the ORDER BY clause. Note that the
actual physical execution of the
statement is determined by the query
processor and the order may vary from
this list.
FROM
ON
JOIN
WHERE
GROUP BY
WITH CUBE or WITH ROLLUP
HAVING
SELECT
DISTINCT
ORDER BY
TOP
The optimizer is free to choose any order it feels appropriate to produce the best execution time. Given any SQL query, is basically impossible to anybody to pretend it knows the execution order. If you add detailed information about the schema involved (exact tables and indexes definition) and the estimated cardinalities (size of data and selectivity of keys) then one can take a guess at the probable execution order.
Ultimately, the only correct 'order' is the one described ion the actual execution plan. See Displaying Execution Plans by Using SQL Server Profiler Event Classes and Displaying Graphical Execution Plans (SQL Server Management Studio).
A completely different thing though is how do queries, subqueries and expressions project themselves into 'validity'. For instance if you have an aliased expression in the SELECT projection list, can you use the alias in the WHERE clause? Like this:
SELECT a+b as c
FROM t
WHERE c=...;
Is the use of c alias valid in the where clause? The answer is NO. Queries form a syntax tree, and a lower branch of the tree cannot be reference something defined higher in the tree. This is not necessarily an order of 'execution', is more of a syntax parsing issue. It is equivalent to writing this code in C#:
void Select (int a, int b)
{
if (c = ...) then {...}
int c = a+b;
}
Just as in C# this code won't compile because the variable c is used before is defined, the SELECT above won't compile properly because the alias c is referenced lower in the tree than is actually defined.
Unfortunately, unlike the well known rules of C/C# language parsing, the SQL rules of how the query tree is built are somehow esoteric. There is a brief mention of them in Single SQL Statement Processing but a detailed discussion of how they are created, and what order is valid and what not, I don't know of any source. I'm not saying there aren't good sources, I'm sure some of the good SQL books out there cover this topic.
Note that the syntax tree order does not match the visual order of the SQL text. For example the ORDER BY clause is usually the last in the SQL text, but as a syntax tree it sits above everything else (it sorts the output of the SELECT, so it sits above the SELECTed columns so to speak) and as such is is valid to reference the c alias:
SELECT a+b as c
FROM t
ORDER BY c;
SQL query is not imperative but declarative, so you have no idea which the statement is executed first, but since SQL is evaluated by SQL query engines, most of the SQL engines follows similar process to obtain the results. You may have to understand how the query engine works internally to understand some SQL execution behavior.
Julia Evens has a great post explaining this, it is worth to check it out:
https://jvns.ca/blog/2019/10/03/sql-queries-don-t-start-with-select/
SQL is a declarative language, meaning that it tells the SQL engine what to do, not how. This is in contrast to an imperative language such as C, in which how to do something is clearly laid out.
This means that not all statements will execute as expected. Of particular note are boolean expressions, which may not evaluate from left-to-right as written. For example, the following code is not guaranteed to execute without a divide by zero error:
SELECT 'null' WHERE 1 = 1 OR 1 / 0 = 0
The reason for this is the query optimizer chooses the best (most efficient) way to execute a statement. This means that, for example, a value may be loaded and filtered before a transforming predicate is applied, causing an error. See the second link above for an example
See: here and here.
"Order of execution" is probably a bad mental model for SQL queries. Its hard to actually write a single query that would actually depend on order of execution (this is a good thing). Instead you should think of all join and where clauses happening simultaneously (almost like a template)
That said you could run display the Execution Plans which should give you insight into it.
However since its's not clear why you want to know the order of execution, I'm guessing your trying to get a mental model for this query so you can fix it in some way. This is how I would "translate" your query, although I've done well with this kind of analysis there's some grey area with how precise it is.
FROM AND WHERE CLAUSE
Give me all the Event Participants rows. from [3rdi_EventParticipants
Also give me all the Event Signup rows that match the Event Participants rows on SignUpID inner join 3rdi_EventSignup] as ES on EP.SignUpId = ES.SignUpId
But Only for Event 13 EP.EventId = 13
And only if the user id has a record in the user roles table where the role id is not in 1,2,19,20,21,22
userid in (
select distinct userid from userroles
--where roleid not in(6,7,61,64) and roleid not in(1,2))
where roleid not in(19, 20, 21, 22) and roleid not in(1,2))
SELECT CLAUSE
For each of the rows give me a unique ID
Row_number() OVER(ORDER BY (SELECT 1)) AS 'Serial Number',
The participants First Name EP.FirstName
The participants Last Name Ep.LastName
The Booking Role name GetBookingRoleName
Go look in the Event Dates and find out what the first eventDate where the EventId = 13 that you find
(select top 1 convert(varchar(10),eventDate,103)from [3rdi_EventDates] where EventId=13) as EventDate
Finally translate the GetBookingRoleName in Category. I don't have a table for this so I'll map it manually (CASE [dbo].[GetBookingRoleName](ES.UserId,EP.BookingRole)
WHEN '90 Day Client' THEN 'DC'
WHEN 'Association Client' THEN 'DC'
WHEN 'Autism Whisperer' THEN 'DC'
WHEN 'CampII' THEN 'AD'
WHEN 'Captain' THEN 'AD'
WHEN 'Chiropractic Assistant' THEN 'AD'
WHEN 'Coaches' THEN 'AD'
END) as Category
So a couple of notes here. You're not ordering by anything when you select TOP. You should probably have na order by there. You could also just as easily put that in your from clause e.g.
from [3rdi_EventParticipants] as EP
inner join [3rdi_EventSignup] as ES on EP.SignUpId = ES.SignUpId,
(select top 1 convert(varchar(10),eventDate,103)
from [3rdi_EventDates] where EventId=13
Order by eventDate) dates
There is a logical order to evaluation of the query text, but the database engine can choose what order execute the query components based upon what is most optimal. The logical text parsing ordering is listed below. That is, for example, why you can't use an alias from SELECT clause in a WHERE clause. As far as the query parsing process is concerned, the alias doesn't exist yet.
FROM
ON
OUTER
WHERE
GROUP BY
CUBE | ROLLUP
HAVING
SELECT
DISTINCT
ORDER BY
TOP
See the Microsoft documentation (see "Logical Processing Order of the SELECT statement") for more information on this.
Simplified order for T-SQL -> SELECT statement:
1) FROM
2) Cartesian product
3) ON
4) Outer rows
5) WHERE
6) GROUP BY
7) HAVING
8) SELECT
9) Evaluation phase in SELECT
10) DISTINCT
11) ORDER BY
12) TOP
as I had done so far - same order was applicable in SQLite.
Source => SELECT (Transact-SQL)
... of course there are (rare) exceptions.