I have a query which is equi-joining two tables, TableA and TableB using a nested loop. Because of the "equi"-join contraint, all rows returned in the result will therefore correspond to at least one row from each of these two tables. However, according to the plan (EXPLAIN ANALYZE) the actual rows count is 0 from TableB, even though a row is returned in the final result. How can the actual rows count equal zero here?
Here is the execution plan:
=> explain analyze select p.id, p.title, s.count from products p, stock s where p.id = s.p_id and s.w_id = 6 and p.type = 9 and s.count > 0 order by p.title;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------
Sort (cost=42.42..42.42 rows=2 width=36) (actual time=0.198..0.199 rows=1 loops=1)
Sort Key: p.title
Sort Method: quicksort Memory: 25kB
-> Nested Loop (cost=0.00..42.41 rows=2 width=36) (actual time=0.170..0.181 rows=1 loops=1)
-> Seq Scan on products p (cost=0.00..9.25 rows=4 width=32) (actual time=0.068..0.106 rows=4 loops=1)
Filter: (type = 9)
-> Index Scan using stock_pk on stock s (cost=0.00..8.28 rows=1 width=8) (actual time=0.015..0.015 rows=0 loops=4)
Index Cond: ((w_id = 6) AND (p_id = p.id))
Filter: (count > 0)
Total runtime: 0.290 ms
And the two table definitions... The products table first:
=> \d products
Table "public.products"
Column | Type | Modifiers
--------+------------------------+-----------
id | integer | not null
title | character varying(100) |
type | integer |
price | double precision |
filler | character(500) |
Indexes:
"products_pkey" PRIMARY KEY, btree (id)
"products_type_idx" btree (type)
Referenced by:
TABLE "orderline" CONSTRAINT "orderline_p_id_fkey" FOREIGN KEY (p_id) REFERENCES products(id)
TABLE "stock" CONSTRAINT "stock_p_id_fkey" FOREIGN KEY (p_id) REFERENCES products(id)
The stock table:
=> \d stock
Table "public.stock"
Column | Type | Modifiers
--------+---------+-----------
w_id | integer | not null
p_id | integer | not null
count | integer |
Indexes:
"stock_pk" PRIMARY KEY, btree (w_id, p_id)
"stock_p_id_idx" btree (p_id)
Foreign-key constraints:
"stock_p_id_fkey" FOREIGN KEY (p_id) REFERENCES products(id)
"stock_w_id_fkey" FOREIGN KEY (w_id) REFERENCES warehouses(id)
The actual rows of the inner index scan is the average number of rows returned in each call of it.
Looking at http://www.postgresql.org/docs/current/static/using-explain.html:
In some query plans, it is possible for a subplan node to be executed more than once. For example, the inner index scan is executed once per outer row in the above nested-loop plan. In such cases, the loops value reports the total number of executions of the node, and the actual time and rows values shown are averages per-execution. This is done to make the numbers comparable with the way that the cost estimates are shown. Multiply by the loops value to get the total time actually spent in the node.
I'm not sure how it's rounded (I'm guessing down to the nearest int, after averaging), but it might be that most rows in products don't have a corresponding row in stock.
Related
This is my (extremely simplified) product table and some test data.
drop table if exists product cascade;
create table product (
product_id integer not null,
reference varchar,
price decimal(13,4),
primary key (product_id)
);
insert into product (product_id, reference, price) values
(1001, 'MX-232', 100.00),
(1011, 'AX-232', 20.00),
(1003, 'KKK 11', 11.00),
(1004, 'OXS SUPER', 0.35),
(1005, 'ROR-MOT', 200.00),
(1006, '234PPP', 30.50),
(1007, 'T555-NS', 110.25),
(1008, 'LM234-XS', 101.20),
(1009, 'MOTOR-22', 12.50),
(1010, 'MOTOR-11', 30.00),
(1002, 'XUL-XUL1', 40.00);
I real life, listing product columns is a taught task, full of joins, case-when-end clauses, etc. On the other hand, there is a large number of queries to be fulfilled, as products by brand, featured products, products by title, by tags, by range or price, etc.
I don't want to repeat and maintain the complex product column listings every time I perform a query so, my current approach is breaking query processes in two tasks:
encapsulate the query in functions of type select_products_by_xxx(), that return product_id arrays, properly selected and ordered.
encapsulate all the product column complexity in a unique function list_products() that takes a product_id array as a parameter.
execute select * from list_products(select_products_by_xxx()) to obtain the desired result for every xxx function.
For example, to select product_id in reverse order (in case this was any meaningful select for the application), a function like this would do the case.
create or replace function select_products_by_inverse ()
returns int[]
as $$
select
array_agg(product_id order by product_id desc)
from
product;
$$ language sql;
It can be tested to work as
select * from select_products_by_inverse();
select_products_by_inverse |
--------------------------------------------------------|
{1011,1010,1009,1008,1007,1006,1005,1004,1003,1002,1001}|
To encapsulate the "listing" part of the query I use this function (again, extremely simplified and without any join or case for the benefit of the example).
create or replace function list_products (
tid int[]
)
returns table (
id integer,
reference varchar,
price decimal(13,4)
)
as $$
select
product_id,
reference,
price
from
product
where
product_id = any (tid);
$$ language sql;
It works, but does not respect the order of products in the passed array.
select * from list_products(select_products_by_inverse());
id |reference|price |
----|---------|--------|
1001|MX-232 |100.0000|
1011|AX-232 | 20.0000|
1003|KKK 11 | 11.0000|
1004|OXS SUPER| 0.3500|
1005|ROR-MOT |200.0000|
1006|234PPP | 30.5000|
1007|T555-NS |110.2500|
1008|LM234-XS |101.2000|
1009|MOTOR-22 | 12.5000|
1010|MOTOR-11 | 30.0000|
1002|XUL-XUL1 | 40.0000|
So, the problem is I am passing a custom ordered array of product_id but the list_products() function does not respect the order inside the array.
Obviously, I could include an order by clause in list_products(), but remember that the ordering must be determined by the select_products_by_xxx() functions to keep the list_products() unique.
Any idea?
EDIT
#adamkg solution is simple and works: adding a universal order by clause like this:
order by array_position(tid, product_id);
However, this means to ordering products twice: first inside select_products_by_xxx() and then inside list_products().
An explain exploration renders the following result:
QUERY PLAN |
----------------------------------------------------------------------|
Sort (cost=290.64..290.67 rows=10 width=56) |
Sort Key: (array_position(select_products_by_inverse(), product_id))|
-> Seq Scan on product (cost=0.00..290.48 rows=10 width=56) |
Filter: (product_id = ANY (select_products_by_inverse())) |
Now I am wondering if there is any other better approach to reduce cost, keeping separability between functions.
I see two promising strategies:
As for the explain clause and the issue itself, it seems that an complete scan of table product is being done inside list_products(). As there may be thousands of products, a better approach would be to scan the passed array instead.
The xxx functions can be refactored to return setof int instead of int[]. However, a set cannot be passed as a function parameter.
For long arrays you typically get (much!) more efficient query plans with unnesting the array and joining to the main table. In simple cases, this even preserves the original order of the array without adding ORDER BY. Rows are processed in order. But there are no guarantees and the order may be broken with more joins or with parallel execution etc. To be sure, add WITH ORDINALITY:
CREATE OR REPLACE FUNCTION list_products (tid int[]) -- VARIADIC?
RETURNS TABLE (
id integer,
reference varchar,
price decimal(13,4)
)
LANGUAGE sql STABLE AS
$func$
SELECT product_id, p.reference, p.price
FROM unnest(tid) WITH ORDINALITY AS t(product_id, ord)
JOIN product p USING (product_id) -- LEFT JOIN ?
ORDER BY t.ord
$func$;
Fast, simple, safe. See:
PostgreSQL unnest() with element number
Join against the output of an array unnest without creating a temp table
You might want to throw in the modifier VARIADIC, so you can call the function with an array or a list of IDs (max 100 items by default). See:
Return rows matching elements of input array in plpgsql function
Call a function with composite type as argument from native query in jpa
Function to select column values by comparing against comma separated list
I would declare STABLE function volatility.
You might use LEFT JOIN instead of JOIN to make sure that all given IDs are returned - with NULL values if a row with given ID has gone missing.
db<>fiddle here
Note a subtle logic difference with duplicates in the array. While product_id is UNIQUE ...
unnest + left join returns exactly one row for every given ID - preserving duplicates in the given IDs if any.
product_id = any (tid) folds duplicates. (One of the reasons it typically results in more expensive query plans.)
If there are no dupes in the given array, there is no difference. If there can be duplicates and you want to fold them, your task is ambiguous, as it's undefined which position to keep.
You're very close, all you need to add is ORDER BY array_position(tid, product_id).
testdb=# create or replace function list_products (
tid int[]
)
returns table (
id integer,
reference varchar,
price decimal(13,4)
)
as $$
select
product_id,
reference,
price
from
product
where
product_id = any (tid)
-- add this:
order by array_position(tid, product_id);
$$ language sql;
CREATE FUNCTION
testdb=# select * from list_products(select_products_by_inverse());
id | reference | price
------+-----------+----------
1011 | AX-232 | 20.0000
1010 | MOTOR-11 | 30.0000
1009 | MOTOR-22 | 12.5000
1008 | LM234-XS | 101.2000
1007 | T555-NS | 110.2500
1006 | 234PPP | 30.5000
1005 | ROR-MOT | 200.0000
1004 | OXS SUPER | 0.3500
1003 | KKK 11 | 11.0000
1002 | XUL-XUL1 | 40.0000
1001 | MX-232 | 100.0000
(11 rows)
I would like to know if performing a SELECT DISTINCT query implies a sequential scan, and how I can optimize it.
I created a dummy table and confirmed that when there is no index, SELECT DISTINCT does a Seq Scan.
test=# create table test2 (id SERIAL, t1 text);
CREATE TABLE
test=# insert into test2 select generate_series(0, 100000) AS id, md5(random()::text) AS t1;
INSERT 0 100001
test=# explain analyze select distinct t1 from test2;
Results in:
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------
HashAggregate (cost=2157.97..2159.97 rows=200 width=32) (actual time=54.086..77.352 rows=100000 loops=1)
Group Key: t1
-> Seq Scan on test2 (cost=0.00..1893.18 rows=105918 width=32) (actual time=0.012..12.232 rows=100001 loops=1)
Planning time: 0.079 ms
Execution time: 86.345 ms
(5 rows)
When we create index:
test=# create index test2_idx_t1 on test2 (t1);
CREATE INDEX
test=# explain analyze select distinct t1 from test2;
Results in:
first time:
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------
HashAggregate (cost=2084.01..2086.01 rows=200 width=32) (actual time=48.871..74.617 rows=100000 loops=1)
Group Key: t1
-> Seq Scan on test2 (cost=0.00..1834.01 rows=100001 width=32) (actual time=0.009..9.891 rows=100001 loops=1)
Planning time: 0.145 ms
Execution time: 83.564 ms
(5 rows)
second time and onwards:
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------
Unique (cost=0.42..7982.42 rows=100001 width=33) (actual time=0.016..80.949 rows=100000 loops=1)
-> Index Only Scan using test2_idx_t1 on test2 (cost=0.42..7732.42 rows=100001 width=33) (actual time=0.015..53.396 rows=100001 loops=1)
Heap Fetches: 100001
Planning time: 0.053 ms
Execution time: 87.552 ms
(5 rows)
Why is it doing a Seq Scan when queried the first time after the index was created?
Why is the index scan more expensive than the seq scan in this case, and why is the query planner choosing it?
To get the result of a query that is about all the rows in a table, the whole table has to be scanned.
The only way you can avoid a sequential table scan is to have an index on t1 and have a recently vacuumed table so that most blocks are "all visible". Then an "index only scan" can be used, which is usually cheaper.
Why is the index only scan not used right away? I cannot answer that with absolute certainty, but a good guess is that autovacuum was still busy on the table when you ran the query the first time.
Lets say I have 3 tables with millions of rows.
CREATE TABLE blog (
blog_id integer NOT NULL,
blog_definition text,
create_date timestamp without time zone,
user_id integer,
CONSTRAINT "Blog_pkey" PRIMARY KEY (blog_id)
);
CREATE TABLE blog_detail (
blog_detail_id integer NOT NULL,
blog_id integer,
blog_header text,
user_id integer,
blog_content text,
create_date timestamp without time zone,
CONSTRAINT "Blog_Detail_pkey" PRIMARY KEY (blog_detail_id)
);
CREATE TABLE users (
user_id integer NOT NULL,
country text,
user_name text,
CONSTRAINT "User_pkey" PRIMARY KEY (user_id)
);
CREATE INDEX blog_create_date_user_id_blog_definition_idx
ON blog
USING btree
(create_date, user_id, blog_definition COLLATE pg_catalog."default");
CREATE INDEX blog_detail_create_date_user_id_blog_content_blog_header_idx
ON blog_detail
USING btree
(create_date, user_id, blog_content COLLATE pg_catalog."default", blog_header COLLATE pg_catalog."default");
CREATE INDEX users_country_user_id_idx
ON users
USING btree
(country COLLATE pg_catalog."default", user_id);
And the query is like that.This query took 35 seconds with these indexes to get the results.
SELECT b.blog_definition, b.create_date, b.user_id, bd.blog_header,
bd.blog_content, bd.user_id, bd.create_date
FROM blog b
FULL OUTER JOIN blog_detail bd ON b.create_date = bd.create_date
WHERE CASE
WHEN b.blog_id IS NULL THEN
bd.user_id IN (SELECT user_id FROM users WHERE country = 'Greece')
WHEN bd.blog_id IS NULL THEN
b.user_id IN (SELECT user_Id FROM users WHERE country = 'Greece')
END
ORDER BY CASE
WHEN b.blog_id IS NULL THEN bd.create_date
WHEN bd.blog_id IS NULL THEN b.create_date
ELSE b.create_date
END DESC
LIMIT 25;
Which columns in 3 tables do i need to index(and what kind of index) to get best query performance?
explain analyze results :
Limit (cost=820038.99..820039.06 rows=25 width=50) (actual time=33047.344..33047.348 rows=25 loops=1)
-> Sort (cost=820038.99..832538.93 rows=4999976 width=50) (actual time=33047.341..33047.343 rows=25 loops=1)
Sort Key: (CASE WHEN (b.blog_id IS NULL) THEN bd.create_date WHEN (bd.blog_id IS NULL) THEN b.create_date ELSE b.create_date END)
Sort Method: top-N heapsort Memory: 26kB
-> Hash Full Join (cost=191546.31..678943.27 rows=4999976 width=50) (actual time=3039.060..28832.090 rows=15000000 loops=1)
Hash Cond: (b.create_date = bd.create_date)
Filter: CASE WHEN (b.blog_id IS NULL) THEN (hashed SubPlan 1) WHEN (bd.blog_id IS NULL) THEN (hashed SubPlan 2) ELSE NULL::boolean END
-> Seq Scan on blog b (cost=0.00..173529.53 rows=9999953 width=22) (actual time=0.035..2090.918 rows=10000000 loops=1)
-> Hash (cost=91666.89..91666.89 rows=4999989 width=28) (actual time=3003.440..3003.440 rows=5000000 loops=1)
Buckets: 8192 Batches: 128 Memory Usage: 2546kB
-> Seq Scan on blog_detail bd (cost=0.00..91666.89 rows=4999989 width=28) (actual time=0.008..1130.650 rows=5000000 loops=1)
SubPlan 1
-> Index Only Scan using users_country_user_id_idx on users (cost=0.56..1496.38 rows=41361 width=4) (actual time=0.050..4.007 rows=20000 loops=1)
Index Cond: (country = 'Germany'::text)
Heap Fetches: 0
SubPlan 2
-> Index Only Scan using users_country_user_id_idx on users users_1 (cost=0.56..1496.38 rows=41361 width=4) (actual time=0.057..4.060 rows=20000 loops=1)
Index Cond: (country = 'Germany'::text)
Heap Fetches: 0
Planning time: 0.253 ms
Execution time: 33048.583 ms
Like Couling commented to your question, FULL JOINs tend to be problematic with indexes. That said, there is much to improve upon your query:
SELECT b.blog_definition, create_date, b.user_id, bd.blog_header,
bd.blog_content, bd.user_id
FROM blog b
FULL JOIN blog_detail bd USING (create_date)
WHERE EXISTS
(SELECT 1 FROM users
WHERE country = 'Greece' AND user_id = coalesce(bd.user_id, b.user_id))
ORDER BY create_date DESC
LIMIT 25;
When you do a JOIN with the USING clause (instead of ON) then only one of the matching columns is included in the select list, so no need to use aliases. The convoluted ORDER BY clause was unnecessary anyway because b.create_date and bd.create_date are equal by virtue of the join.
The CASE WHEN clause in the WHERE filter can also be avoided by using the coalesce() function and the obvious condition that either table has to have a value for blog_id and one for user_id too (otherwise your query would fail because the filter would evaluate to WHERE NULL). Since b.blog_id is the primary key of table blog it is therefore never NULL so by that same logic b.user_id could never be NULL and you could replace the coalesce() function with the column name. But that is left for you to ponder. If you look at your EXPLAIN ANALYZE you see that the very same sub-query gets evaluated twice (SubPlan 1 and SubPlan 2). This query will access table users only once. That's a proper 4ms saved! Plus another few ms because the sub-query is faster than in your code.
The create_date field is a timestamp. Joining on timestamp equality is only possible if both records were created in the same session or when the value in one of the records is retrieved from the other record, such that their values are exactly the same.
You define an index on table blog_detail, but the index will be quite large because you include two potentially large text fields. Using an index on create_date alone will be much smaller (so fewer disk reads) and faster to process.
I have news_alerts and news_items tables in postgres with the following index:
CREATE TABLE news_items (
id INTEGER NOT NULL,
content character varying
);
CREATE TABLE news_alerts (
id INTEGER NOT NULL,
user_id INTEGER NOT NULL,
terms character varying(255)
);
CREATE INDEX news_alerts_terms ON news_alerts
USING gin (to_tsvector('english'::regconfig, (terms)::text));
And the following news alerts:
INSERT INTO news_alerts (user_id, terms) values (1, 'Jim Jarmusch');
INSERT INTO news_alerts (user_id, terms) values (1, 'Kim Kardashian');
INSERT INTO news_alerts (user_id, terms) values (2, 'Kim Kardashian');
When a new news item comes in, I add it to news_items:
INSERT INTO news_items (content) values ('Breaking: Kim Kardashian posts unconscionable new selfies from birthday party');
At this point, I want to alert the two users who have (however unwisely) chosen to receive Kim Kardashian news. I want to find all the news_alerts whose terms match the content of that news_item and notify those users.
Is there some way to do this with postgres indexes or even an external tool or service?
You can try abusing the ts_debug function. Gist is below. The t table would contain your news alerts. You probably should not directly use ts_debug - it is a sql language function, look at how it is implemented.
When you have lots of rows, you will not need to disable sequential scans. I did it just to show that an index can get used.
This is a bit terse, If you have questions, ask.
db=# create temp table t as select array_agg(distinct lexeme) as x from (select unnest(lexemes) as lexeme from ts_debug('english', 'Kim Kardashian')) a;
SELECT 1
db=# create index on t using gin(x);
CREATE INDEX
db=# set enable_seqscan to off;
SET
db=# select * from t where x <# (select array_agg(distinct lexeme) as x from (select unnest(lexemes) as lexeme from ts_debug('english', 'Kim Kardashian is a woman')) a);
x
------------------
{kardashian,kim}
(1 row)
db=# explain select * from t where x <# (select array_agg(distinct lexeme) as x from (select unnest(lexemes) as lexeme from ts_debug('english', 'Kim Kardashian is a woman')) a);
QUERY PLAN
-----------------------------------------------------------------------------------
Bitmap Heap Scan on t (cost=1765.76..1769.78 rows=1 width=32)
Recheck Cond: (x <# $0)
InitPlan 1 (returns $0)
-> Aggregate (cost=1757.75..1757.76 rows=1 width=32)
-> Function Scan on ts_debug (cost=0.25..507.75 rows=100000 width=32)
-> Bitmap Index Scan on t_x_idx (cost=0.00..8.00 rows=1 width=0)
Index Cond: (x <# $0)
(7 rows)
I have two tables -
Table A : 1MM rows,
AsOfDate, Id, BId (foreign key to table B)
Table B : 50k rows,
Id, Flag, ValidFrom, ValidTo
Table A contains multiple records per day between 2011/01/01 and 2011/12/31 across 100 BId's.
Table B contains multiple non overlapping (between validfrom and validto) records for 100 Bids.
The task of the join will be to return the flag that was active for the BId on the given AsOfDate.
select
a.AsOfDate, b.Flag
from
A a inner Join B b on
a.BId = b.BId and b.ValidFrom <= a.AsOfDate and b.ValidTo >= a.AsOfDate
where
a.AsOfDate >= 20110101 and a.AsOfDate <= 20111231
This query takes ~70 seconds on a very high end server (+3Ghz) with 64Gb of memory.
I have indexes on every combination of field as I'm testing this - to no avail.
Indexes : a.AsOfDate, a.AsOfDate+a.bId, a.bid
Indexes : b.bid, b.bid+b.validfrom
Also tried the range queries suggested below (62seconds)
This same query on the free version of Sql Server running in a VM takes ~1 second to complete.
any ideas?
Postgres 9.2
Query Plan
QUERY PLAN
---------------------------------------------------------------------------------------
Aggregate (cost=8274298.83..8274298.84 rows=1 width=0)
-> Hash Join (cost=1692.25..8137039.36 rows=54903787 width=0)
Hash Cond: (a.bid = b.bid)
Join Filter: ((b.validfrom <= a.asofdate) AND (b.validto >= a.asofdate))
-> Seq Scan on "A" a (cost=0.00..37727.00 rows=986467 width=12)
Filter: ((asofdate > 20110101) AND (asofdate < 20111231))
-> Hash (cost=821.00..821.00 rows=50100 width=12)
-> Seq Scan on "B" b (cost=0.00..821.00 rows=50100 width=12)
see http://explain.depesz.com/s/1c5 for the analyze output
Consider using the range types available in postgresql 9.2:
create index on a using gist(int4range(asofdate, asofdate, '[]'));
create index on b using gist(int4range(validfrom, validto, '[]'));
You can query for a date in a matching a range like so:
select * from a
where int4range(asofdate,asofdate,'[]') && int4range(20110101, 20111231, '[]');
And for rows in b overlapping a record in a like so:
select *
from b
join a on int4range(b.validfrom,b.validto,'[]') #> a.asofdate
where a.id = 1
(&& means "overlaps", #>means "contains", and '[]' indicates to create a range that includes both end points)
The issues was with the indexes - for some reason unclear to me, the indexes on the tables were not being referenced correctly by the query analyzer - i removed them all, added them back (exactly the same - via script) and the query now takes ~303ms.
thanks for all the help on this very frustrating problem.