ArangoDB: how to optimize FILTER on a subquery on big collections - graph-databases

Question

I have a fairly simple data model (inspired by IMDB data) to link films to their directors or actors.
This includes 3 collections:
document collection Titles
document collection Persons
edge collection core_edge_values_links with an attribute field to determine the role of the person (director, actor...)
Titles are linked to Persons through a core_edge_values_links edge:
Now, I'm trying to find all Titles which have a link to any director.
Query is:
FOR r IN titles
LET rlinkVal = FIRST(
FOR rv, re IN 1 OUTBOUND r._id
core_edge_values_links
FILTER re.attribute == 'directors'
RETURN re
)
FILTER rlinkVal != null
LIMIT 0, 20
RETURN r
This query will take about 2.5s (and up to almost 2 minutes with fullCount enable) on a DB with 1M titles, about 200k persons and about 4M edges. I have about 600k matching titles.
To try to speed this up, I added an index on the core_edge_values_links colllection on fields _from and attribute. Unfortunately, the traversal won't take advantage of this index.
I also tried to use a simple join instead of the traversal:
FOR r IN titles
LET rlinkVal = FIRST(
FOR e IN core_edge_values_links
FILTER e._from == r._id AND e.attribute == 'directors'
RETURN true
)
FILTER rlinkVal != null
LIMIT 0, 20
RETURN r
This time, the index is used, but query time is pretty much the same as with the traversal (about 2.5s, see profiling data below).
This makes me wondering if the problem just comes from the number of titles in the collection (about 1M) as they're all scanned in the initial FOR but I don't see how I could index it. This is a fairly simple use case, I don't feel this should take so long.
This is a fairly simple use case, I don't feel this should take so long.
Here is profiling data provided by arango.
With the traversal:
Query String (241 chars, cacheable: false):
FOR r IN titles
LET rlinkVal = FIRST(
FOR rv, re IN 1 OUTBOUND r._id
core_edge_values_links
FILTER re.attribute == 'directors'
RETURN re
)
FILTER rlinkVal != null
LIMIT 0, 20
RETURN r
Execution plan:
Id NodeType Calls Items Runtime [s] Comment
1 SingletonNode 1 1 0.00001 * ROOT
2 EnumerateCollectionNode 305 304000 0.08433 - FOR r IN titles /* full collection scan */
4 CalculationNode 305 304000 0.07084 - LET #6 = r.`_id` /* attribute expression */ /* collections used: r : titles */
16 SubqueryStartNode 609 608000 0.12038 - LET #3 = ( /* subquery begin */
5 TraversalNode 306 305000 1.93808 - FOR /* vertex optimized away */, re /* edge */ IN 1..1 /* min..maxPathDepth */ OUTBOUND #6 /* startnode */ core_edge_values_links
6 CalculationNode 306 305000 0.02857 - LET #10 = (re.`attribute` == "directors") /* simple expression */
7 FilterNode 305 304000 0.05040 - FILTER #10
15 LimitNode 305 304000 0.00927 - LIMIT 0, 1
17 SubqueryEndNode 304 303000 0.05520 - RETURN re ) /* subquery end */
10 CalculationNode 304 303000 0.02094 - LET rlinkVal = FIRST(#3) /* simple expression */
11 CalculationNode 304 303000 0.01909 - LET #12 = (rlinkVal != null) /* simple expression */
12 FilterNode 1 20 0.01508 - FILTER #12
13 LimitNode 1 20 0.00001 - LIMIT 0, 20
14 ReturnNode 1 20 0.00001 - RETURN r
Indexes used:
By Name Type Collection Unique Sparse Selectivity Fields Ranges
5 edge edge core_edge_values_links false false 34.87 % [ `_from` ] base OUTBOUND
Traversals on graphs:
Id Depth Vertex collections Edge collections Options Filter / Prune Conditions
5 1..1 core_edge_values_links uniqueVertices: none, uniqueEdges: path FILTER (re.`attribute` == "directors")
Optimization rules applied:
Id RuleName
1 move-calculations-up
2 optimize-subqueries
3 optimize-traversals
4 splice-subqueries
Query Statistics:
Writes Exec Writes Ign Scan Full Scan Index Filtered Peak Mem [b] Exec Time [s]
0 0 304000 0 302977 1605632 2.41360
Query Profile:
Query Stage Duration [s]
initializing 0.00000
parsing 0.00006
optimizing ast 0.00001
loading collections 0.00001
instantiating plan 0.00006
optimizing plan 0.00120
executing 2.41225
finalizing 0.00003
With the join:
Query String (232 chars, cacheable: false):
FOR r IN titles
LET rlinkVal = FIRST(
FOR e IN core_edge_values_links
FILTER e._from == r._id AND e.attribute == 'directors'
RETURN true
)
FILTER rlinkVal != null
LIMIT 0, 20
RETURN r
Execution plan:
Id NodeType Calls Items Runtime [s] Comment
1 SingletonNode 1 1 0.00001 * ROOT
7 CalculationNode 1 1 0.00000 - LET #7 = true /* json expression */ /* const assignment */
2 EnumerateCollectionNode 305 304000 0.09654 - FOR r IN titles /* full collection scan */
17 SubqueryStartNode 609 608000 0.13655 - LET #2 = ( /* subquery begin */
16 IndexNode 305 304000 2.29080 - FOR e IN core_edge_values_links /* persistent index scan, scan only */
15 LimitNode 305 304000 0.00992 - LIMIT 0, 1
18 SubqueryEndNode 304 303000 0.05765 - RETURN #7 ) /* subquery end */
10 CalculationNode 304 303000 0.02272 - LET rlinkVal = FIRST(#2) /* simple expression */
11 CalculationNode 304 303000 0.02060 - LET #9 = (rlinkVal != null) /* simple expression */
12 FilterNode 1 20 0.01656 - FILTER #9
13 LimitNode 1 20 0.00000 - LIMIT 0, 20
14 ReturnNode 1 20 0.00000 - RETURN r
Indexes used:
By Name Type Collection Unique Sparse Selectivity Fields Ranges
16 idx_1742224911374483456 persistent core_edge_values_links false false n/a [ `_from`, `attribute` ] ((e.`_from` == r.`_id`) && (e.`attribute` == "directors"))
Optimization rules applied:
Id RuleName
1 move-calculations-up
2 optimize-subqueries
3 use-indexes
4 remove-filter-covered-by-index
5 remove-unnecessary-calculations-2
6 splice-subqueries
Query Statistics:
Writes Exec Writes Ign Scan Full Scan Index Filtered Peak Mem [b] Exec Time [s]
0 0 304000 492 302977 1441792 2.65203
Query Profile:
Query Stage Duration [s]
initializing 0.00000
parsing 0.00006
optimizing ast 0.00001
loading collections 0.00001
instantiating plan 0.00003
optimizing plan 0.00056
executing 2.65136
finalizing 0.00003