SPARQL Speed up federated query - query-optimization

Question

I have my own dataset and I want to perform a federated query in SPARQL. Here is the query:
PREFIX : <http://myURIsNamespace#>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX p: <http://www.wikidata.org/prop/>
PREFIX ps: <http://www.wikidata.org/prop/statement/>
PREFIX pq: <http://www.wikidata.org/prop/qualifier/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
select * where {
?bioentity :hasMutatedVersionOf ?gene .
?gene :partOf wd:Q430258 .
SERVICE <https://query.wikidata.org/sparql> {
?gene p:P644 ?statement;
wdt:P31 wd:Q7187 ;
wdt:P703 wd:Q15978631 ;
wdt:P1057 wd:Q430258 .
?statement ps:P644 ?start .
?statement pq:P659 wd:Q20966585 .
?gene p:P645 ?statement2.
?statement2 ps:P645 ?end .
?statement2 pq:P659 wd:Q20966585 .
FILTER (xsd:integer(?start)>21000000 && xsd:integer(?start)<30000000)
}
}
I run the query via graphDB SPARQL interface but it's really really slow. It takes more than a minute to return 8 records. If I split the query in two parts, they are ridiculously fast.
Query#1
select * where {
?bioentity :hasMutatedVersionOf ?gene .
?gene :partOf wd:Q430258 .
}
56 records in 0.1s
Query#2
select * where {
SERVICE <https://query.wikidata.org/sparql> {
?gene p:P644 ?statement;
wdt:P31 wd:Q7187 ;
wdt:P703 wd:Q15978631 ;
wdt:P1057 wd:Q430258 .
?statement ps:P644 ?start .
?statement pq:P659 wd:Q20966585 .
?gene p:P645 ?statement2.
?statement2 ps:P645 ?end .
?statement2 pq:P659 wd:Q20966585 .
FILTER (xsd:integer(?start)>21000000 && xsd:integer(?start)<30000000)
}
}
158 records in 0.5s
Why the is the federation so slow? Is there a way to optimize the performance?

Answer 1

Short answer
Just place your SERVICE part first, i. e. before ?bioentity :hasMutatedVersionOf ?gene .
Read a good article on the topic (e. g. chapter 5 of this book)
Relevant quote from the aforementioned article:
3.3.2 Query Optimization and Execution
The execution order of query operators significantly influences the overall query evaluation cost.
Besides the important query execution time there are also other
aspects in the federated scenario which are relevant for the query
optimization:
Minimizing communication cost. The number of contacted
data sources directly influences the performance of the query
execution due to the communication overhead. However, reducing the
number of involved data source trades off against completeness of
results.
Optimizing execution localization. The standard query
interfaces of linked data sources are generally only capable of
answering queries on their provided data. Therefore, joins with
other data results usually need to be done at the query issuer. If
possible at all, a better strategy will move parts of the result
merging operations to the data sources, especially if they can be
executed in parallel.
Streaming results. Retrieving a complete result
when evaluating a query on a large dataset may take a while even with
a well optimized execution strategy. Thus one can return results as
soon as they become available, which can be optimized by trying to
return relevant results first.
Long answer
Example data
PREFIX : <http://myURIsNamespace#>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX p: <http://www.wikidata.org/prop/>
PREFIX ps: <http://www.wikidata.org/prop/statement/>
PREFIX pq: <http://www.wikidata.org/prop/qualifier/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
INSERT { ?gene rdf:type owl:Thing }
WHERE {
SERVICE <https://query.wikidata.org/sparql> {
?gene p:P644 ?statement;
wdt:P31 wd:Q7187 ;
wdt:P703 wd:Q15978631 ;
wdt:P1057 wd:Q430258 .
?statement ps:P644 ?start .
?statement pq:P659 wd:Q20966585 .
?gene p:P645 ?statement2.
?statement2 ps:P645 ?end .
?statement2 pq:P659 wd:Q20966585 .
FILTER (xsd:integer(?start)>26000000 && xsd:integer(?start)<30000000)
}
}
The total number of triples is 79. Please note that 26000000 is used instead of 21000000.
Query 1
PREFIX : <http://myURIsNamespace#>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX p: <http://www.wikidata.org/prop/>
PREFIX ps: <http://www.wikidata.org/prop/statement/>
PREFIX pq: <http://www.wikidata.org/prop/qualifier/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
SELECT * WHERE {
?gene rdf:type owl:Thing .
SERVICE <https://query.wikidata.org/sparql> {
?gene p:P644 ?statement;
wdt:P31 wd:Q7187 ;
wdt:P703 wd:Q15978631 ;
wdt:P1057 wd:Q430258 .
?statement ps:P644 ?start .
?statement pq:P659 wd:Q20966585 .
?gene p:P645 ?statement2.
?statement2 ps:P645 ?end .
?statement2 pq:P659 wd:Q20966585 .
FILTER (xsd:integer(?start)>20000000 && xsd:integer(?start)<30000000)
}
}
Query 2
PREFIX : <http://myURIsNamespace#>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX p: <http://www.wikidata.org/prop/>
PREFIX ps: <http://www.wikidata.org/prop/statement/>
PREFIX pq: <http://www.wikidata.org/prop/qualifier/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
SELECT * WHERE {
SERVICE <https://query.wikidata.org/sparql> {
?gene p:P644 ?statement;
wdt:P31 wd:Q7187 ;
wdt:P703 wd:Q15978631 ;
wdt:P1057 wd:Q430258 .
?statement ps:P644 ?start .
?statement pq:P659 wd:Q20966585 .
?gene p:P645 ?statement2.
?statement2 ps:P645 ?end .
?statement2 pq:P659 wd:Q20966585 .
FILTER (xsd:integer(?start)>20000000 && xsd:integer(?start)<30000000)
}
?gene rdf:type owl:Thing
}
Performance
 
Query 1
Query 2
GraphDB
30 sec
1 sec
Blazegraph
1 sec
1 sec
GraphDB behaviour
Executing Query 1, GraphDB performs 79 distinct GET requests to Wikidata¹:
These requests are queries of this kind:
SELECT ?start ?statement ?end ?statement2 WHERE {
<http://www.wikidata.org/entity/Q18031286> p:P644 ?statement;
wdt:P31 wd:Q7187 ;
wdt:P703 wd:Q15978631 ;
wdt:P1057 wd:Q430258 .
?statement ps:P644 ?start .
?statement pq:P659 wd:Q20966585 .
<http://www.wikidata.org/entity/Q18031286> p:P645 ?statement2.
?statement2 ps:P645 ?end .
?statement2 pq:P659 wd:Q20966585 .
FILTER (xsd:integer(?start)>20000000 && xsd:integer(?start)<30000000)
It seems interesting, that on another machine, GraphDB performs GET requests of another kind:
GET /sparql?queryLn="Sparql"&query=<original_query_service_part>&$gene=<http://www.wikidata.org/entity/Q18031286>
In this request, Sesame protocol is used, these bindings in URL are not a part of SPARQL 1.1 Protocol.
Perhaps the exact kind of a request depends on the value of the internal reuse.vars.in.subselects parameter, which default value is presumably different on Windows and on Linux.
Blazegraph behaviour
Executing Query 1, Blazegraph performs single POST request to Wikidata²:
SELECT ?gene ?statement ?start ?statement2 ?end
WHERE {
?gene p:P644 ?statement;
wdt:P31 wd:Q7187 ;
wdt:P703 wd:Q15978631 ;
wdt:P1057 wd:Q430258 .
?statement ps:P644 ?start .
?statement pq:P659 wd:Q20966585 .
?gene p:P645 ?statement2.
?statement2 ps:P645 ?end .
?statement2 pq:P659 wd:Q20966585 .
FILTER (xsd:integer(?start)>20000000 && xsd:integer(?start)<30000000)
}
VALUES ( ?gene) {
( wd:Q14908148 ) ( wd:Q15320063 ) ( wd:Q17861651 ) ( wd:Q17917753 ) ( wd:Q17928333 )
( wd:Q18024923 ) ( wd:Q18026347 ) ( wd:Q18030710 ) ( wd:Q18031220 ) ( wd:Q18031457 )
( wd:Q18031551 ) ( wd:Q18031832 ) ( wd:Q18032918 ) ( wd:Q18033094 ) ( wd:Q18033798 )
( wd:Q18034311 ) ( wd:Q18035006 ) ( wd:Q18035085 ) ( wd:Q18035609 ) ( wd:Q18036516 )
( wd:Q18036676 ) ( wd:Q18037580 ) ( wd:Q18038385 ) ( wd:Q18038459 ) ( wd:Q18038737 )
( wd:Q18038763 ) ( wd:Q18039997 ) ( wd:Q18040291 ) ( wd:Q18041261 ) ( wd:Q18041415 )
( wd:Q18041558 ) ( wd:Q18045881 ) ( wd:Q18047232 ) ( wd:Q18047373 ) ( wd:Q18047918 )
( wd:Q18047966 ) ( wd:Q18048744 ) ( wd:Q18049145 ) ( wd:Q18049164 ) ( wd:Q18053139 )
( wd:Q18056540 ) ( wd:Q18057411 ) ( wd:Q18060804 ) ( wd:Q18060856 ) ( wd:Q18060876 )
( wd:Q18060905 ) ( wd:Q18060958 ) ( wd:Q20773708 ) ( wd:Q15312971 ) ( wd:Q17860819 )
( wd:Q17917713 ) ( wd:Q18026310 ) ( wd:Q18027015 ) ( wd:Q18031286 ) ( wd:Q18032599 )
( wd:Q18032797 ) ( wd:Q18035169 ) ( wd:Q18035627 ) ( wd:Q18039938 ) ( wd:Q18041207 )
( wd:Q18041512 ) ( wd:Q18041930 ) ( wd:Q18045491 ) ( wd:Q18045762 ) ( wd:Q18046301 )
( wd:Q18046472 ) ( wd:Q18046487 ) ( wd:Q18047149 ) ( wd:Q18047491 ) ( wd:Q18047719 )
( wd:Q18048527 ) ( wd:Q18049774 ) ( wd:Q18051886 ) ( wd:Q18053875 ) ( wd:Q18056212 )
( wd:Q18056538 ) ( wd:Q18065866 ) ( wd:Q20766978 ) ( wd:Q20781543 )
}
Conclusion
With federated queries, it is hard to create effective execution plan, since selectivity of remote patterns is unknown.
In your particular case, it should be not very important, whether to join results locally or remotely, because both local and remote resultsets are small. However, in GraphDB, joining results remotely is less effective, because GraphDB does not reduce communication costs.
¹ For screenshots creation, <http://query.wikidata.org/sparql> instead of <https://query.wikidata.org/sparql> was used.
² In Blazegraph, one might write hint:Query hint:optimizer "None" to ensure sequential evaluation.