Error during application initialization : ClusterSearcher should have a top level dispatch - vespa

Question

Application with multiple content and each content with multiple documents is throwing "ClusterSearcher should have a top level dispatch." when deployed on multi node cluster with multiple content nodes. The same is working on a single node cluster with a single content node.
Using [Vespa version: ] 7.51.13
com.yahoo.container.di.componentgraph.core.ComponentNode$ComponentConstructorException: Error constructing 'com.yahoo.prelude.cluster.ClusterSearcher in acme'
Caused by: java.lang.IllegalStateException: ClusterSearcher should have a top level dispatch.
================= services.xml ==============
<?xml version="1.0" encoding="utf-8" ?>
<admin version="2.0">
<adminserver hostalias="admin0"/>
<configservers>
<configserver hostalias="admin0"/>
</configservers>
</admin>
<container id="container" version="1.0">
<config name="search.config.qr-start">
<jvm>
<heapSizeAsPercentageOfPhysicalMemory>50</heapSizeAsPercentageOfPhysicalMemory>
</jvm>
</config>
<document-api />
<http>
<server id="stateless1" port="8080">
</server>
</http>
<search>
<chain id="default" inherits="vespa">
<searcher id="com.acme.search.CatalogSearcher" bundle="sth-search">
<config name="com.acme.search.Searcher">
<redirectsFile>redirects.txt</redirectsFile>
<autoCorrectAPI>https://apps02.acme.com:9815/search/</autoCorrectAPI>
<connectionTimeout>250</connectionTimeout>
<connectionRequestTimeout>250</connectionRequestTimeout>
<socketTimeout>250</socketTimeout>
</config>
</searcher>
</chain>
</search>
<nodes jvmargs="-verbose:gc">
<node hostalias="stateless0"/>
<node hostalias="stateless1"/>
</nodes>
</container>
<content id="sth" version="1.0">
<redundancy>2</redundancy>
<documents>
<document type="sth" mode="index" />
</documents>
<nodes>
<node hostalias="content0" distribution-key="0"/>
<node hostalias="content1" distribution-key="1"/>
</nodes>
</content>
<content id="thesaurus" version="1.0">
<redundancy>2</redundancy>
<documents>
<document type="thesaurus" mode="index"/>
</documents>
<nodes>
<node hostalias="content0" distribution-key="0"/>
<node hostalias="content1" distribution-key="1"/>
</nodes>
</content>
<content id="acme" version="1.0">
<redundancy>2</redundancy>
<documents>
<document type="vc_products" mode="index" />
<document type="vc_thesaurus" mode="index"/>
</documents>
<nodes>
<node hostalias="content0" distribution-key="0"/>
<node hostalias="content1" distribution-key="1"/>
</nodes>
</content>

Answer 1

From comments, you get this not when you actually deploy on multiple nodes but when you try to instantiate a mock Application instance in a unit test, as in Application.fromApplicationPackage(...).
The reason for this is that Application is not able to fully mock a full application, only the stateless container parts. The ClusterSearcher which is instantiated in this setup complains that it can't see any real content clusters downstream (which is correct), it does not know it has been created in a mocked setup.
For this reason you need to create special services.xmls for unit tests as in general the one you use in production will give problems like this. Using Application works well for testing specific functionality of a set of components but not for unit testing your true production application.
We would like to improve this by mocking component clusters inside application but nobody's working on it at the moment. if you want to have a go at it the code is in
https://github.com/vespa-engine/vespa/tree/master/application