I am trying to set up a solrCloud with external zookeeper ensemble of 3 servers and a replicated solr on 2 servers.
Assumed that an external zookeeper should be independent from other storages I can't find out how to set the -solrhome parameter. Is the zookeeper supposed to read data from the worker nodes?
How do you upload the config and link it with target collection?
We had a lot of problems using solr.home so save yourself some stress and just keep your directories how solr likes them by default.
Example:
/example/solr/collection1/conf/schema.xml
/example/solr/collection1/conf/solrconfig.xml
/example/solr/collection1/core.properties
/example/start.jar
To get your configuration into Zookeeper, get familiar with solr's zkcli.sh script. You want to use this to manage your solr configs. It will create/update the files in ZK under the /configs node.
./zkcli.sh -cmd upconfig -confdir /example/solr/collection1/conf -confname collection1 -z 127.0.0.1
After running the upconfig cmd above, the files in /example/solr/collection1/conf will be uploaded to ZK under /configs/collection1.
Also need to link your config to your collection (creates a node under the /collections node in ZK)
# only need to link the config once
./zkcli.sh -cmd linkconfig -collection collection1 -confname collection1 -z 127.0.0.1
Then you can just start solr like this:
java -DzkHost=127.0.0.1 -jar start.jar
The other servers in your cloud will now get the configuration from zookeeper! Some more info in a pretty good blog post here: SolrCloud Cluster (Single Collection) Deployment
Note: 127.0.0.1 is a comma delimited list of your ZK servers and collection1 is your collection
You can specify the root of the Solr configuration as part of your Zookeeper connection string: -zkhost host1,host2,hostN/solr
Related
I am trying to setup up in my local laptop with 3 Solr, 3 ZooKeeper and 1 Load Balancer.
I have followed the post https://www.codehousegroup.com/insight-and-inspiration/tech-stream/how-to-configure-sitecore-with-solr-cloud and https://medium.com/#sarkaramrit2/setting-up-solr-cloud-6-3-0-with-zookeeper-3-4-6-867b96ec4272 and few others
I have setup successfully 3 solr in my laptop and URL's are as follows
https://solrcloud1:6161/solr/#/
https://solrcloud2:6162/solr/#/
https://solrcloud3:6163/solr/#/
I have installed the local load balancer "GoBetween" and mapped my above 3 solr paths over there. When I hit https://solrcloud:3010/solr/#/ this URL, I am able to receive response from different Solr instances also. It looks it's working fine as well.
ZooKeeer
I have downloaded zookeeper and placed that in all the 3 Solr locations as well
E.g.: SolrCloud1 = \LocalSolrCloud\SolrCloud1\ contains "solr-8.8.2" and "zookeeper-3.5.6" same structure for SolrCloud2 and SolrCloud3 also.
In Each zookeeper location, I have created a data folder and created "myid" file and placed the value as "1", "2", "3" respectively.
Each zoopkeeper's "zoo.cfg" file contains the below
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/SolrCloud1/zookeeper-3.5.6/data
clientPort=2181
autopurge.snapRetainCount=4
autopurge.purgeInterval=24
server.1=locahost:2888:3888
server.2=locahost:2889:3889
server.3=locahost:2890:3890
When I run the zooKeeper from the command prompt, I am getting the below error.
2022-12-15 11:47:20,051 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager#679] -
Cannot open channel to 2 at election address localhost/127.0.0.1:3889
java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)
at java.lang.Thread.run(Thread.java:748)
I have TYPO3 development environment set up using DDEV.
Now I want to add Solr server to it, and use Ext:Solr extension for indexing and searching.
How can I set up Solr server to be available from the ddev web container?
EDIT:
The instruction how to configure Solr Server for TYPO3 has been included in the official ddev-contrib repository, and can be found here:
https://github.com/drud/ddev-contrib/tree/master/docker-compose-services/typo3-solr
The original answer:
In order to configure Solr server for TYPO3 using DDEV you need to follow these steps:
Configure Solr container
Create .ddev/docker-compose.solr.yaml file:
version: '3.6'
services:
solr:
container_name: ddev-${DDEV_SITENAME}-solr
image: typo3solr/ext-solr:10.0.1
restart: "no"
ports:
- 8983
labels:
com.ddev.site-name: ${DDEV_SITENAME}
com.ddev.approot: $DDEV_APPROOT
environment:
- VIRTUAL_HOST=$DDEV_HOSTNAME
- HTTP_EXPOSE=8983
volumes:
- "./solr:/opt/solr/server/solr"
# If you want your solr to persist over `ddev stop` and `ddev start` then uncomment the following line
# If you uncomment it and want to flush your data you have to `ddev stop` and then
# `docker volume rm ddev-<projectname>_solrdata` to destroy it.
# - solrdata:/var/solr
web:
links:
- solr:$DDEV_HOSTNAME
volumes:
# solrdata is a persistent Docker volume for this project's solr data
solrdata:
For more details about the configuration, take a look at ddev documentation: https://ddev.readthedocs.io/en/latest/users/extend/custom-compose-files/
This configuration will create Solr container and make Solr admin panel being available through http://<project-name>.ddev.local:8983/solr/ from your host machine.
Copy default Solr configuration from Ext:Solr to ddev
Create the folder path .ddev/solr. Then copy
the Solr configuration and cores configuration from typo3conf/ext/solr/Resources/Private/Solr to .ddev/solr.
So you should have a structure under .ddev/solr:
/solr.xml
/cores/
/configsets/
Configure TYPO3 to access Solr
You can do it using TypoScript:
plugin.tx_solr {
solr {
host = solr
port = 8983
path = /solr/core_en/
}
}
I have solr clouds 5.2.1. I deploy solr and zookeeper. When I try to create a core this errors are throwing :
org.apache.solr.common.SolrException: Could not load conf for core contracts_shard1_replica1: Error loading solr config from solrconfig.xml
at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:78)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:635)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:611)
at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:628)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:213)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:193)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:431)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
org.apache.solr.common.SolrException: Error CREATEing SolrCore 'contracts_shard1_replica1': Unable to create core [contracts_shard1_replica1] Caused by: Can't find resource 'solrconfig.xml' in classpath or '/configs/contracts', cwd=C:\CM_10.1.0\INDEXSERVER\searchserver-distribution\target\searchserver\solr\server
at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:661)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:213)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:193)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:431)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
]
I created contracts inside of C:\CM_10.1.0\INDEXSERVER\searchserver-distribution\target\searchserver\solr\server and copied " conf" folder which is in solr\configsets\basic_configs" into contracts. But problem didn't solved.
I do need help to solve this problem. Does anyone help me?
Thanks
Since you are using the zookeeper, you must first send the config files to the zookeeper. I'm not sure how it is in Windows :P, but in Linux it would be:
cd /searchserver/solr/server/scripts/cloud-scripts
./zkcli.sh -cmd upconfig -confdir /searchserver/solr/server/solr/corename/conf -confname myconfname -z zoo1:2181,zoo2:2181,zoo3:2181
In Windows, use zkcli.bat in the same directory.
Another way to do this is by adding
SOLR_OPTS="$SOLR_OPTS -Dbootstrap_confdir=./solr/corename/conf/"
SOLR_OPTS="$SOLR_OPTS -Dcollection.configName=myconfname"
to the solr.in.sh file, then (re)starting solr. In Windows, the file is solr.in.cmd, and you add the following lines:
set SOLR_OPTS=%SOLR_OPTS% -Dbootstrap_confdir=./solr/corename/conf/
set SOLR_OPTS=%SOLR_OPTS% -Dcollection.configName=myconfname
The solr.in.sh/solr.in.cmd file is included into the solr (colr.cmd) command that you use to start the solr server. Myconfname above (in both methods) is an arbitrary name you give to indicate the sets of config files that you've added to the zookeeper. Then you can create the core using the collections API:
http://localhost:8983/solr/admin/collections?action=CREATE&name=coreName&numShards=2&shards=shard1,shard2&collection.configName=myconfname&createNodeSet=localhost:8983_solr
I am using external zookeper for testing I am using on local system steps I followed are as bellow.
Step 1. created 3 zookeper server with data containing myid file containing unique numbers 1,2,3 respectively.
Step 2. I started all three zookeper server useing command
./zkServer.sh start
Step 3. check status of each server 2 showing status as leader and remaining 2 as Mode: follower
Step4 : try to run solr cloud example as
/opt/solr$bin/solr start -e cloud -z localhost:2181,localhost:2182,localhost:2183
It ask me no of shards replicas etc. System ask me collection name I entered test but it throws an exception like
`basic_configs, data_driven_schema_configs, or sample_techproducts_configs [data_driven_schema_configs]
Exception in thread "main" org.apache.solr.client.solrj.SolrServerException: Error loading config name for collection phrases
at org.apache.solr.util.SolrCLI.getJson(SolrCLI.java:537)
at org.apache.solr.util.SolrCLI.getJson(SolrCLI.java:471)
at org.apache.solr.util.SolrCLI$StatusTool.getCloudStatus(SolrCLI.java:721)
at org.apache.solr.util.SolrCLI$StatusTool.reportStatus(SolrCLI.java:704)
at org.apache.solr.util.SolrCLI.getZkHost(SolrCLI.java:1160)
at org.apache.solr.util.SolrCLI$CreateCollectionTool.runTool(SolrCLI.java:1210)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:215)
Enabling auto soft-commits with maxTime 3 secs using the Config API
POSTing request to Config API: http://localhost:8990/solr/sai/config
{"set-property":{"updateHandler.autoSoftCommit.maxTime":"3000"}}
Exception in thread "main" org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://localhost:8990/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/sai/config. Reason:
<pre> Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:529)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
at org.apache.solr.util.SolrCLI.postJsonToSolr(SolrCLI.java:1882)
at org.apache.solr.util.SolrCLI$ConfigTool.runTool(SolrCLI.java:1856)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:215)
SolrCloud example running, please visit http://localhost:8990/solr `
It showing error Exception in thread "main" org.apache.solr.client.solrj.SolrServerException: Error loading config name for collection phrases and I am trying to create collection test
If you are trying to create a collection on solr cloud with your custom configuration, you have to upload it first on to Zookeeper. Then you can create a collection using that config. You can also check what all configurations are there currently on the solrcloud through solr admin UI (http://localhost:8983/solr/#/~cloud?view=tree).
Uploading configuration to zookeeper:
Using zookeeper client in solr (solr-6.5.1\server\scripts\cloud-scripts)
zkcli -zkhost <zookeeper host> -cmd upconfig -confname <configname> -solrhome <solr home directory> -confdir <config directory path>
ex: zkcli -zkhost localhost:2181 -cmd upconfig -confname sampleconfig -solrhome ../solr -confdir ../../solr/configsets/sampleconfig/conf
Now that your configuration is uploaded, you can create a collection on solrcloud
start solr cloud with external zookeeper
solr start -c -z localhost:2181
Create collection
http://localhost:8983/solr/admin/collections?action=CREATE&name=<collectionname>&numShards=1&replicationFactor=1&collection.configName=<configname>
-e cloud is an example provided by SOLR and it works with implicit ZooKeepers.
For explicit ZooKeeper implementation refer either of the below ones
http://amn-solr.blogspot.in/
SolrCloud 5 and Zookeeper config upload
I have 3 servers running with each Solr 5.3 and Zookeeper (solr-cloud-01/zookeeper-01, solr-cloud-02/zookeeper-02 & solr-cloud-03/zookeeper-03)
Zookeeper is up and running and one of the servers is a leader, others are follower
# zkServer.sh status
If I try to create a solr collection, the config is created correctly in Zookeeper, but the core itself will not create, but timeout after 180s
# solr create_collection -c [collection_name] -d [config_name]
Connecting to ZooKeeper at zookeeper-01:2181,zookeeper-02:2181,zookeeper-03:2181 ...
Uploading /opt/solr/server/solr/configsets/[config_name]/conf for config
[collection_name] to ZooKeeper at zookeeper-01:2181,zookeeper-02:2181,zookeeper-03:2181
(or)
Re-using existing configuration directory [collection_name]
next:
Creating new collection '[collection_name]' using command:
http://localhost:8983/solr/admin/collections?action=CREATE&name=
[collection_name]&numShards=1&replicationFactor=1&maxShardsPerNode=1&
collection.configName=[collection_name]
ERROR: Failed to create collection '[collection_name]' due to:
create the collection time out:180s
The solr admin console log shows 2 identical error messages, one from SolrCore, the other from SolrDispatchFilter
null:org.apache.solr.common.SolrException: create the collection time out:180s
at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:239)
at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:170)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:675)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:443)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
If I then edit /opt/zookeeper/conf/zoo.cfg and uncomment the other zookeepers (reducing the quorum to 1 server)
server.1=zookeeper-01:2888:3888
#server.2=zookeeper-02:2888:3888
#server.3=zookeeper-03:2888:3888
And change the ZK_HOSTS option in /var/solr/solr.in.sh
#ZK_HOST="zookeeper-01:2181,zookeeper-02:2181,zookeeper-03:2181"
ZK_HOST="zookeeper-01:2181"
And restart both zookeeper and solr => The core is created (it was queued somehow?). But offline becausethe quorum was down (1 of 3 zookeeper nodes)
So then I experimented with a standalone solr / zookeeper setup (solr-cloud-01 / zookeeper-01)
# zkServer.sh status
JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: standalone
# zkServer.sh status
JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: standalone
I executed the same command:
# solr create_collection -c [collection_name] -d [config_name]
Connecting to ZooKeeper at zookeeper-01:2181 ...
Uploading /opt/solr/server/solr/configsets/[config_name]/conf for config [collection_name]
to ZooKeeper at zookeeper-01:2181
Creating new collection '[collection_name]' using command:
http://localhost:8983/solr/admin/collections?action=CREATE
&name=[collection_name]&numShards=1&replicationFactor=1&
maxShardsPerNode=1&collection.configName=[collection_name]
{
"responseHeader":{
"status":0,
"QTime":9417},
"success":{"":{
"responseHeader":{
"status":0,
"QTime":8869},
"core":"[collection_name]_shard1_replica1"}}}
So that works!
In conclusion, I have the feeling that some routes are not correctly configured, but I can't seem to find out which... Because Zookeeper seems to work and all individual solr instances as well
Here my hosts file:
127.0.0.1 localhost
10.0.0.1 solr-cloud-01
10.0.0.2 solr-cloud-02
10.0.0.3 solr-cloud-03
10.0.0.1 zookeeper-01
10.0.0.2 zookeeper-02
10.0.0.3 zookeeper-03
So, I finally found the answer!
After inspecting the /clusterstate.json via the zkCli.sh I saw that when disconnected 3 'rogue' replica's were mad to the standalone cluster. All pointing to 127.0.1.1, (which is a debian specific loopback to localhost, see https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_hostname_resolution)
The clue was in my hosts file.
So when I changed all reference to hostnames from 127.0.1.1 to the outside IP (in my case 10.0.0.x) it started working!
My new hosts file:
127.0.0.1 localhost
10.0.0.1 solr-cloud-01
10.0.0.2 solr-cloud-02
10.0.0.3 solr-cloud-03
10.0.0.1 zookeeper-01
10.0.0.2 zookeeper-02
10.0.0.3 zookeeper-03