Reload Solr configuration without multicore - solr

Is it possible to reload Solr configuration without setting up Multicore or restarting the servlet container?
I would like to tweak some <analyzer> chains with the analysis tab in the admin, and tweak the parameters to my <requestHandler>, but having to restart the servlet container after every small change to schema.xml or solrconfig.xml is a bit of a pain and time consuming.

There is always a core in SOLR. By default, SOLR instance creates a core named collection1. In case you have a single core and not sure how to reload at runtime, you can use this,
http://localhost:8080/solr/admin/cores?action=RELOAD&core=collection1

As best as I can tell, online reloading requires a Multicore configuration, which it turns out isn't too hard:
Put this solr.xml into the solr home directory
<solr persistent="false" sharedLib="lib">
<cores adminPath="/admin/cores" defaultCoreName="core0">
<core name="core0" instanceDir="." />
</cores>
</solr>
Restart the servlet container.
Hit a URL like this to reload the configuration:
http://localhost:8983/solr/admin/cores?action=RELOAD&core=core0
To remove the rest of the friction, you can set it up to automatically reload the configuration by running the following script within the conf directory.
get_on_fsevent.rb "http://localhost:8983/solr/admin/cores?action=RELOAD&core=core0"
get_on_fsevent.rb:
#!/usr/bin/env ruby
require 'rubygems'
require 'rb-fsevent'
require 'net/http'
require 'uri'
uri = URI.parse(ARGV.first)
fsevent = FSEvent.new
fsevent.watch Dir.pwd do |directories|
puts "Detected change. Requesting #{ARGV.first}"
puts Net::HTTP.get_response(uri)
end
fsevent.run

Related

can data-import-handler use HikariCP?

I'm using solr4.5.1 in work.
The trouble is that a lot of getConnection occurred, when I execute data-import(full-import). So I thought if HikariCP could be used in data-import, but I haven't found similar problem.
Is it possible? If so please advice.
Solr 4.5.1 with Tomcat
data-config.xml
<dataSource driver="oracle.jdbc.driver.OracleDriver"
name="jdbc"
url="jdbc:oracle:thin:#address/mydb"
user="user" password="pass"/>
Heavily borrowed from David H Nebinger's post: Tomcat and HikariCP.
Install HikariCP
To make use of JNDI, you need to declare the JNDI datasource with all its' settings, password and cache options within the JNDI declaration. This has nothing to do with Solr at this point. This is a Tomcat mechanism. How you do this is described in this tutorial that also makes use of HikariCP.
First is to download the .zip or .tar.gz file from http://brettwooldridge.github.io/HikariCP/. This is actually a source release that you'll need to build yourself.
Second option is to download the built jar from a source like Maven Central, https://mvnrepository.com/artifact/com.zaxxer/HikariCP
Once you have the jar, copy to the Tomcat lib/ext directory. Note that Hikari CP does have a dependency on SLF4J, so you'll need to put that jar into lib/ext too.
Do not forget to place your datasource's JDBC driver in the lib/ext folder.
Configure the JNDI datasource
Location of your JNDI datasource <Resource /> definitions depends upon the scope for the connections. You can define them globally by specifying them in Tomcat's conf/server.xml and conf/context.xml, or you can scope them to individual applications by defining them in conf/Catalina/localhost/WebAppContext.xml (where WebAppContext is the web application context for the app, basically the directory name from Tomcat's webapps directory).
Create the file conf/Catalina/localhost/ROOT.xml if it doesn't already exist. Use the table from https://github.com/brettwooldridge/HikariCP#popular-datasource-class-names to find your data source class name, we'll need it when we define the element.
<Resource name="jdbc/SolrPool" auth="Container"
factory="com.zaxxer.hikari.HikariJNDIFactory"
type="javax.sql.DataSource"
minimumIdle="5"
maximumPoolSize="10"
connectionTimeout="300000"
dataSourceClassName="oracle.jdbc.pool.OracleDataSource"
dataSource.url="jdbc:oracle:thin:#address/mydb"
dataSource.implicitCachingEnabled="true"
dataSource.user="user"
dataSource.password="pass" />
Make use of the JNDI datasource in Solr
After you have followed this tutorial, you will need to use the declared JNDI datasource, this would be like described in the Solr Wiki:
<dataSource
jndiName="java:jdbc/SolrPool"
type="JdbcDataSource"
user="" password=""/> <!-- leave out user/password here -->

Solrcloud multicore configuration

I have a standalone Solr instance with 4 different cores working fine using the embedded Jetty server. I configured the cores for v4.10.3 but since I moved to v5.1 and all seems to work fine without any changes.
Before going into production, I need to set it up as a Solrcloud installation, initially with 2 nodes (two different machines) with 1 shard per node (to keep it simple). I have been trying to get it to work but I have not been able to do it.
I tried to run it like this (I think using start.jar is not the preferred way), having read that Solr will look for multiple configured cores in any nested folders (which works for standalone Solr):
java -DzkRun -DnumShards=2 -Dbootstrap_confdir=solr/ -jar start.jar
but that did not work, it does not find the needed solrconfig.xml file.
My Solr directory looks like this:
My solr.xml file is the standard one:
<solr>
<solrcloud>
<str name="host">${host:}</str>
<int name="hostPort">${jetty.port:8983}</int>
<str name="hostContext">${hostContext:solr}</str>
<int name="zkClientTimeout">${zkClientTimeout:30000}</int>
<bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
</solrcloud>
<shardHandlerFactory name="shardHandlerFactory"
class="HttpShardHandlerFactory">
<int name="socketTimeout">${socketTimeout:0}</int>
<int name="connTimeout">${connTimeout:0}</int>
</shardHandlerFactory>
</solr>
Each core looks like this:
And the core.properties just has the name of the core:
name=users
My question is:
How do I start Solrcloud v5.1 so the 4 cores are picked up?
In SolrCloud each of your Core will become a Collection.
Each Collection will have its own set of Config Files and data.
You might find this helpful Moving multi-core SOLR instance to cloud
Solr 5.0 (onwards) has made some changes on how to create a SolrCloud setup with shards, and how to add collections etc.
Everything listed below is my understanding of the Solr Reference Guide. I will highly recommend going through it thoroughly.
https://cwiki.apache.org/confluence/display/solr/Apache+Solr+Reference+Guide
I setup my servers on a Linux(CentOS) server, but the steps can be used to setup solr on Windows system also. For example, there is solr.cmd file instead of solr.sh
Here are the steps I followed to create a simple two shard SolrCloud setup.
Setup the zookeeper ensemble. I am assuming you are trying to use the
embedded ZK in solr. For a production system, it is highly
recommended to create a external ZK ensemble. You can find steps to install a external ensemble in this section of reference guid
Download solr to /opt folder.
Extract the install file ONLY.
tar xzf solr-5.0.0.tgz solr-5.0.0/bin/install_solr_service.sh --strip components=2
This command will install solr on your system
sudo bash ./install_solr_service.sh solr-5.0.0.tgz
The above command will create a new user called "solr" if it does not exist.
These are some of the default options it will assume. You can view this in /var/solr/solr.in.sh . This is the include file where you can specify other options.
* SOLR_PID_DIR=/var/solr
* SOLR_HOME=/var/solr/data
* LOG4J_PROPS=/var/solr/log4j.properties
* SOLR_LOGS_DIR=/var/solr/logs
* SOLR_PORT=8983
Running install_solr_service start in the above step will start a solr server. Stop the server using service solr stop before doing any of the changes below.
Change Java heap value
SOLR_HEAP="3g"
This will set Xmx and Xms as 3GB . (optional)
This variable is not mentioned in the solr.in.sh file in Solr 5.1 . Its a bug and has been fixed, will be released in next version.
SOLR_MODE="solrcloud" Required
this is what you need start solr in cloud mode.
ZK_HOST=ZK1:2181,ZK2:2181,ZK3:2181 Required
(replace zk with you zookeeper host names)
Running the install_solr_service.sh command also creates a init.d file as /etc/init.d/solr
This init.d script in turn calls the /opt/solr/bin/solr script and includes all the variables from /var/solr/solr.in.sh
Once you have made the above changes, start solr again using service solr start
You can check the status using service solr status
Creating Collections Shards and Replicas
- All shard, collection, replica related commands are now made using Collections API.
Before creating a collection a config folder should be uploaded to ZK .
This can be done using the zkcli.sh script in the solr folder (not on the zookeeper servers)
Folder: /opt/solr/server/scripts/cloud-scripts
The command to upload the confg folder is
sh zkcli.sh -cmd upconfig -zkhost zk1:2181,zk2:2181,zk3:2181 -confname yourconfigname -confdir /var/solr/configs/conf
You will run this command 4 times for each of your 4 cores, each time changing the path of the conf folder and config name.
This will upload all the config files in conf folder with the name 'yourconfigname' in zookeeper.
Creating a collection
I used the following command to create a new collection.
http://1.1.1.1:8983/solr/admin/collections?action=CREATE&name=yourcollectionname&numShards=2&replicationFactor=1&maxShardsPerNode=1&createNodeSet=1.1.1.1:8983_solr,2.2.2.2:8983_solr&collection.configName=yourconfigname
Happy Searching!
SolrCloud does not use configuration files stored in core conf directory. To make your cores visible in SolrCloud structure you need to upload the configuration files to ZooKeeper and keep it manage the files to you. All the time a Solr instance comes up it get the configuration files stored in ZooKeeper. This way your cores doesn't need to have conf directory to work. To upload your core configuration files to ZooKeeper follow the link bellow and take a look at Upload a configuration directory
https://cwiki.apache.org/confluence/display/solr/Command+Line+Utilities

Solr - How to change the default core for Healthcheck

I am working with Solr 4.7.1
With a fresh installation, the following link:
http://localhost:8983/solr/admin/ping
Only works when core collection1 exists in the following folder hierarchy:
/example/solr/collection1
I have created a new core: collection2
I want this link to work when I replace core collection1 with core collection2:
/example/solr/collection2
Problem:
When I replace collection1 with collection2, the following link gives me error 404:
http://localhost:8983/solr/admin/ping
It seems collection1 is set as a default somewhere. But I don't know how to change it.
Thanks.
i tried using this url instead of solr/admin/ping. It seems to work fine
http://localhost:8983/solr/collection1/admin/ping
in your case, you might want to change it to collection2 in the url above and try.
To change default core names in Solr 4.x you need to define them in solr.xml which should be placed in your Solr solr.home. Here is example:
<solr persistent="true" sharedLib="lib">
<cores adminPath="/admin/cores">
<core name="core0" instanceDir="core0" />
<core name="core1" instanceDir="core1" />
</cores>
</solr>
In Solr 5.x you can simple create a core.properties anywhere under solr.home and it should be discovered automatically and define the name by:
name=my_core_name
Then double check your other XML Solr configuration files that you don't hardcode any old names within the files it-self.

Solr (4.4+) solrconfig.xml location when creating cores

I'm Trying to setup a multi core solr server for our webapplication but i'm having trouble creating new core through the coreadmin service.
I'm using Solr-4.4 because 4.3 ran into problems persisting the cores in solr.xml (datadir wasn't preserved) So i'm using the new Solr.xml configuration 4.4 and beyond
My solr.xml currently looks like:
<solr>
<str name="coreRootDirectory">default-instance/cores/</str>
</solr>
solrconfig.xml is located at (solrhome)/default-instance/conf/solrconfig.xml
When trying to create a core with the url
http:/example.org/solr/admin/cores?action=CREATE&name=test-name&schema=schema-test.xml&loadOnStartup=false
gives me the error:
Error CREATEing SolrCore 'test-name': Unable to create core: test-name
Caused by: Can't find resource 'solrconfig.xml' in classpath or
'default-instance/cores/test-name/conf/', cwd=/var/lib/tomcat7
The following seems to work:
http:/example.org/solr/admin/cores?action=CREATE&name=test-name&schema=schema-test.xml&loadOnStartup=false&config=/absolute/file/path/to/solrconfig.xml
The problem is this only seems to work with a absolute path (or possibly a relative path from /var/lib/tomcat7) which is not a workable solution.
What i'm looking for is a way to place solrconfig.xml so it can be used to create new cores with that config (or a way the create those cores with the current location).
More or less the same will be needed for schemas
This worked. Ran on command line and was viewable in admin console:
solr create -c (name for core or collection)
See README.txt for more info.
In my case I took advantage of the Core Discovery feature in 4.4+, rather than creating the core using the management web interface.
This simply involved copying the example collection1 folder from the examples directory (which I usually use as a starting point).
Then I had to make sure that there is core.properties in the root of my new core with name=<new core name> inside. Solr automatically detected the new core and allowed me to use it without any fuss.
This avoided the trouble of having to copying solrconfig.xml and schema.xml into any special location.
I had the same problem: solrconfig.xml was not in the classpath. I solved it by copying my configuration file templates into the classpath.
So I took a look at http://localhost:8983/solr/#/~java-properties to see solrs classpath definition and then i copied the template solrconfig.xml and schema.xml into the folder C:\servers\solr-4.4.0\example\resources. Furthermore i copied all the stopwords stuff there...
This solution is not a fully satisfying, but it works. Adding another path to the classpath should work, too. I'm slightly astonished that no default configuration for new cores can be declared within solr.xml
I recommend the new Config Sets for this use case.
If you place your schema.xml and solrconfig.xml (and other config files like stopwords etc.) in a directory $SOLR_HOME/configsets/myConfig/conf, you can create a new core with this config by calling:
http://solr/admin/cores?action=CREATE&name=mycore&instanceDir=my_instance&configSet=myConfig
See https://cwiki.apache.org/confluence/display/solr/Config+Sets
But they are not available until Solr 4.8, see https://issues.apache.org/jira/browse/SOLR-4478

How to set solr/home in linux OS?

I know how to configure solr.home by using Tomcat 6, but I don't know how to set solr.home by using Glassfish(V2.1). I have tried to set the solr.home in .profile as fellows:
export solr.home=/home/huenzhao/search/solr
export solr/home=/home/huenzhao/search/solr
export solr.solr.home=/home/huenzhao/search/solr
export JAVA_OPTS=$JAVA_OPTS -Dsolr.solr.home=/home/huenzhao/search/solr
and they all not work. The error is:
HTTP Status 500 - Severe errors in solr configuration. Check your log
files for more detailed information on what may be wrong. If you want
solr to continue after configuration errors, change:
false in null
------------------------------------------------------------- java.lang.RuntimeException: Can't find resource 'solrconfig.xml' in
classpath or 'solr/conf/',
cwd=/home/huenzhao/search/glassfish/domains/domain1/config at
org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:194)
at
org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:162)
at org.apache.solr.core.Config.(Config.java:100) at
org.apache.solr.core.SolrConfig.(SolrConfig.java:113) at
org.apache.solr.core.SolrConfig.(SolrConfig.java:70) at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:117)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
at
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:273)
at
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:385)
at
org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:119)
at
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4498)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:5317)
at com.sun.enterprise.web.WebModule.start(WebModule.java:353) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:989)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:973)
at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:704)
at
com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1627)
at
com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1232)
at
com.sun.enterprise.server.WebModuleDeployEventListener.moduleDeployed(WebModuleDeployEventListener.java:182)
at
com.sun.enterprise.server.WebModuleDeployEventListener.moduleDeployed(WebModuleDeployEventListener.java:278)
at
com.sun.enterprise.admin.event.AdminEventMulticaster.invokeModuleDeployEventListener(AdminEventMulticaster.java:1005)
at
……
Anybody knows?
if you are running solr inside tomcat as a container you can specify the solr home inside the XML descriptor for this webapp. (my terminology for this is probably a little off).
I've got xml fragments for each solr instance I want to run and they specify their own local solr home directory inside the xml fragment. The fragments live at /conf/Catalina/localhost and each one manages a solr instance. This way I can have multiple solr instances on the same machine each with their own solr home variable.
The info is here:
http://wiki.apache.org/solr/SolrTomcat
In paticular
Create a Tomcat Context fragment to
point docBase to the
$SOLR_HOME/apache-solr-1.3.0.war file
and solr/home to $SOLR_HOME:
Symlink
or place the file in
$CATALINA_HOME/conf/Catalina/localhost/solr-example.xml,
where Tomcat will automatically pick
it up. Tomcat deletes the file on
undeploy (which happens automatically
if the configuration is invalid).
Try to set the following:
export JAVA_OPTS="$JAVA_OPTS -Dsolr.solr.home=/home/huenzhao/search/solr/"
Try setting a Java environment parameter from Java or edit your VM configuration:
System.setProperty("solr.solr.home", "/home/user/apache-solr-1.4/example/solr");
In my case I simply copied the 'solr' folder to glassfish/domains/domain1/config and it worked.

Resources