I have changed some rights within a new core (solr 7.7.1) I am trying to configure. That has now led to failing to start the core on solr start. The rights look allright to me and everything else as well. Not different to other cores.
The error msg:
SolrCore Initialization Failures
test: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not load conf for core test: Error loading solr config from /var/solr/data/test/conf/solrconfig.xml
Please check your logs for more information
log:
core test is not available due to init failure: Could not load conf for core test: Error loading solr config from /var/solr/data/test/conf/solrconfig.xml
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1644)
The file itself:
ls -ls /var/solr/data/test/conf/solrconfig.xml
4 lrwxrwxrwx 1 solr solr 71 Mar 25 09:52 /var/solr/data/test/conf/solrconfig.xml -> /home/www/test/solrconfig.xml
Looks 1:1 with another core config. What am I missing?
Check, if the file /home/www/test/solrconfig.xml exits
Find out, under which user account solr is running. Use something like ps -efa | grep solr and check, if this user has access to /home/www/test/solrconfig.xml
If /var/solr/data/test/conf/solrconfig.xml and /home/www/test/solrconfig.xml resists at the same filesystem: try to use hardlinks instead of softlinks.
I'm trying to run solr as a SystemD service. When I start the service I get this warning then the solr server stops.
jun 22 16:20:07 solr_start[1488]: *** [WARN] *** Your Max Processes Limit is currently 14972.
jun 22 16:20:07 solr_start[1488]: It should be set to 65000 to avoid operational disruption.
jun 22 16:20:07 solr_start[1488]: If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
jun 22 16:20:09 solr_start[1488]: Warning: Available entropy is low. As a result, use of the UUIDField, SSL, or any other features that require
jun 22 16:20:09 solr_start[1488]: RNG might not work properly. To check for the amount of available entropy, use 'cat /proc/sys/kernel/random/entropy_avail'.
jun 22 16:20:14 solr_start[1488]: [146B blob data]
jun 22 16:20:14 solr_start[1488]: Started Solr server on port 8983 (pid=1579). Happy searching!
jun 22 16:20:15 solr_stop[1680]: Sending stop command to Solr running on port 8983 ... waiting up to 180 seconds to allow Jetty process 1579 to stop gracefully.
I checked the code at https://github.com/apache/lucene-solr/blob/master/solr/bin/solr#L1509.
I can see that solr runs a check on ulimit -u and ulimit -n.
The strange thing is that I made sure SystemD would run the solr server with the "solr" user.
[Unit]
Description=Apache SOLR
After=syslog.target network.target remote-fs.target nss-lookup.target systemd-journald-dev-log.socket
Before=multi-user.target
[Service]
User=solr
#PIDFile=/mnt/solrdata/solr-8983.pid
Environment=SOLR_INCLUDE=/opt/solr/bin/solr.in.sh
ExecStart=/opt/solr/bin/solr_start
ExecStop=/opt/solr/bin/solr_stop
#Restart=on-failure
[Install]
WantedBy=multi-user.target
And this "solr" user (that I configured especially to run Solr server) can create an unlimited number of processes and open an unlimited number of files.
[solr#xxx ~]# ulimit -n
unlimited
[root#xxx ~]# ulimit -u
unlimited
When SystemD starts solr I see it uses the correct user (the "solr" user I configured). Yet I see the error above.
When I run solr directly with the "solr" user (no with SystemD), it works.
Why are the user limitations applied at runtime when I run the Solr server myself different from the ones used when run by SystemD ???
According to this limit defined in /etc/security/limit.conf dosen't work with systemD.
To define new limit in a systemD unit file add those line in the service section:
LimitNOFILE=65000
LimitNPROC=65000
For a complete list see the man page
Version: Solr 6.3
OS: CentOs 7.3
After installation when running service solr restart, after 180 seconds I always get the same message before the INFO messages print out.
$ service solr restart
Archiving 1 old GC log files to /var/solr/logs/archived
Archiving 1 console log files to /var/solr/logs/archived
Rotating solr logs, keeping a max of 9 generations
Waiting up to 180 seconds to see Solr running on port 8983 [-] Still not seeing Solr listening on 8983 after 180 seconds!
What's weird is that the Solr server comes up and is accessible via the web interface almost immediately, however the full 180 seconds are spent waiting only to throw that message out each time. What causes this message and how can I get Solr identified to be running sooner?
Thanks!
This looks like either Solr not running on that port or it listens on specific interface and the checker scripts is using a default (localhost?) one. Can you run that with debug or check the definitions in the startup script?
I was getting the same error message trying to start solr: "Still not seeing Solr listening on 8983 after 180 seconds!". However, I couldn't access solr's web interface either. Checking the log files in /var/log/solr I read the following error message:
java.nio.file.AccessDeniedException: /tmp/start_6692986047430088693.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.createFile(Files.java:632)
at java.nio.file.TempFileHelper.create(TempFileHelper.java:138)
at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161)
at java.nio.file.Files.createTempFile(Files.java:897)
at org.eclipse.jetty.start.StartArgs.getMainArgs(StartArgs.java:596)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:205)
at org.eclipse.jetty.start.Main.start(Main.java:458)
at org.eclipse.jetty.start.Main.main(Main.java:76)
The problem was that I was inside a FreeBSD jail that had unusual permissions set on the /tmp directory (also on /var/tmp). Fixing the permissions on these directories solved the problem:
# chmod 1777 /tmp /var/tmp
I realize the cause of your problem is probably different. But since the error message is the same, I thought it could be useful to add this solution here.
I've deployed Solr in Jetty 9.0.6 on a Debian machine and sometimes when i try to start it up, it fails without giving me any clues why. The <>.stderrout.log file is allways empty.
How could I enable logging or verbose output for the start operation?
I'm doing:
$ sudo service jetty start
Starting Jetty . . . . . . . FAILED Tue Oct 15 11:10:18 CEST 2013
thanks!
With Jetty 9.0.6 you pass into the part of your service that calls ${jetty.home}/start.jar the extra --daemon command line variable and look for the start.log files. (These files are usually found at the ${jetty.logs}/start.log location.
Along with that, also include the -DDEBUG=true to enable debug of the startup process, as well as enabling the debug on Jetty itself (until the actual logger kicks in).
Finally, if you want the default Jetty StdErrLog to log at debug level too, add the -Dorg.eclipse.jetty.LEVEL=DEBUG to the command line as well.
To put this in non-service terms ...
[jetty-distribution-9.0.6.v20130930]$ java -jar start.jar --daemon -DDEBUG=true -Dorg.eclipse.jetty.LEVEL=DEBUG
As for how to do that with a debian service, I have no clue.
Something that I've noticed recently on two different machines is that Apache2 installed via MacPorts seems to have stopped launching when I boot up. The honest truth is that I can't swear it did so before, but it's something I think I'd notice because installing the LaunchDaemon is part of my install process. In fact, if I try to reload the LaunchDaemon, it fails:
$ sudo launchctl load -w /Library/LaunchDaemons/org.macports.apache2.plist
org.macports.apache2: Already loaded
Unless I start Apache manually (using sudo apachectl restart), grep'ing for either "apache2" or "httpd" in my process list only produces this:
$ sudo ps -ef | egrep "apache2|httpd"
0 52 1 0 0:00.06 ?? 0:00.08 /opt/local/bin/daemondo --label=apache2 --start-cmd /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper start ; --stop-cmd /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper stop ; --restart-cmd /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper restart ; --pid=none
1410639199 6960 6792 0 0:00.00 ttys001 0:00.00 egrep apache2|httpd
Looks like the daemon itself is in place, but no executable. As far as I know/can tell, the relevant executables (httpd and apachectl) are executable by everyone.
Has anyone else noticed this? Any ideas?
UPDATE
As requested below, I did execute launchctl list. The list is long and I'm not sure how to snip it, but suffice to say that no org.macports.* items are listed. That in itself is interesting because my MySQL daemon is loaded the same way. It works, but also doesn't appear in the list. Let me know if the entire output is really needed.
UPDATE
I assumed that I had executed launchctl list under sudo, but prompted by mipadi's comment below, I tried again ensuring that I did so and I assumed incorrectly. When executed under sudo, the MacPorts items appear:
51 - org.macports.mysql5
52 - org.macports.apache2
I'm not sure whether that will help, but it's a little more info nonetheless.
UPDATE
I've asked a different, but related, question at LaunchDaemons and Environment Variables. I'll update both questions as I learn more.
UPDATE
Today, based on mailing list input, I tried using a wildcard home directory. Academically, it's a little more inclusive than I'd like, but the practical reality is that I'm the only one using this computer; certainly the only one who'd have Apache config files laying around.
Include "/Users/*/Dropbox/Application Support/apache/conf.d.osx/*.conf"
Include "/Users/*/Library/Application Support/MacPorts/apache/conf.d/*.conf"
Unfortunately...
httpd: Syntax error on line 512 of /opt/local/apache2/conf/httpd.conf: Wildcard patterns not allowed in Include /Users/*/Dropbox/Application Support/apache/conf.d.osx/*.conf
I found my answer to this problem here:
https://trac.macports.org/ticket/36101
"I apparently fixed this when changing my local dnsmasq config. In /etc/hosts I added my servername (gala) to the loopback entry:
127.0.0.1 localhost gala
and then I changed ServerName in /opt/local/apache2/conf/httpd.conf to match:
ServerName gala
Apache now starts at boot for me."
Since I now know why Apache has stopped loading on startup, I'm going to articulate that answer and mark this question as answered. The reason Apache has stopped launching on boot is that I'm trying to share an httpd.conf file across systems. The config file needs to Include files from directories that exist within my home directory. Since the home directory is different on each machine, I was trying to reference the ${HOME} environment variable.
This works fine when manually starting after the machine is booted, but fails on startup because the environment variable isn't yet set. As mentioned above, see this question for more information.
Rob:
Had the same problem: "sudo launchctl load -w ..." started Apache2 while I was logged in, but did not work during startup (the "-w" should have taken care of that). Also, as you noticed, the daemon seems to be registered with launchctl. It will show up with "sudo launchctl list" and another "sudo launchctl load ..." will result in the error message.
I played with "sudo port load apache2" and "sudo port unload apache2", but could not get httpd running on reboot.
In the end, I got rid of the MacPorts startup item: "sudo port unload apache2", checked with "sudo launchctl list" that org.macports.apache2 is no longer registered for startup.
Afterwards, I followed the steps on http://diymacserver.com > Docs > Tiger > Starting Apache. I only had to adapt the path from /usr/local/... to /opt/local/...
Now the MacPorts Apache2 is starting fine with every reboot.
Good luck, Klaus
I found that my MacPorts apache2 was not starting on boot because of an “error” in my httpd.conf.
I was using
Listen 127.0.0.1:80
Listen 192.168.2.1:80
Listen 123.123.123.123:80 # Example IP, not the one I was really using
And in Console.app I was seeing
4/8/12 4:59:06.208 PM org.macports.apache2: (49)Can't assign requested address: make_sock: could not bind to address 192.168.2.1:80
4/8/12 4:59:06.208 PM org.macports.apache2: no listening sockets available, shutting down
4/8/12 4:59:06.208 PM org.macports.apache2: Unable to open logs
I tried adjusting permissions on all the log folders (despite the fact that logs were being written just fine when I manually started apache2) and that didn't help.
Even though the Apache Documentation for Listen clearly states
Multiple Listen directives may be used to specify a number of addresses and ports to listen to. The server will respond to requests from any of the listed addresses and ports.
I decided to try switching back to just using
Listen 80
And after doing so apache2 is starting on boot with no errors or warnings.
If you're using Subversion with Apache, you may find that Apache is not starting because the mod_dav_svn.so file has moved to /opt/local/libexec. You'll need to adjust your Apache startup files to account for the new location of this file.
In newer versions of MacPorts you can run sudo port load apache2 to instruct MacPorts to take care of the launchctl setup and automatically start the process. To stop the process run port unload.
After loading check /opt/local/apache2/logs/error_log for errors, including configuration issues.
In addition to my previous answer I have also found that sometimes Apache fails to start because something else with the system is not yet ready.
On one OS X Server machine I also use the DNS to create a “internal only” DNS name for the machine and that name is used in my Apache configuration. Sometimes when Apache tries to start the DNS server is not yet ready and Apache fails to load because the hostname isn’t valid.
I have also seen this on other non-Server systems without local DNS as well where something else required by Apache must not be ready yet.
One thing that has worked is to edit the apache2.wrapper located at /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper that MacPorts’ daemondo uses to start up Apache.
Edit the Start() function to add a sleep command to wait a bit before launching Apache.
Original (Lines 14-17 on my machine)
Start()
{
[ -x /opt/local/apache2/bin/apachectl ] && /opt/local/apache2/bin/apachectl start > /dev/null
}
With wait time added
Start()
{
[ -x /opt/local/apache2/bin/apachectl ] && sleep 10 && /opt/local/apache2/bin/apachectl start > /dev/null
}