Debugging jetty on a failed start - solr

I've deployed Solr in Jetty 9.0.6 on a Debian machine and sometimes when i try to start it up, it fails without giving me any clues why. The <>.stderrout.log file is allways empty.
How could I enable logging or verbose output for the start operation?
I'm doing:
$ sudo service jetty start
Starting Jetty . . . . . . . FAILED Tue Oct 15 11:10:18 CEST 2013
thanks!

With Jetty 9.0.6 you pass into the part of your service that calls ${jetty.home}/start.jar the extra --daemon command line variable and look for the start.log files. (These files are usually found at the ${jetty.logs}/start.log location.
Along with that, also include the -DDEBUG=true to enable debug of the startup process, as well as enabling the debug on Jetty itself (until the actual logger kicks in).
Finally, if you want the default Jetty StdErrLog to log at debug level too, add the -Dorg.eclipse.jetty.LEVEL=DEBUG to the command line as well.
To put this in non-service terms ...
[jetty-distribution-9.0.6.v20130930]$ java -jar start.jar --daemon -DDEBUG=true -Dorg.eclipse.jetty.LEVEL=DEBUG
As for how to do that with a debian service, I have no clue.

Related

dev_appserver.py slow start

After some time I needed to do changes on GAE Python (2.7) First Generation app, but when I develop and run locally, it's basically nonusable due to a very slow start.
These are printed logs:
INFO 2019-10-18 07:56:35,533 devappserver2.py:278] Skipping SDK update check.
INFO 2019-10-18 07:56:35,595 api_server.py:275] Starting API server at: http://localhost:36159
INFO 2019-10-18 07:56:35,599 dispatcher.py:270] Starting module "default" running at: http://127.0.0.1:8080
INFO 2019-10-18 07:56:35,600 admin_server.py:152] Starting admin server at: http://localhost:8000
INFO 2019-10-18 08:01:01,644 instance.py:294] Instance PID: 28496
What I notice that the last line (instance.py) is printed after ~5 minutes and only after that app responds to requests, not before. Interesting that the admin server (localhost:8000) is available right away. Of course, when I do some code change it automatically reloads and it repeats again.
Things I tried/found out:
it behaves like that on my all GAE projects Python First Gen.
tried to create a bare minimal version (webapp2 with one URL), clean virtualenv, still the same behavior
tried to reinstall Google Cloud SDK. delete the whole google-cloud-sdk folder and install again, no changes
tried to install older version of Cloud SDK
used clean VM and it works ok!!!, so it looks like there could be something wrong with my system (outside of SDK), but I'm not sure what.
It's interesting that the pause between the last two log lines is always about 5 minutes, not sure why exactly that time.
Python 2.7.14
OS: OpenSuse Leap 15.0
I'm running out of ideas so any advice would be appreciated.
I solved this accidentally.
I wanted to run Jupyter notebook, but I got the error:
error: [Errno 99] Cannot assign requested address
after debugging in /tornado/netutil.py, I noticed that it tries to work with IP 192.168.1.50 which I wasn't sure where did that come from, (probably I set it since I was playing with my home network some time ago), but when I deleted it from /etc/hosts, Jupyter, as well as GAE, works ok.
What a coincidence :)

Not able to start SOLR service

I have installed the solr service on a LINUX environment. Now trying to start the service using the below command
service solr start
After executing this command, am getting below error from the server
Waiting to see Solr listening on port 8080 [-] Still not seeing Solr listening on 8080 after 30 seconds!
tail: cannot open `/var/solr/logs/solr.log' for reading: No such file or directory
I created the solr.log file manually and placed it under the above mentioned path, But as soon, i issue the command "service solr start" . The solr.log file will be renamed and there won't be any new file created with the solr.log. hence the service fails to start. Could anyone let me know how to tackle this issue.
Thanks in advance.
I had a similar issue, and was able to find a hint in the /var/solr/logs/solr-8983-console.log
Originally I had been using Java 8, and Solr was working just fine for me.
When I switched to Java 11, Solr would have the issue you reported.
The log file contained the following:
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Unrecognized VM option 'UseParNewGC'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
I switched back to Java 8 and Solr started just fine.
First, did you use the procedure provided to install the Solr service page 461
Second, did you set the proper overrides to the environment defaults in a solr.in.sh script (p. 462 of the ref manual). You also need to make sure that the LOG4J_PROPS in the solr.in.sh file points to your log4j.properties file. Also make sure that SOLR_LOGS_DIR is pointing to the correct place.
If all that is correct, then check that the values in your log4j.properties file are set correctly (p. 468 of the ref manual).
You can get the reference manual here: https://www.apache.org/dyn/closer.lua/lucene/solr/ref-guide/ if you don't have it already.
I had a tough time getting Solr to run as a service, but in the end I simply wasn't reading carefully enough.

Starting and stopping App Engine instances with Docker

I've set up an App Engine project locally using Docker (on OSX), and have been running a server using the usual "gcloud preview app run app.yaml" command. From what I can tell, this keeps creating new images over and over again. After an hour or so of work I end up with something like 30 docker images, each taking 130MB.
Eventually I'm told I can no longer bind to localhost:8080. I tried killing all containers and images, but still cannot use localhost:8080 until I reboot.
Seems like I'm not using Docker/gcloud correctly. Anyone have an idea what I might be doing wrong? Is there another way I should be restarting App Engine instances other than hitting command C and running the "run" command again?
UPDATE: After looking closer, I noticed I'm getting this message when I run an app locally and a container is created: "http: Hijack is incompatible with use of CloseNotifier". I'm not familiar enough with Docker to understand what's going on here. All searches seem to point to Go, which I am not using.
UPDATE 2: Here is the trace:
Creating container...
INFO 2015-05-05 02:23:28,293 containers.py:560] Container 1564ce4344957114312d6d1dc696ffbb4176b40ace6dcff5e4239e13ee04a8f6 created.
Exception in thread Thread-2:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/Users/judeosborn/google-cloud-sdk/platform/google_appengine/google/appengine/tools/docker/containers.py", line 643, in _ListenToLogs
for line in log_lines:
File "/Users/judeosborn/google-cloud-sdk/./lib/docker/docker/client.py", line 225, in _multiplexed_response_stream_helper
socket = self._get_raw_response_socket(response)
File "/Users/judeosborn/google-cloud-sdk/./lib/docker/docker/client.py", line 167, in _get_raw_response_socket
self._raise_for_status(response)
File "/Users/judeosborn/google-cloud-sdk/./lib/docker/docker/client.py", line 119, in _raise_for_status
raise errors.APIError(e, response, explanation=explanation)
APIError: 500 Server Error: Internal Server Error ("http: Hijack is incompatible with use of CloseNotifier")
INFO 2015-05-05 02:23:28,606 module.py:1745] New instance for module "default" serving on:
http://localhost:8080
There's an ongoing issue with Docker 1.6.x [reference] that prevents gcloud to work well with Managed VMs (as you seem to be using). Easiest workaround until it gets fixed is to downgrade Docker in your development machine to version 1.5.0, which is the latest version known to work.
For Ubuntu, you can do something like:
$ curl -sSL https://get.docker.com/ubuntu | sed 's/lxc-docker/lxc-docker-1.5.0/' | sudo sh
For other Linux distros, you might have to modify that sed pattern, though.
On the other hand, if you're using Boot2Docker under Mac OS X, follow these steps:
Fully uninstall your previous Boot2Docker/Docker setup; there is a nice guide here
Reinstall Boot2Docker/Docker following instructions here. IMPORTANT: You MUST stop right after completing "Install Boot2Docker" step and before "Start the Boot2Docker Application". Once you get there, open up a terminal and execute the following commands:
$ mkdir ~/.boot2docker
$ echo 'ISOURL="https://github.com/boot2docker/boot2docker/releases/download/v1.5.0/boot2docker.iso"' > ~/.boot2docker/profile
At this point, you can continue with "Start the Boot2Docker Application" section and finish the installation. You should now have a valid Docker launchpad with which to start Managed VMs. It'd be nice to double check that you have the right versions installed by issuing:
$ boot2docker ssh docker version | egrep "(Client|Server) version"
The output should look like:
Client version: 1.5.0
Server version: 1.5.0
Now you can try again your original command:
$ gcloud preview app run app.yaml
Try running:
$ ps uax | egrep "gcloud|appserver"
If you see anything running, kill it... you may even need to kill -9 it.

Problems loading xdebug in wampserver2.0

I have an issue loading xdebug in apache server.
Here are my settings:
Apache 2.2.11
MySQL 5.1.36
PHP 5.2.0
Windows 7
Wampserver 2.0
php -i and php -m show that the xdebug module php_xdebug-2.1.0RC1-5.2-vc6.dll is installed.
However phpinfo() does not show that xdebug is installed. I have tried starting apache as a service, through the Wampserver UI, and by running httpd directly (as administrator) and I have checked permissions on the php_xdebug-2.1.0RC1-5.2-vc6.dll file.
There are no warnings in the logs. In the apache error log I get:
[Thu Jun 24 11:20:55 2010] [notice] Apache/2.2.11 (Win32) PHP/5.2.0 configured -- resuming normal operations
Has anyone any ideas?
Thank you,
John
I already do like you said but nothing happens
Using WAMPSERVER 2.1 , php_xdebug-2.1.0-5.3-vc6.dll (however I renamed it becoming php_xdebug.dll)
In php.ini, I also modified as instructions
extension=php_xdebug.dll
...
zend_extension = "C:/wamp/bin/php/php5.3.3/ext/php_xdebug.dll"
xdebug.remote_enable=1
xdebug.remote_handler=dbgp
xdebug.remote_host=127.0.0.1
xdebug.remote_port=9000
xdebug.idekey="netbeans-xdebug"
After starting the server again, there are no error in Apache log, everything seems work normal.Still do not see any information about xdebug from phpinfo. I totally gave up :(
Any ideas for my case, appreciated that.
I solved my issue as I was using wampserver2.0 and naively assumed I could use c:\wamp\bin\php\php{version#}\php.ini to affect my configuration.
In actual fact one must use c:\wamp\apache\Apache{version#}\bin\php.ini to configure php on the wamp server.
Hope this helps anyone else who might one day tear their hair out over this.
John

MacPorts Apache2 Stopped Launching on Boot

Something that I've noticed recently on two different machines is that Apache2 installed via MacPorts seems to have stopped launching when I boot up. The honest truth is that I can't swear it did so before, but it's something I think I'd notice because installing the LaunchDaemon is part of my install process. In fact, if I try to reload the LaunchDaemon, it fails:
$ sudo launchctl load -w /Library/LaunchDaemons/org.macports.apache2.plist
org.macports.apache2: Already loaded
Unless I start Apache manually (using sudo apachectl restart), grep'ing for either "apache2" or "httpd" in my process list only produces this:
$ sudo ps -ef | egrep "apache2|httpd"
0 52 1 0 0:00.06 ?? 0:00.08 /opt/local/bin/daemondo --label=apache2 --start-cmd /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper start ; --stop-cmd /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper stop ; --restart-cmd /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper restart ; --pid=none
1410639199 6960 6792 0 0:00.00 ttys001 0:00.00 egrep apache2|httpd
Looks like the daemon itself is in place, but no executable. As far as I know/can tell, the relevant executables (httpd and apachectl) are executable by everyone.
Has anyone else noticed this? Any ideas?
UPDATE
As requested below, I did execute launchctl list. The list is long and I'm not sure how to snip it, but suffice to say that no org.macports.* items are listed. That in itself is interesting because my MySQL daemon is loaded the same way. It works, but also doesn't appear in the list. Let me know if the entire output is really needed.
UPDATE
I assumed that I had executed launchctl list under sudo, but prompted by mipadi's comment below, I tried again ensuring that I did so and I assumed incorrectly. When executed under sudo, the MacPorts items appear:
51 - org.macports.mysql5
52 - org.macports.apache2
I'm not sure whether that will help, but it's a little more info nonetheless.
UPDATE
I've asked a different, but related, question at LaunchDaemons and Environment Variables. I'll update both questions as I learn more.
UPDATE
Today, based on mailing list input, I tried using a wildcard home directory. Academically, it's a little more inclusive than I'd like, but the practical reality is that I'm the only one using this computer; certainly the only one who'd have Apache config files laying around.
Include "/Users/*/Dropbox/Application Support/apache/conf.d.osx/*.conf"
Include "/Users/*/Library/Application Support/MacPorts/apache/conf.d/*.conf"
Unfortunately...
httpd: Syntax error on line 512 of /opt/local/apache2/conf/httpd.conf: Wildcard patterns not allowed in Include /Users/*/Dropbox/Application Support/apache/conf.d.osx/*.conf
I found my answer to this problem here:
https://trac.macports.org/ticket/36101
"I apparently fixed this when changing my local dnsmasq config. In /etc/hosts I added my servername (gala) to the loopback entry:
127.0.0.1 localhost gala
and then I changed ServerName in /opt/local/apache2/conf/httpd.conf to match:
ServerName gala
Apache now starts at boot for me."
Since I now know why Apache has stopped loading on startup, I'm going to articulate that answer and mark this question as answered. The reason Apache has stopped launching on boot is that I'm trying to share an httpd.conf file across systems. The config file needs to Include files from directories that exist within my home directory. Since the home directory is different on each machine, I was trying to reference the ${HOME} environment variable.
This works fine when manually starting after the machine is booted, but fails on startup because the environment variable isn't yet set. As mentioned above, see this question for more information.
Rob:
Had the same problem: "sudo launchctl load -w ..." started Apache2 while I was logged in, but did not work during startup (the "-w" should have taken care of that). Also, as you noticed, the daemon seems to be registered with launchctl. It will show up with "sudo launchctl list" and another "sudo launchctl load ..." will result in the error message.
I played with "sudo port load apache2" and "sudo port unload apache2", but could not get httpd running on reboot.
In the end, I got rid of the MacPorts startup item: "sudo port unload apache2", checked with "sudo launchctl list" that org.macports.apache2 is no longer registered for startup.
Afterwards, I followed the steps on http://diymacserver.com > Docs > Tiger > Starting Apache. I only had to adapt the path from /usr/local/... to /opt/local/...
Now the MacPorts Apache2 is starting fine with every reboot.
Good luck, Klaus
I found that my MacPorts apache2 was not starting on boot because of an “error” in my httpd.conf.
I was using
Listen 127.0.0.1:80
Listen 192.168.2.1:80
Listen 123.123.123.123:80 # Example IP, not the one I was really using
And in Console.app I was seeing
4/8/12 4:59:06.208 PM org.macports.apache2: (49)Can't assign requested address: make_sock: could not bind to address 192.168.2.1:80
4/8/12 4:59:06.208 PM org.macports.apache2: no listening sockets available, shutting down
4/8/12 4:59:06.208 PM org.macports.apache2: Unable to open logs
I tried adjusting permissions on all the log folders (despite the fact that logs were being written just fine when I manually started apache2) and that didn't help.
Even though the Apache Documentation for Listen clearly states
Multiple Listen directives may be used to specify a number of addresses and ports to listen to. The server will respond to requests from any of the listed addresses and ports.
I decided to try switching back to just using
Listen 80
And after doing so apache2 is starting on boot with no errors or warnings.
If you're using Subversion with Apache, you may find that Apache is not starting because the mod_dav_svn.so file has moved to /opt/local/libexec. You'll need to adjust your Apache startup files to account for the new location of this file.
In newer versions of MacPorts you can run sudo port load apache2 to instruct MacPorts to take care of the launchctl setup and automatically start the process. To stop the process run port unload.
After loading check /opt/local/apache2/logs/error_log for errors, including configuration issues.
In addition to my previous answer I have also found that sometimes Apache fails to start because something else with the system is not yet ready.
On one OS X Server machine I also use the DNS to create a “internal only” DNS name for the machine and that name is used in my Apache configuration. Sometimes when Apache tries to start the DNS server is not yet ready and Apache fails to load because the hostname isn’t valid.
I have also seen this on other non-Server systems without local DNS as well where something else required by Apache must not be ready yet.
One thing that has worked is to edit the apache2.wrapper located at /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper that MacPorts’ daemondo uses to start up Apache.
Edit the Start() function to add a sleep command to wait a bit before launching Apache.
Original (Lines 14-17 on my machine)
Start()
{
[ -x /opt/local/apache2/bin/apachectl ] && /opt/local/apache2/bin/apachectl start > /dev/null
}
With wait time added
Start()
{
[ -x /opt/local/apache2/bin/apachectl ] && sleep 10 && /opt/local/apache2/bin/apachectl start > /dev/null
}

Resources