We are trying to deploy one of our node js applications(AngularJS/NodeJS tech stack) to Production environment. Randomly, some of the ec2 instances just goes down as node process is getting killed. But no application/system level error messages are getting logged. We are unable to replicate this issue in local/dev/it environments. Has anyone faced something similar? Any help is much appreciated. Thanks!
I know this is an old question, just in case someone else run into the same problem.
Most probably is that your a running out of memory, as explained here. You can verify with dmesg command (linux). To fix it you can add more memory or add a swap volume.
One way is following this answer:
sudo /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024
sudo /sbin/mkswap /var/swap.1
sudo chmod 600 /var/swap.1
sudo /sbin/swapon /var/swap.1
// enable after reboot
/var/swap.1 swap swap defaults 0 0
By default, nodejs application would be stopped when no client connect to it, you could use some tools like forever to keep your nodejs app running
Related
So I downloaded ArangoDB via homebrew yesterday. The documentation said to use the following to get the server started:
/usr/local/Cellar/arangodb/<VERSION>/sbin/arangod &
I ended up using the prompt shown on the terminal
/usr/local/opt/arangodb/sbin/arangod
Don't know if that necessarily makes a difference. Nonetheless, I got the server working.
Then I wanted to set up authentication. The docs did not say how to stop a server. Since I didn't know how to, I just closed the terminal window
Then I ran
/usr/local/opt/arangodb/sbin/arango-secure-installation
This then gave me an error
FATAL [...] database is locked by process ..: please stop it first and check that the lockfile 'usr/local/var/lib/arangodb3/LOCK' goes away. If you are sure no other arangod process is running, please remove the lockfile '/usr/local/var/lib/arangodb3/LOCK' and try again
How do I fix this? I had the webgui running but closing that didn't make a difference.
Another stack overflow question said to use the following commands:
brew services stop arangod
brew services start arangod
brew services restart arangod
None of those worked?
So, two questions:
a) how do I solve this problem
b) and, in the future, how do I properly terminate a server?
Thanks
You can use "brew services start/stop/restart arangod". It is no longer necessary to start arangod explicitly. If it has been started explicitly just do a "killall arangod" to stop it gracefully and start it using "brew services start".
I am trying to upload a yesod Docker container on Google App Engine. The source code is here and the Docker image is here.
I followed the documentation in the Custom runtime quickstart, and when invoking gcloud app deploy the app builds fine after increasing the build timeout, but the container either the readiness check when trying to start or shows the following timeout message:
ERROR: (gcloud.app.deploy) Operation [apps/meeshkan-github-webhook-router/operations/xxxx-xxxx-xxxx] timed out. This operation may still be underway.
I have tried experimenting with several things, including a manual readiness check, creating an /_ah/health endpoint, and increasing the timeout of the readiness check all the way to 1799 seconds, but none of these actions seem to work.
One issue may be the size of the container (it is 3.2gb), and I could try to prune it down, but I'd only do that if someone could confirm that container size is a contributing factor to deployment problems. Other than that, I'm not sure what could be causing this failure. The docker image starts fine on our local machines.
Thanks in advance for your help and suggestions!
The issue turned out to be that, because I was building on Windows, images built using Docker Desktop on Windows gave all shell scripts executable permission automatically, whereas Docker on Linux needs shell scripts to be given the executable permission. By adding this line to my Dockerfile:
RUN chmod +x /usr/src/app/run.sh
Everything worked fine!
I have an AngularJS application that does an ajax call but it results in a chromium error:
net::ERR_NETWORK_CHANGED
I tried to disable any adapters that I don't need. I have multiple ones and multiple dockers containers running.
I disabled ipv6 on each adapter. I don't use any proxy and use default Chromium browser without any addon nor browser profile.
Disabled Wifi interface, only using ethernet.
Any idea how to fix this?
I was constantly getting ERR_NETWORK_CHANGED.
This is what finally worked for my current browsers:
Chromium, Opera and FlashPeak Slimjet.
sudo service docker stop
The following actions did not solve my issue:
Checked modem, router, and cables to isolate the issue.
Disabled IPv6 from my wired Network
Commands:
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
After I stopped Docker, I am not getting any more console errors.
I hope this can help someone saving hours of annoying troubleshooting.
Ron.
sudo service docker stop
But this is not a solution because I need docker in my daily work.
So I found out that docker networks cause this problem
docker network prune helped me
Or try to delete one by one except of none, bridge, host
Based on the original answers, I want to go into more detail what fixed it in my case.
Stopping the docker service sudo service docker stop in my case fixed the issue.
The underlying issue one of my docker-compose setups having restart=always.
Unfortunatly I had a bug causing a container to terminate and restart. This restart caused the network change.
It is determinable by running docker ps and noticing the container restarted.
I fixed the problem and ran docker-compose down for my docker compose setup. Both actions would fix it independently.
Furthermore a Bugreport for chromium exists regarding this issue, but it has the state wontfix.
I'm trying to get Tomcat 6 running on Fedora 14. I'm quite new to this sort of thing, though I have managed to get Apache running before.
I suspect I've made mistakes on some quite basic levels, because the test page (http://localhost:8080) can't be found.
If I try to check the service's status, as root or as an ordinary user, this is what I get:
tomcat6 (pid) is running...[OK]
tomcat6 lockfile exists but process is not running [FAILED]
So I'm confused, because I seem to be being told that Tomcat 6 both is and isn't running. If anyone can tell me what this really is telling me, and where my suspicions should turn next, I'd be very grateful.
There's no mention of any of this sort of thing in the docs.
I had the same problem. I found beetstra's answer to be necessary but insufficient. I also needed to change some permissions. As indicated (mostly) by this bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=708694
# get rid of pid
rm /var/run/tomcat6.pid
# give tomcat sufficient permissions
chown tomcat:tomcat -R /usr/share/tomcat6/conf/
chown tomcat:tomcat -R /usr/share/tomcat6/conf
chown tomcat:tomcat -R /usr/share/tomcat6/logs
After that I could start tomcat successfully:
service tomcat6 start
Starting tomcat6: [ OK ]
I had the same problem, and it was caused by a stale lock file. The process was not really running, there was just a file that said it might be running. The solution was to remove that file:
rm /var/run/tomcat6.pid
Something that I've noticed recently on two different machines is that Apache2 installed via MacPorts seems to have stopped launching when I boot up. The honest truth is that I can't swear it did so before, but it's something I think I'd notice because installing the LaunchDaemon is part of my install process. In fact, if I try to reload the LaunchDaemon, it fails:
$ sudo launchctl load -w /Library/LaunchDaemons/org.macports.apache2.plist
org.macports.apache2: Already loaded
Unless I start Apache manually (using sudo apachectl restart), grep'ing for either "apache2" or "httpd" in my process list only produces this:
$ sudo ps -ef | egrep "apache2|httpd"
0 52 1 0 0:00.06 ?? 0:00.08 /opt/local/bin/daemondo --label=apache2 --start-cmd /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper start ; --stop-cmd /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper stop ; --restart-cmd /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper restart ; --pid=none
1410639199 6960 6792 0 0:00.00 ttys001 0:00.00 egrep apache2|httpd
Looks like the daemon itself is in place, but no executable. As far as I know/can tell, the relevant executables (httpd and apachectl) are executable by everyone.
Has anyone else noticed this? Any ideas?
UPDATE
As requested below, I did execute launchctl list. The list is long and I'm not sure how to snip it, but suffice to say that no org.macports.* items are listed. That in itself is interesting because my MySQL daemon is loaded the same way. It works, but also doesn't appear in the list. Let me know if the entire output is really needed.
UPDATE
I assumed that I had executed launchctl list under sudo, but prompted by mipadi's comment below, I tried again ensuring that I did so and I assumed incorrectly. When executed under sudo, the MacPorts items appear:
51 - org.macports.mysql5
52 - org.macports.apache2
I'm not sure whether that will help, but it's a little more info nonetheless.
UPDATE
I've asked a different, but related, question at LaunchDaemons and Environment Variables. I'll update both questions as I learn more.
UPDATE
Today, based on mailing list input, I tried using a wildcard home directory. Academically, it's a little more inclusive than I'd like, but the practical reality is that I'm the only one using this computer; certainly the only one who'd have Apache config files laying around.
Include "/Users/*/Dropbox/Application Support/apache/conf.d.osx/*.conf"
Include "/Users/*/Library/Application Support/MacPorts/apache/conf.d/*.conf"
Unfortunately...
httpd: Syntax error on line 512 of /opt/local/apache2/conf/httpd.conf: Wildcard patterns not allowed in Include /Users/*/Dropbox/Application Support/apache/conf.d.osx/*.conf
I found my answer to this problem here:
https://trac.macports.org/ticket/36101
"I apparently fixed this when changing my local dnsmasq config. In /etc/hosts I added my servername (gala) to the loopback entry:
127.0.0.1 localhost gala
and then I changed ServerName in /opt/local/apache2/conf/httpd.conf to match:
ServerName gala
Apache now starts at boot for me."
Since I now know why Apache has stopped loading on startup, I'm going to articulate that answer and mark this question as answered. The reason Apache has stopped launching on boot is that I'm trying to share an httpd.conf file across systems. The config file needs to Include files from directories that exist within my home directory. Since the home directory is different on each machine, I was trying to reference the ${HOME} environment variable.
This works fine when manually starting after the machine is booted, but fails on startup because the environment variable isn't yet set. As mentioned above, see this question for more information.
Rob:
Had the same problem: "sudo launchctl load -w ..." started Apache2 while I was logged in, but did not work during startup (the "-w" should have taken care of that). Also, as you noticed, the daemon seems to be registered with launchctl. It will show up with "sudo launchctl list" and another "sudo launchctl load ..." will result in the error message.
I played with "sudo port load apache2" and "sudo port unload apache2", but could not get httpd running on reboot.
In the end, I got rid of the MacPorts startup item: "sudo port unload apache2", checked with "sudo launchctl list" that org.macports.apache2 is no longer registered for startup.
Afterwards, I followed the steps on http://diymacserver.com > Docs > Tiger > Starting Apache. I only had to adapt the path from /usr/local/... to /opt/local/...
Now the MacPorts Apache2 is starting fine with every reboot.
Good luck, Klaus
I found that my MacPorts apache2 was not starting on boot because of an “error” in my httpd.conf.
I was using
Listen 127.0.0.1:80
Listen 192.168.2.1:80
Listen 123.123.123.123:80 # Example IP, not the one I was really using
And in Console.app I was seeing
4/8/12 4:59:06.208 PM org.macports.apache2: (49)Can't assign requested address: make_sock: could not bind to address 192.168.2.1:80
4/8/12 4:59:06.208 PM org.macports.apache2: no listening sockets available, shutting down
4/8/12 4:59:06.208 PM org.macports.apache2: Unable to open logs
I tried adjusting permissions on all the log folders (despite the fact that logs were being written just fine when I manually started apache2) and that didn't help.
Even though the Apache Documentation for Listen clearly states
Multiple Listen directives may be used to specify a number of addresses and ports to listen to. The server will respond to requests from any of the listed addresses and ports.
I decided to try switching back to just using
Listen 80
And after doing so apache2 is starting on boot with no errors or warnings.
If you're using Subversion with Apache, you may find that Apache is not starting because the mod_dav_svn.so file has moved to /opt/local/libexec. You'll need to adjust your Apache startup files to account for the new location of this file.
In newer versions of MacPorts you can run sudo port load apache2 to instruct MacPorts to take care of the launchctl setup and automatically start the process. To stop the process run port unload.
After loading check /opt/local/apache2/logs/error_log for errors, including configuration issues.
In addition to my previous answer I have also found that sometimes Apache fails to start because something else with the system is not yet ready.
On one OS X Server machine I also use the DNS to create a “internal only” DNS name for the machine and that name is used in my Apache configuration. Sometimes when Apache tries to start the DNS server is not yet ready and Apache fails to load because the hostname isn’t valid.
I have also seen this on other non-Server systems without local DNS as well where something else required by Apache must not be ready yet.
One thing that has worked is to edit the apache2.wrapper located at /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper that MacPorts’ daemondo uses to start up Apache.
Edit the Start() function to add a sleep command to wait a bit before launching Apache.
Original (Lines 14-17 on my machine)
Start()
{
[ -x /opt/local/apache2/bin/apachectl ] && /opt/local/apache2/bin/apachectl start > /dev/null
}
With wait time added
Start()
{
[ -x /opt/local/apache2/bin/apachectl ] && sleep 10 && /opt/local/apache2/bin/apachectl start > /dev/null
}