Warning: Return code of 127 for check of service was out of bounds. Make sure the plugin you're trying to run actually exists - nagios

I'm trying to use the phanton-js plugin for Nagios:
https://github.com/hggh/phantomjs-nagios
but despite me double checking the set-up, I'm getting
"Return code 127 is out of bounds : Plugin may be missing"
The plugin file exists in the plugins folder:
# stat /usr/lib64/nagios/plugins/check_http_load_time.rb
File: `/usr/lib64/nagios/plugins/check_http_load_time.rb'
Size: 9108 Blocks: 24 IO Block: 4096 regular file
Device: fc01h/64513d Inode: 275201 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
I can execute it when su'd to the nagios user locally:
-bash-4.1$ whoami
nagios
-bash-4.1$ /usr/lib64/nagios/plugins/phantomjs-nagios/check_http_load_time.rb -u https://google.com -w 2 -c 3
OK: https://google.com load time: 0.43
This is how it is defined in the checkcommands.cfg file:
define command {
command_name check_web_page_load_time
command_line $USER1$/check_http_load_time.rb -u $ARG1$ -w $ARG2$ -c $ARG3$
}
and this is the service definition
define service {
use generic-service
host_name test_host
service_description https://google.com web load time
check_command check_web_page_load_time!https://google.com!2!3
contact_groups support-emails
}
I've restarted nagios and confirmed that the config check passes.
Am I missing something obvious?
edit:
plugin file is located in 2 folders:
/usr/lib64/nagios/plugins/check_http_load_time.rb
and
/usr/lib64/nagios/plugins/phantomjs-nagios/check_http_load_time.rb
both files are identical, both can be executed by the nagios user successfully and neither location works with command definition. I tried calling it with full path and with the $USR1$ variable, which points to the standard location where all the other plugins are located
/usr/lib64/nagios/plugins/

Manged to eventually solve it. The error message was not very helpful, as problem was with the environment variables of the nagios user.
The original script had the following shebang:
#!/usr/bin/env ruby
After changing it to the actual location of ruby binary the script works:
#!/usr/local/rvm/rubies/ruby-2.5.1/bin/ruby
Still surprised that it worked when su'd to the Nagios user.

Related

nagios check_http HTTP CRITICAL - Unable to open TCP socket, works fine from command line

I have the following service definition:
define service{
use my-service ; Name of service template to use
host_name dra
service_description https://www.example.com
check_command check_http!-I my.ip.address --ssl -H www.example.com
notifications_enabled 1
retry_check_interval 2
normal_check_interval 5
contact_groups myadmins
}
The service check keeps failing with
Name or service not known
HTTP CRITICAL - Unable to open TCP socket
However, if I run http_check from the command line, I get a 200 OK result:
/usr/lib/nagios/plugins/check_http -I my.ip.address --ssl -H www.example.com -v
.....
HTTP OK: HTTP/1.1 200 OK - 9176 bytes in 0.074 second response time |time=0.073543s;;;0.000000 size=9176B;;;0
Note also that the URL in question works just fine from a browser, the certificate is valid, etc. I also use the exact same service definition for a bunch of other sites, and they all work fine. The only thing I can think of is that this remote host is running on DigitalOcean and has a "Floating IP" assigned to it. I tried replacing my.ip.address above (and also in the host definition of the nagios config file) with either the Floating IP or the "standard" IP assigned to the host, and it makes no difference.
How is it possible that the same command would fail when run by nagios, but succeed when run manually?
The answer to my question is: don't use check_http, use
use check_https_hostname, and
make sure that the host_name stanza is the actual hostname
which requires matching the host_name stanzas in all the service and host definitions in the same cfg file.
So:
define service{
use my-service ; Name of service template to use
host_name www.example.com
service_description https://www.example.com
check_command check_https_hostname
notifications_enabled 1
retry_check_interval 2
normal_check_interval 5
contact_groups myadmins
}
Here is why: it becomes clear by looking at the definitions of check_http and check_https_hostname which are in the /etc/nagios-plugins/config/http.cfg file in my installation.
# 'check_http' command definition
define command{
command_name check_http
command_line /usr/lib/nagios/plugins/check_http -H '$HOSTADDRESS$' -I '$HOSTADDRESS$' '$ARG1$'
}
# 'check_https_hostname' command definition
define command{
command_name check_https_hostname
command_line /usr/lib/nagios/plugins/check_http --ssl -H '$HOSTNAME$' -I '$HOSTADDRESS$' '$ARG1$'
}
You will notice that the -H and -I arguments in check_http get the same value $HOSTADDRESS$, while in check_https_hostname they get $HOSTNAME$ and $HOSTADDRESS$, respectively.
The fact that I built my original command as check_http!-I my.ip.address --ssl -H www.example.com did not really matter. In the end, the /usr/lib/nagios/plugins/check_http command got two values for -I and two for -H, and the second pair was being ignored.
This did break "thanks" to Cloudflare, because the IP address dynamically assigned by Cloudflare to my www.example.com was not the same as the actual host IP address that I had specified in my host definition.
Finally, I wanted to mention that what helped me figure this out was setting
debug_level=-1
debug_verbosity=1
in my /etc/nagios3/nagios.cfg file and then looking through /var/log/nagios3/nagios.debug.
Also, check out all the different variants of the check_http commands in /etc/nagios-plugins/config/http.cfg. There are some very useful ones.

How to output hostname in "service Description" for nagios core?

I have currently the following two service defined as below:
define service {
use my-webapp-service
hostgroup_name all
service_description System check - PING
check_command check_ping!100.0,20%!500.0,60%
}
define service {
use my-webapp-service
hostgroup_name all
service_description System check - Swap Usage
check_command check_nrpe!check_swap
check_interval 1
}
What I want is output string to be:
System check - PING - "Actual hostname where this alarm got fired off"
System check - Swap Usage - "Actual hostname where this alarm got fired off"
I think this could be possible but I just don't know how to make it possible.
Would sincerely appreciate your guidance on that.
Many Thanks
Output are handled by scripts. Default behavior is that script donĀ“t return hostname, because it is not necessary.
If you wanna add hostname in output, you must edit already exist scripts or create new one.
Here is basic info how create script for Nagios - http://www.yourownlinux.com/2014/06/how-to-create-nagios-plugin-using-bash-script.html
For your needs you must add $HOSTNAME to echo. For instance:
echo "$HOSTNAME - WARNING- $output"
If you want the script that is executing to be aware of the hostname, you'll need to pass the hostname as an argument to the Nagios command. That also means that the script will need to accept the hostname as an argument. Take for example:
define service {
use my-webapp-service
hostgroup_name all
service_description System check - PING
check_command check_ping!100.0,20%!500.0,60%
}
check_ping probably looks something like:
define command {
command_name check_ping
command_line $USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5
}
The problem here is that the executable at $USER1$/check_ping doesn't know that you want to pass the host's name as an argument. So you'll need to make a wrapper script. I'm not going to write the script for you, but to give you a hint, the command definition would look something like:
define command {
command_name check_ping_print_hostname
command_line $USER1$/my_check_ping_wrapper.sh --hostname $HOSTNAME$ -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5
}
And then the script at $USER1$/my_check_ping_wrapper.sh is obviously going to need grab that --hostname argument, and then pass the other arguments directly to check_ping, wait for the output, and then amend the output with the information given in the --hostname arg.
Hope this helps!

Nagios Monitoring Hosts with check_ping

I've deployed a new instance of Nagios on a fresh install of CentOS 7 via the EPEL repository. So the Nagios Core version is 3.5.1.
After installing nagios and nagios-plugins-all (via yum), I've created a number of hosts and service definitions, have tested my configuration with nagios -v /etc/nagios/nagios.cfg, and have Nagios up and running!
Unfortunately, my host checks are failing (although my service checks are working perfectly fine).
Within the Nagios Web GUI / Dashboard, if I drill down into a Host page with the "Host State Information", I see this being reported for "Status Information" (IP address removed):
Status Information: /usr/bin/ping -n -U -w 30 -c 5 {my-host-ip-address}
CRITICAL - Could not interpret output from ping command
So in my troubleshooting, I drilled down into the Nagios Plugins directory (/usr/lib64/nagios/plugins), and ran a test with the check_ping plugin consistent with the way check-host-alive runs the command (see below for my check-host-alive command definition):
./check_ping -H {my-ip-address} -w 3000.0,80% -c 5000.0,100% -p 5
This check_ping command returns the following output:
PING OK - Packet loss = 0%, RTA = 0.63
ms|rta=0.627000ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0
I haven't changed the definition of how check_ping works, and can confirm that I'm getting a "PING OK" whenever the command is run the same way that check-host-alive runs the command, so I cannot figure out what's going on!
Below are the command definitions for check-host-alive as well as check_ping.
# 'check-host-alive' command definition
define command{
command_name check-host-alive
command_line $USER1$/check_ping -H $HOSTADDRESS$ -w 3000.0,80% -c 5000.0,100% -p 5
}
{snip}
# 'check_ping' command definition
define command{
command_name check_ping
command_line $USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5
}
Any suggestions on how I can fix my check-host-alive command definition to work properly and evaluate the output of check_ping properly?
Edit
Below is the full define host {} template I'm using:
define host {
host_name myers ; The name of this host template
alias Myers
address [redacted]
check_command check-host-alive
contact_groups admins
notifications_enabled 0 ; Host notifications are enabled
event_handler_enabled 1 ; Host event handler is enabled
flap_detection_enabled 1 ; Flap detection is enabled
failure_prediction_enabled 1 ; Failure prediction is enabled
process_perf_data 1 ; Process performance data
retain_status_information 1 ; Retain status information across program restarts
retain_nonstatus_information 1 ; Retain non-status information across program restarts
notification_period 24x7 ; Send host notifications at any time
register 1
max_check_attempts 2
}
For anyone else who runs into this issue, there's another option than changing permissions on ping. Simply change the host check command to use check_host rather than check_ping. While there are certainly some differences in the functionality, the overall end result is the same.
There are those who will say this isn't a good option because of the ability to range the check_ping command, but it should be remembered that host checks aren't even executed until all service checks for a given host have failed. Anyway, if you're interested in testing throughput, there are MUCH better ways of going about it than relying on ICMP, which is the lowest priority traffic type on a network.
I'm sure the OP is well on to other things by now, but hopefully someone else who has this issue will benefit.
I could not found the ping on /usr/bin/ping
# chmod u+s /bin/ping
# ls -al /bin/ping
-rwsr-xr-x 1 root root 40760 Sep 26 2013 /bin/ping*
Finally run the below command,
/usr/local/nagios/libexec/check_ping -H 127.0.0.1 -w 100.0,20% -c 500.0,60% -p 5
I was fairly certain that running chmod U+s /usr/bin/ping would solve the issue, but I was (and still am) wary about chmod'ing system files. It seems to me that there has to be a safer way to do it.
However, in the end, that's what I did - and it works. I don't like it, from a security standpoint.
I also had same problem and the above answers did not work for me. After some checking the issue further noticed that the reason is IP protocol. once I passed the correct IP protocol , It worked fine.
/usr/local/nagios/libexec/check_ping -H localhost -w 3000.0,80% -c 5000.0,100% -4
output
PING OK - Packet loss = 0%, RTA = 0.05 ms|rta=0.051000ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0
By default It's getting IPv6.
/usr/local/nagios/libexec/check_ping -H localhost -w 3000.0,80% -c 5000.0,100% -6
output
/sbin/ping6 -n -U -W 30 -c 5 localhost
CRITICAL - Could not interpret output from ping command
But when integrating with Nagios server, I could not able to pass this value as an argument. Therefore I have done below workaround in client side nrpe.cfg file
command[check_ping_args]=/usr/local/nagios/libexec/check_ping -H $ARG1$ -w $ARG2$ -c $ARG3$ -4
Here Host, warning and critical thresholds were passing by Nagios host as below,
define service{
use generic-service
hostgroup_name all-servers
service_description Host Ping Status
check_command check_nrpe_args!check_ping_args!localhost!3000.0,80%!5000.0,100%
}

nagios Check result path is not a valid directory

i'm getting error when i run below command
nagios3 -v /etc/nagios3/nagios.cfg
Error in configuration file '/etc/nagios3/nagios.cfg' - Line 469 (Check result path is not a valid directory) Error processing main config file
So i looked ls -l /var/lib/nagios3/
drwxr-x--- 3 nagios nagios 1024 Mar 14 21:13 spool
In this case, why i'm getting error? Probably i think my /var/lib/nagios3/spool/checkresult/check2JcDx5 file contains wrong line. And when i run below command, i get this output.
#cat check2JcDx5
file_time=1363378360
host_name=localhost
service_description=HTTP
check_type=0
check_options=0
scheduled_check=1
reschedule_check=1
latency=0.122000
start_time=1363378360.122234
Disable SELinux:
# getenforce
# setenforce 0
Edit /etc/selinux/config. Set SELINUX=disabled.
You may be able to install the nagios-selinux package to add the policy to run nagios in an selinux environment. Better than disabling your existing security.

why is checking a URL failing when run through icinga?

I created my own command to check a specific URL
define command{
command_name check_url
command_line /usr/lib/nagios/plugins/check_http -f follow -H '$HOSTNAME$' -I '$HOSTADDRESS$' -u '$ARG1$'
}
If I run my command from the command line, it works:
/usr/lib/nagios/plugins/check_http -f follow -H www.example.com -u http://www.example.com/server-status
HTTP OK: HTTP/1.1 200 OK - 4826 bytes in 0.011 second response time |time=0.010625s;;;0.000000 size=4826B;;;0
But when run through Icinga, I'm getting
HTTP WARNING: HTTP/1.1 404 NOT FOUND - 314 bytes in 0.011 second response time
My guess is for check_http plugin for -u option you should provide the url appended after the server name not the whole url.
Ex.
/usr/lib/nagios/plugins/check_http -f follow -H www.example.com -u /server-status
Your manual test is not equivalent to your command definition.
The distinction with -H/-I is subtle, but very important.
When I have problems like this, where Icinga is abstracting exactly how it is executing the command, I find it helpful to find out precicely what Icinga is executing. I would accomplish this as follows:
Move check_http to a temporary location
# mv /usr/lib/nagios/plugins/check_http /usr/lib/nagios/plugins/check_http_actual
Make a bash script that Icinga will call instead of the actual check_http script
# vi /usr/lib/nagios/plugins/check_http
In that file, create this simple bash script, which simply echos the command line arguments it was called with, then exits:
#!/bin/bash
echo $#
Then of course, make that bash script executable:
# chmod +x /usr/lib/nagios/plugins/check_http
Now in Icinga, run the check_http command. At that point, the return status shown in the Icinga web interface will show exactly how Icinga is calling check_http. Seeing the raw command, it should be obvious as to what Icinga is doing wrong. Once you correct Icinga's mistake, you can simply move the original check_http script back into place:
# mv /usr/lib/nagios/plugins/{check_http_actual,check_http}

Resources