Split brain with keepalived - keepalived

I have keepalived attempting to hold a vip in master/slave configuration on 2 servers.
Master config:
vrrp_script chk_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
}
vrrp_instance NAME {
interface ens224
state MASTER
virtual_router_id 70
priority 104
virtual_ipaddress {
SOME_IP
}
track_script {
chk_haproxy
}
}
Slave config:
vrrp_script chk_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
}
vrrp_instance scc_elastic {
interface ens224
state BACKUP
virtual_router_id 70
priority 103
virtual_ipaddress {
SOME_IP
}
track_script {
chk_haproxy
}
}
But this is causing split brain.. i.e. each one of the servers are showing the ip show up under
"ip addr show" How can I prevent this from happening? I am not sure how to diagnose this. I dont have much network expertise.
I looked at the ip tables:
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
No rules present.

Ensure you have the proper kernel configuration:
sudo sysctl -p
Result:
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.vs.conntrack = 1
You need to ensure vrrp traffic is getting to the slave from master e.g.
sudo tcpdump -i ens224 host master_ip_address && proto vrrp
Result:
14:45:25.207338 IP master_ip_address > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype simple, intvl 10s, length
Ensure iptables allows VRRP and multicast except if you are using unicast:
sudo iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT
sudo iptables -I INPUT -p vrrp -j ACCEPT

Related

nMap scan ssl cipher list fail if argument -sV added

To All,
I am writing a service running HTTPS protocol that accept secure connection using Openssl.
After that, I tested SSL connection using nmap with the following command:
nmap --script ssl-enum-ciphers -p 443 192.168.2.1
Nmap scan report for 192.168.2.1
Host is up (0.0029s latency).
PORT STATE SERVICE
443/tcp open https
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256k1) - A
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256k1) - A
| compressors:
| NULL
| cipher preference: client
|_ least strength: A
However, if the the argument -sV is added, then it displays following
nmap --script ssl-enum-ciphers -sV -p 443 192.168.2.1
Starting Nmap 7.01 ( https://nmap.org ) at 2021-05-25 09:15 CST
Nmap scan report for 192.168.2.1
Host is up (0.0030s latency).
PORT STATE SERVICE VERSION
443/tcp open ssl/https?
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 12.79 seconds
the -sV is used to probe service/version info, I am wondering is it because I am using ECHDE only?
Anyway, here's how I setup my SSL connection (Remove error checking for easy reading).
SSL_library_init();
SSL_load_error_strings();
CTX = SSL_CTX_new(TLSv1_2_server_method());
SL_CTX_set_cipher_list(ctx, "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384");
SSL_CTX_ctrl((CTX),SSL_CTRL_SET_ECDH_AUTO,1,NULL);
SSL_CTX_use_certificate_file(CTX, pem, SSL_FILETYPE_PEM);
SSL_CTX_use_PrivateKey_file(CTX, pem, SSL_FILETYPE_PEM);
SSL_CTX_use_certificate_chain_file(CTX, chain);
I am suspecting the ciphers ECDHE, because if I use Cipher list "AES128-SHA256:AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384", everything seems to work fine.
Any help is appreciated, thanks.

Unable to fetch port count on nagios

I have formed a command for fetching established port connection using nagios check_by_ssh module.
I am able to get the output when I run the command, however after placing the command in the commands.cfg file I am seeing "check_by_ssh: skip-stderr argument must be an integer " in the GUI. Any suggestion on this would be of great help.
Command:
/usr/local/nagios/libexec/check_by_ssh -l fuseadmin -H <hostname> -C "netstat -punta | grep -i ESTABLISHED | wc -l | awk '{if (\$0>2500) {print \"CRITICAL: Established Socket Count: \"\$0} else {print \"OK: Established Socket Count: \"\$0}}'" -i ~/.ssh/id_dsa -E
OK: Established Socket Count: 67
Commands.cfg:
define command {
command_name netstat_cnt_estanblished_gt_2500_fuse01
command_line /usr/local/nagios/libexec/check_by_ssh -l fuseadmin -H a0110pcsgesb01 -C "netstat -punta | grep -i ESTABLISHED | wc -l 2>&1 | awk '{if (\$0>2500) {print \"CRITICAL: Established Socket Count: \"\$0} else {print \"OK: Established Socket Count: \"\$0}}'" -i ~/.ssh/id_dsa -E
}
Service Definition
#netstat_cnt_estanblished_gt_2500_csg2.0
define service{
use generic-service ; Name of service template to use
host_name <hostname>
service_description Netstat Established Count
event_handler send-service-trap-fms
event_handler_enabled 1
check_command netstat_cnt_estanblished_gt_2500_fuse01
max_check_attempts 1
notifications_enabled 1 ; Service notifications are enabled
check_period 24x7 ; The service can be checked at any time of the day
max_check_attempts 3 ; Re-check the service up to 3 times in order to determine its final (hard) state
check_interval 2 ; Check the service every 10 minutes under normal conditions
retry_interval 2 ; Re-check the service every two minutes until a hard state can be determined
contact_groups fuse_users ; Notifications get sent out to everyone in the 'admins' group
notification_options w,u,c,r ; Send notifications about warning, unknown, critical, and recovery events
notification_interval 30 ; Re-notify about service problems every hour
notification_period 24x7
}
**I have changed the actual hostname to due to compliance
here it says:
check_by_ssh: print command output in verbose mode
right now it is not possible to print the command output of ssh. check_by_ssh
only prints the command itself. This patchs adds printing the output too. This
makes it possible to use ssh with verbose logging which helps debuging any
connection, key or other ssh problems.
Note: you must use -E,--skip-stderr=<high number>, otherwise check_by_ssh would
always exit with unknown state.
Example:
./check_by_ssh -H localhost -o LogLevel=DEBUG3 -C "sleep 1" -E 999 -v
Meaning: you should just have to specify a number after "-E", like -E 999, in your definition (like the example in above code-block says)
... even though, it's confusing (maybe a bug?), because the command help of check_by_ssh says:
-E, --skip-stderr[=n]
Ignore all or (if specified) first n lines on STDERR [optional]

conntrack delete does not stop runnig copy of big file

I have a router with nat port forwarding configured. I launched a http copy of big file via the nat. The http server is hosted on the LAN PC which contains the big file to download. I launched the file download from WAN PC.
I disabled the nat rule when file copy is running. the copy of file keep remaining. I want to stop the copy of file when I disable the nat forward rule with conntrack-tool.
my conntrack list contains the following conntrack session
# conntrack -L | grep "33.13"
tcp 6 431988 ESTABLISHED src=192.168.33.13 dst=192.168.33.215 sport=52722 dport=80 src=192.168.3.17 dst=192.168.33.13 sport=80 dport=52722 [ASSURED] use=1
I tried to remove it with the following command:
# conntrack -D --orig-src 192.168.33.13
tcp 6 431982 ESTABLISHED src=192.168.33.13 dst=192.168.33.215 sport=52722 dport=80 src=192.168.3.17 dst=192.168.33.13 sport=80 dport=52722 [ASSURED] use=1
conntrack v1.4.3 (conntrack-tools): 1 flow entries have been deleted.
the conntrack session is removed I can see in the following command. But another conntrack session was created with src ip address is the lan address of the removed conntrack
# conntrack -L | grep "33.13"
tcp 6 431993 ESTABLISHED src=192.168.3.17 dst=192.168.33.13 sport=80 dport=52722 src=192.168.33.13 dst=192.168.33.215 sport=52722 dport=80 [ASSURED] use=1
conntrack v1.4.3 (conntrack-tools): 57 flow entries have been shown.
I tried to remove the new conntrack but it keep remaining
# conntrack -D --orig-src 192.168.3.17
# conntrack -L | grep "33.13"
conntrack v1.4.3 (conntrack-tools): 11 flow entries have been shown.
tcp 6 431981 ESTABLISHED src=192.168.3.17 dst=192.168.33.13 sport=80 dport=52722 src=192.168.33.13 dst=192.168.33.215 sport=52722 dport=80 [ASSURED] use=1
What I m missing?
first, if "conntrack -D" command succeed, you can see below Messsage.
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been deleted.
So we guess that track deleltion working was failed.
Why do not conntrack delete track?
Perhaps you are referencing a session you want to delete from a specific skb or track.
if you want to get detail infomation, you try to follow "ctnetlink_del_conntrack " call stack funcion in linux kernel.

Find original owning process of a Linux socket

In Linux and other UNIX-like operating systems, it is possible for two (or more) processes to share an Internet socket. Assuming there is no parent-child relationship between the processes, is there any way to tell what process originally created a socket?
Clarification: I need to determine this from "outside" the processes using the /proc filesystem or similar. I can't modify the code of the processes. I can already tell what processes are sharing sockets by reading /proc/<pid>/fd, but that doesn't tell me what process originally created them.
You can use netstat for this. You should look in the columns 'Local Address' and 'PID/Program name'.
xxx#xxx:~$ netstat -tulpen
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 127.0.0.1:4005 0.0.0.0:* LISTEN 1000 68449 7559/sbcl
tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN 0 3938 -
tcp6 0 0 :::6000 :::* LISTEN 0 3937 -
udp 0 0 0.0.0.0:68 0.0.0.0:* 0 4528 -
doesn't 'lsof -Ua' help?
You can likely find the shared sockets by parsing /proc/net/tcp (and similar "files" for other protocols). There's some docs on /proc/net/tcp here.
You would need to find the socket (perhaps by its IP addresses/port numbers ?) and parse out the inode number. Once you have the inode, you can search through all of /proc/*/fd/* , calling stat for every link and inspect the st_ino member of struct stat until you find a match.
The inode number should match between the 2 processes, so when you've gone through all /proc/*/fd/* you should have found them both.
If what you do know is the process id and socket fd of the first, you might not need to go through /proc/net/tcp, all you need to do is stat the /proc/<pid>/fd/<fd> and search the rest of /proc/*/fd/* for a matching inode. You'd need /proc/net/tcp if you want to fetch the ip addresses/port number though - which you can find if you know the inode number
For purposes creating a test case, consider a situation where multiple ssh-agent processes are running and have open sockets. I.e. A user runs ssh-agent multiple times and loses the socket/PID information given when the agent started:
$ find /tmp -path "*ssh*agent*" 2>/dev/null
/tmp/ssh-0XemJ4YlRtVI/agent.14405
/tmp/ssh-W1Tl4i8HiftZ/agent.21283
/tmp/ssh-w4fyViMab8wr/agent.10966
Later, the user wants to programmatically determine the PID owner of a particular ssh-agent socket (i.e. /tmp/ssh-W1Tl4i8HiftZ/agent.21283):
$ stat /tmp/ssh-W1Tl4i8HiftZ/agent.21283
File: '/tmp/ssh-W1Tl4i8HiftZ/agent.21283'
Size: 0 Blocks: 0 IO Block: 4096 socket
Device: 805h/2053d Inode: 113 Links: 1
Access: (0600/srw-------) Uid: ( 4000/ myname) Gid: ( 4500/ mygrp)
Access: 2018-03-07 21:23:08.373138728 -0600
Modify: 2018-03-07 20:49:43.638291884 -0600
Change: 2018-03-07 20:49:43.638291884 -0600
Birth: -
In this case, because ssh-agent named its socket nicely as a human onlooker can guess that the socket belongs to PID 21284, because the socket name contains a numeric component that is one-off from a PID identified with ps:
$ ps -ef | grep ssh-agent
myname 10967 1 0 16:54 ? 00:00:00 ssh-agent
myname 14406 1 0 20:35 ? 00:00:00 ssh-agent
myname 21284 1 0 20:49 ? 00:00:00 ssh-agent
It seems highly unwise to make any assumption that the PIDs will be so reliable as to always only be off by one, but also, one might suppose that not all socket creators will name the sockets so nicely.
#Cypher's answer points to a straightforward solution to the problem of identifying the PID of the socket owner, but is incomplete as lsof actually can only identify this PID with elevated permissions. Without elevated permissions, no results are forthcoming:
$ lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283
$
With elevated permissions, however, the PID is identified:
$ sudo lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ssh-agent 21284 myname 3u unix 0xffff971aba04cc00 0t0 1785049 /tmp/ssh-W1Tl4i8HiftZ/agent.21283 type=STREAM
In this case, the owner of the PID (myname) and socket was the one doing the query, so it seemed elevated permissions should not be needed. Furthermore, the task performing the query was not supposed to be able to elevate permissions, so I looked for another answer.
This led me to #whoplisp's answer proposing netstat -tulpen as a solution to the OP's problem. While it may have been effective for the OP, the command line is too restrictive to serve as a general purpose command and was completely ineffective in this case (even with elevated permissions).
$ sudo netstat -tulpen | grep -E -- '(agent.21283|ssh-agent)'
$
netstat, however, can come close if a different command-line is used:
$ netstat -ap | grep -E -- '(agent.21283)'
(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)
unix 2 [ ACC ] STREAM LISTENING 1785049 - /tmp/ssh-W1Tl4i8HiftZ/agent.21283
Sadly, here too, the PID is elusive without elevated permissions:
$ sudo netstat -ap | grep -E -- '(agent.21283|ssh-agent)'
unix 2 [ ACC ] STREAM LISTENING 1765316 10967/ssh-agent /tmp/ssh-w4fyViMab8wr/agent.10966
unix 2 [ ACC ] STREAM LISTENING 1777450 14406/ssh-agent /tmp/ssh-0XemJ4YlRtVI/agent.14405
unix 2 [ ACC ] STREAM LISTENING 1785049 21284/ssh-agent /tmp/ssh-W1Tl4i8HiftZ/agent.21283
Of the two solutions, however, lsof clearly wins at the races:
$ time sudo netstat -ap | grep -E -- '(agent.21283|ssh-agent)' >/dev/null
real 0m5.159s
user 0m0.010s
sys 0m0.019s
$ time sudo lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283 >/dev/null
real 0m0.120s
user 0m0.038s
sys 0m0.066s
Yet another tool exists according to the netstat man page:
$ man netstat | grep -iC1 replace
NOTES
This program is mostly obsolete. Replacement for netstat is ss. Replacement for netstat -r is ip route. Replacement for netstat -i
is ip -s link. Replacement for netstat -g is ip maddr.
Sadly, ss also requires elevated permissions to identify the PID, but, it beats both netstat and lsof execution times:
$ time sudo ss -ap | grep -E "(agent.21283|ssh-agent)"
u_str LISTEN 0 128 /tmp/ssh-w4fyViMab8wr/agent.10966 1765316 * 0 users:(("ssh-agent",pid=10967,fd=3))
u_str LISTEN 0 128 /tmp/ssh-0XemJ4YlRtVI/agent.14405 1777450 * 0 users:(("ssh-agent",pid=14406,fd=3))
u_str LISTEN 0 128 /tmp/ssh-W1Tl4i8HiftZ/agent.21283 1785049 * 0 users:(("ssh-agent",pid=21284,fd=3))
real 0m0.043s
user 0m0.018s
sys 0m0.021s
In conclusion, it might seem that for some PID identification, it appears that elevated permissions are required.
Note: Not all operating systems require elevated permissions. For example, SCO Openserver 5.0.7's lsof seemed to work just fine without elevating permissions.
Caveat: This answer may fail with respect to the OP's qualification for finding "the original creator" of the socket. In the example used, no doubt PID 21283 was the originator of the socket's creation as this PID is identified in the socket name. Neither lsof nor netstat identified PID 21283 as the original creator, though clearly PID 21284 is the current maintainer.

Linux API to determine sockets owned by a process

Is there a Linux library that will enable me to tell what IP sockets are owned by what processes? I guess I'm looking for the programmatic equivalent of lsof -i. Ultimately, I want to correlate packets seen through libpcap to processes.
UPDATE: A couple of people have suggested using /proc/<pid>/net/tcp and udp, but on my system, the same data is shown for every process, so it doesn't help.
I think you first have to look through the open fds in /proc/*/fd, e.g.
4 -> socket:[11147]
and then look for the referenced sockets (by the inode) in /proc/net/tcp (or /proc/net/udp), e.g.
12: B382595D:8B40 D5C43B45:0050 01 00000000:00000000 00:00000000 00000000 1000 0 11065 1 ffff88008bd35480 69 4 12 4 -1
To determine sockets owned by a process you can just use netstat. Here's an example w/output (shortened) of netstat with options that will do what you want.
$ sudo netstat -apeen
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 127.0.0.1:8118 0.0.0.0:* LISTEN 138 744850 13248/privoxy
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 117 9612 2019/postgres
udp 0 0 127.0.0.1:51960 127.0.0.1:51960 ESTABLISHED 117 7957 2019/postgres
udp 0 0 0.0.0.0:68 0.0.0.0:* 0 7740 1989/dhclient
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 7937 2019/postgres /var/run/postgresql/.s.PGSQL.5432
unix 2 [ ACC ] STREAM LISTENING 958058 8080/emacs /tmp/emacs1000/server
unix 2 [ ACC ] STREAM LISTENING 6969 1625/Xorg /tmp/.X11-unix/X0
unix 2 [ ] DGRAM 9325 1989/dhclient
unix 3 [ ] STREAM CONNECTED 7720 1625/Xorg #/tmp/.X11-unix/X0
Make sure you run netstat as root otherwise you'll get this message:
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
An explanation of the -apeen options from the netstat manpage:
-a, --all
Show both listening and non-listening sockets. With the
--interfaces option, show interfaces that are not up
-p, --program
Show the PID and name of the program to which each socket
belongs.
-e, --extend
Display additional information. Use this option twice for
maximum detail.
--numeric , -n
Show numerical addresses instead of trying to determine symbolic host, port or user names.
--numeric-hosts
shows numerical host addresses but does not affect the resolution of port or user names.
--numeric-ports
shows numerical port numbers but does not affect the resolution of host or user names.
--numeric-users
shows numerical user IDs but does not affect the resolution of host or port names.
The /proc filesystem provides details on each process, including networking information. Open socket information is listed in /proc/net/tcp. The IPv6 sockets are listed separately in the tcp6 file. The socket information includes information such as the local and remote ports, and the socket inode number, which can be mapped back to the process by parsing the /proc/{pid}/fd/* information.
If you aren't familiar with the /proc filesystem, it is basically a virtual filesystem that allows the kernel to publish all sorts of useful information to user-space. The files are normally simple structured text files that are easy to parse.
For example, on my Ubuntu system I used netcat for testing, and ran nc -l -p 8321 to listen on port 8321. Looking at the tcp socket information:
$ cat /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:2081 00000000:0000 0A 00000000:00000000 00:00000000 00000000 1000 0 26442 1 de0c8e40 300 0 0 2 -1
1: 0100007F:0277 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7019 1 de0c84c0 300 0 0 2 -1
The first line shows it is listening on all addresses to point 8321 (0x2081). The inode number is 26442, which we can use to look up the matching pid in /proc/{pid}/fd/*, which consists of a bunch of symlinks from the file handle number to the device. So if we look up the pid for netcat, and check its fd mapping:
$ ls -l /proc/7266/fd
total 0
lrwx------ 1 gavinb gavinb 64 2009-12-31 09:10 0 -> /dev/pts/1
lrwx------ 1 gavinb gavinb 64 2009-12-31 09:10 1 -> /dev/pts/1
lrwx------ 1 gavinb gavinb 64 2009-12-31 09:10 2 -> /dev/pts/1
lrwx------ 1 gavinb gavinb 64 2009-12-31 09:10 3 -> socket:[26442]
And there we see that file descriptor 3 in this process is mapped to the socket with inode 26442, just as we expect.
So obviously to build a complete map of sockets, you will need to first enumerate all the /proc/**/fd/* files, look up the socket symlinks, then match the socket inode against the tables from /proc/net/tcp which has the endpoint information.
This is the way the lsof tool works (see lsof/dialects/linux/dsocket.c for the implementation).
Wikipedia on procfs
The Linux /proc filesystem as a Programmer's Tool
/proc/<pid>/net is equivalent to /proc/net for all processes in the same network namespace as you – in other words, it's "global" information.
You can do what lsof and fuser do, which is to iterate through both /proc/<pid>/fd/* and /proc/net/* looking for matching inodes. Quick demonstration:
#!/bin/sh
pgrep "$#" | while read pid; do
for fd in /proc/$pid/fd/*; do
name=$(readlink $fd)
case $name in
socket:\[*\])
ino=${name#*:}
for proto in tcp:10 tcp6:10 udp:10 udp6:10 unix:7; do
[[ ! -e /proc/net/${proto%:*} ]] ||
awk "
\$${proto##*:} == ${ino:1:${#ino}-2} {
print \"${proto%:*}:\", \$0
exit 1
}
" /proc/net/${proto%:*} || break
done
;;
esac
done
done
You can extend this to other protocols (I see ax25, ipx, packet, raw, raw6, udplite, udp6lite in /proc/net/ too) or rewrite in a language of your choosing.
You can read them from proc filesystem. The 'files' you probably want to look at are found in
/proc/<pid>/net (namely tcp, udp, unix)
Here's some examples on using the proc filesystem
You could try running lsof with strace and see just which files in /proc it gets data from.
I'd go to the source:
http://ubuntuforums.org/showthread.php?t=1346778

Resources