I have the next 'zfs pool' in the machine A:
root#machineA:/ # zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
tank 7.44M 28.8G 20K /tank
tank/test 92K 28.8G 19K /tank/test
tank/test#SNAP_2017-June-30_10:00:00 9K - 19K -
tank/test#SNAP_2017-July-01_10:00:00 9K - 19K -
tank/test#SNAP_2017-July-02_10:00:00 9K - 19K -
tank/test#SNAP_2017-July-03_10:00:00 9K - 19K -
tank/test#SNAP_2017-July-04_10:00:00 0 - 19K -
tank/test#BACKUP_from_2017-June-30 0 - 19K -
tank/test/exe 37K 28.8G 19K /tank/test/exe
tank/test/exe#EXE_2017-June-29_13:58:49 9K - 19K -
tank/test/exe#EXE_2017-July-03_10:00:00 9K - 19K -
tank/test/exe#EXE_2017-July-04_10:00:00 0 - 19K -
tank/test/exe#BACKUP_from_2017-June-29
And I want to send a snapshot to the machine B:
root#machineB:/ # zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
tank 6.04M 28.8G 23K /tank
With the netcat I can send the snapshots but the system returns me an error very unusual...
If I do:
B: nc -w 5 -l 7766 | zfs recv tank/test/exe
A: zfs send -R tank/test/exe#EXE_2017-July-04_10:00:00 | nc -w 5 192.168.99.2 7766
All it's ok, but if I do:
B: nc -w 5 -l 7766 | zfs recv tank/test
A: zfs send -R tank/test#SNAP_2017-July-04_10:00:00 | nc -w 5 192.168.99.2 7766
The stream of snapshots is sent but in the source side I can show:
root#machineA:/ # zfs send -R tank/test#SNAP_2017-July-04_10:00:00 | nc -w 5 192.168.99.2 7766
WARNING: could not send tank/test/exe#SNAP_2017-July-04_10:00:00: does not exist
WARNING: could not send tank/test/exe#SNAP_2017-July-04_10:00:00: does not exist
Why ZFS takes the dataset tank/test/exe? Any suggestions?
Actually, the snapshots it's complaining about don't exist on the source system -- tank/test/exe only has #EXE_<date> snapshots, while you're trying to send tank/test/exe#SNAP_<date>. This warning is appearing because you're sending with -R (recursive) from the top-level tank/test filesystem, which sends the specified snapshot on the parent filesystem first, and then searches the children for the same snapshot name to try to send those as well. Usually, this only does what you are expecting when you've taken a snapshot on the parent filesystem with -R -- on your system, you only snapshotted the parent without snapshotting the child at the same time.
On the sending system, you probably want to change your command to be:
zfs send -R tank/test/snap#SNAP_2017-July-04_10:00:00 | nc -w 5 192.168.99.2 7766
Related
I want to save 1 line from the output of top into a Bash array to later access its components:
$ timeout 1 top -d 2 | awk 'NR==8'
2436 USER 20 0 1040580 155268 91100 S 6.2 1.0 56:38.94 Xorg
Terminated
I tried:
$ gpu=($(timeout 1s top -d 2 | awk 'NR==8'))
$ mapfile -t gpu < <($(timeout 1s top -d 2 | awk 'NR==8'))
and, departing from the array requisite, even:
$ read -r gpu < <(timeout 1s top -d 2 | awk 'NR==8')
all returned a blank for either ${gpu[#]} (first two) or $gpu (last).
Edit:
As pointed out by #Cyrus and others gpu=($(top -n 1 -d 2 | awk 'NR==8')) is the obvious solution. However I want to build the cmd dynamically so top -d 2 may be replaced by other cmds such as htop -d 20 or intel_gpu_top -s 1. Only top can limit its maximum number of iterations, so that is not an option in general, and for that reason I resort to timeout 1s to kill the process in all shown attempts...
End edit
Using a shell other than Bash is not an option. Why did the above attempts fail and how can I achieve that ?
Why did the above attempts fail
Because redirection to pipe does not have terminal capabilities, top process receives SIGTTOU signal when it tries to write the terminal and take the terminal "back" from the shell. The signal causes top to terminate.
how can I achieve that ?
Use top -n 1. Generally, use the tool specific options to disable using terminal utilities by that tool.
However I want to build the cmd dynamically so top -d 2 may be replaced by other cmds such as htop -d 20 or intel_gpu_top -s 1
Write your own terminal emulation and extract the first line from the buffer of the first stuff the command displays. See GNU screen and tmux source code for inspiration.
I dont think you need the timeout there if its intended to quit top. You can instead use the -n and -b flags but feel free to add it if you need it
#!/bin/bash
arr=()
arr[0]=$(top -n 1 -b -d 2 | awk 'NR==8')
arr[1]=random-value
arr[2]=$(top -n 1 -b -d 2 |awk 'NR==8')
echo ${arr[0]}
echo ${arr[1]}
echo ${arr[2]}
output:
1 root 20 0 99868 10412 7980 S 0.0 0.5 0:00.99 systemd
random-value
1 root 20 0 99868 10412 7980 S 0.0 0.5 0:00.99 systemd
from top man page:
-b :Batch-mode operation
Starts top in Batch mode, which could be useful for sending output from top to other programs or to a
file. In this mode, top will not accept input and runs until the iterations limit you've set with the
`-n' command-line option or until killed.
-n :Number-of-iterations limit as: -n number
Specifies the maximum number of iterations, or frames, top should produce before ending.
-d :Delay-time interval as: -d ss.t (secs.tenths)
Specifies the delay between screen updates, and overrides the corresponding value in one's personal
configuration file or the startup default. Later this can be changed with the `d' or `s' interactive
commands.
I have formed a command for fetching established port connection using nagios check_by_ssh module.
I am able to get the output when I run the command, however after placing the command in the commands.cfg file I am seeing "check_by_ssh: skip-stderr argument must be an integer " in the GUI. Any suggestion on this would be of great help.
Command:
/usr/local/nagios/libexec/check_by_ssh -l fuseadmin -H <hostname> -C "netstat -punta | grep -i ESTABLISHED | wc -l | awk '{if (\$0>2500) {print \"CRITICAL: Established Socket Count: \"\$0} else {print \"OK: Established Socket Count: \"\$0}}'" -i ~/.ssh/id_dsa -E
OK: Established Socket Count: 67
Commands.cfg:
define command {
command_name netstat_cnt_estanblished_gt_2500_fuse01
command_line /usr/local/nagios/libexec/check_by_ssh -l fuseadmin -H a0110pcsgesb01 -C "netstat -punta | grep -i ESTABLISHED | wc -l 2>&1 | awk '{if (\$0>2500) {print \"CRITICAL: Established Socket Count: \"\$0} else {print \"OK: Established Socket Count: \"\$0}}'" -i ~/.ssh/id_dsa -E
}
Service Definition
#netstat_cnt_estanblished_gt_2500_csg2.0
define service{
use generic-service ; Name of service template to use
host_name <hostname>
service_description Netstat Established Count
event_handler send-service-trap-fms
event_handler_enabled 1
check_command netstat_cnt_estanblished_gt_2500_fuse01
max_check_attempts 1
notifications_enabled 1 ; Service notifications are enabled
check_period 24x7 ; The service can be checked at any time of the day
max_check_attempts 3 ; Re-check the service up to 3 times in order to determine its final (hard) state
check_interval 2 ; Check the service every 10 minutes under normal conditions
retry_interval 2 ; Re-check the service every two minutes until a hard state can be determined
contact_groups fuse_users ; Notifications get sent out to everyone in the 'admins' group
notification_options w,u,c,r ; Send notifications about warning, unknown, critical, and recovery events
notification_interval 30 ; Re-notify about service problems every hour
notification_period 24x7
}
**I have changed the actual hostname to due to compliance
here it says:
check_by_ssh: print command output in verbose mode
right now it is not possible to print the command output of ssh. check_by_ssh
only prints the command itself. This patchs adds printing the output too. This
makes it possible to use ssh with verbose logging which helps debuging any
connection, key or other ssh problems.
Note: you must use -E,--skip-stderr=<high number>, otherwise check_by_ssh would
always exit with unknown state.
Example:
./check_by_ssh -H localhost -o LogLevel=DEBUG3 -C "sleep 1" -E 999 -v
Meaning: you should just have to specify a number after "-E", like -E 999, in your definition (like the example in above code-block says)
... even though, it's confusing (maybe a bug?), because the command help of check_by_ssh says:
-E, --skip-stderr[=n]
Ignore all or (if specified) first n lines on STDERR [optional]
I'm reading a stream with curl and grep some highlights.
curl url | grep desired_key_word
I've noticed that curl is providing me some nice download statistics such as:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10.9M 0 10.9M 0 0 1008k 0 --:--:-- 0:00:11 --:--:-- 1092k
How can I save those statistics e.g. every second in a file?
I found this: http://curl.haxx.se/mail/archive-2002-11/0115.html however it was not able to abstract it to my problem.
curl -n agent.mtconnect.org/sample\?interval=0 -o xml_stream.log 2>> dl.log
The dl.log should have the statistics included, however is does not work.
Here is the unabstracted version.
curl -s -S -n http://speedtest.fremont.linode.com/100MB-fremont.bin -o /dev/null -w "%{time_total},%{size_download},%{speed_download}\n" >> stats.log
Only the stdout get redirected by the -o flag.
For the -o flag the man page states:
-o/--output <file>
Write output to <file> instead of stdout...
If you want stderr, you need something like this:
curl -n agent.mtconnect.org/sample\?interval=0 >> xml_stream.log 2>> dl.log
In Linux and other UNIX-like operating systems, it is possible for two (or more) processes to share an Internet socket. Assuming there is no parent-child relationship between the processes, is there any way to tell what process originally created a socket?
Clarification: I need to determine this from "outside" the processes using the /proc filesystem or similar. I can't modify the code of the processes. I can already tell what processes are sharing sockets by reading /proc/<pid>/fd, but that doesn't tell me what process originally created them.
You can use netstat for this. You should look in the columns 'Local Address' and 'PID/Program name'.
xxx#xxx:~$ netstat -tulpen
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 127.0.0.1:4005 0.0.0.0:* LISTEN 1000 68449 7559/sbcl
tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN 0 3938 -
tcp6 0 0 :::6000 :::* LISTEN 0 3937 -
udp 0 0 0.0.0.0:68 0.0.0.0:* 0 4528 -
doesn't 'lsof -Ua' help?
You can likely find the shared sockets by parsing /proc/net/tcp (and similar "files" for other protocols). There's some docs on /proc/net/tcp here.
You would need to find the socket (perhaps by its IP addresses/port numbers ?) and parse out the inode number. Once you have the inode, you can search through all of /proc/*/fd/* , calling stat for every link and inspect the st_ino member of struct stat until you find a match.
The inode number should match between the 2 processes, so when you've gone through all /proc/*/fd/* you should have found them both.
If what you do know is the process id and socket fd of the first, you might not need to go through /proc/net/tcp, all you need to do is stat the /proc/<pid>/fd/<fd> and search the rest of /proc/*/fd/* for a matching inode. You'd need /proc/net/tcp if you want to fetch the ip addresses/port number though - which you can find if you know the inode number
For purposes creating a test case, consider a situation where multiple ssh-agent processes are running and have open sockets. I.e. A user runs ssh-agent multiple times and loses the socket/PID information given when the agent started:
$ find /tmp -path "*ssh*agent*" 2>/dev/null
/tmp/ssh-0XemJ4YlRtVI/agent.14405
/tmp/ssh-W1Tl4i8HiftZ/agent.21283
/tmp/ssh-w4fyViMab8wr/agent.10966
Later, the user wants to programmatically determine the PID owner of a particular ssh-agent socket (i.e. /tmp/ssh-W1Tl4i8HiftZ/agent.21283):
$ stat /tmp/ssh-W1Tl4i8HiftZ/agent.21283
File: '/tmp/ssh-W1Tl4i8HiftZ/agent.21283'
Size: 0 Blocks: 0 IO Block: 4096 socket
Device: 805h/2053d Inode: 113 Links: 1
Access: (0600/srw-------) Uid: ( 4000/ myname) Gid: ( 4500/ mygrp)
Access: 2018-03-07 21:23:08.373138728 -0600
Modify: 2018-03-07 20:49:43.638291884 -0600
Change: 2018-03-07 20:49:43.638291884 -0600
Birth: -
In this case, because ssh-agent named its socket nicely as a human onlooker can guess that the socket belongs to PID 21284, because the socket name contains a numeric component that is one-off from a PID identified with ps:
$ ps -ef | grep ssh-agent
myname 10967 1 0 16:54 ? 00:00:00 ssh-agent
myname 14406 1 0 20:35 ? 00:00:00 ssh-agent
myname 21284 1 0 20:49 ? 00:00:00 ssh-agent
It seems highly unwise to make any assumption that the PIDs will be so reliable as to always only be off by one, but also, one might suppose that not all socket creators will name the sockets so nicely.
#Cypher's answer points to a straightforward solution to the problem of identifying the PID of the socket owner, but is incomplete as lsof actually can only identify this PID with elevated permissions. Without elevated permissions, no results are forthcoming:
$ lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283
$
With elevated permissions, however, the PID is identified:
$ sudo lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ssh-agent 21284 myname 3u unix 0xffff971aba04cc00 0t0 1785049 /tmp/ssh-W1Tl4i8HiftZ/agent.21283 type=STREAM
In this case, the owner of the PID (myname) and socket was the one doing the query, so it seemed elevated permissions should not be needed. Furthermore, the task performing the query was not supposed to be able to elevate permissions, so I looked for another answer.
This led me to #whoplisp's answer proposing netstat -tulpen as a solution to the OP's problem. While it may have been effective for the OP, the command line is too restrictive to serve as a general purpose command and was completely ineffective in this case (even with elevated permissions).
$ sudo netstat -tulpen | grep -E -- '(agent.21283|ssh-agent)'
$
netstat, however, can come close if a different command-line is used:
$ netstat -ap | grep -E -- '(agent.21283)'
(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)
unix 2 [ ACC ] STREAM LISTENING 1785049 - /tmp/ssh-W1Tl4i8HiftZ/agent.21283
Sadly, here too, the PID is elusive without elevated permissions:
$ sudo netstat -ap | grep -E -- '(agent.21283|ssh-agent)'
unix 2 [ ACC ] STREAM LISTENING 1765316 10967/ssh-agent /tmp/ssh-w4fyViMab8wr/agent.10966
unix 2 [ ACC ] STREAM LISTENING 1777450 14406/ssh-agent /tmp/ssh-0XemJ4YlRtVI/agent.14405
unix 2 [ ACC ] STREAM LISTENING 1785049 21284/ssh-agent /tmp/ssh-W1Tl4i8HiftZ/agent.21283
Of the two solutions, however, lsof clearly wins at the races:
$ time sudo netstat -ap | grep -E -- '(agent.21283|ssh-agent)' >/dev/null
real 0m5.159s
user 0m0.010s
sys 0m0.019s
$ time sudo lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283 >/dev/null
real 0m0.120s
user 0m0.038s
sys 0m0.066s
Yet another tool exists according to the netstat man page:
$ man netstat | grep -iC1 replace
NOTES
This program is mostly obsolete. Replacement for netstat is ss. Replacement for netstat -r is ip route. Replacement for netstat -i
is ip -s link. Replacement for netstat -g is ip maddr.
Sadly, ss also requires elevated permissions to identify the PID, but, it beats both netstat and lsof execution times:
$ time sudo ss -ap | grep -E "(agent.21283|ssh-agent)"
u_str LISTEN 0 128 /tmp/ssh-w4fyViMab8wr/agent.10966 1765316 * 0 users:(("ssh-agent",pid=10967,fd=3))
u_str LISTEN 0 128 /tmp/ssh-0XemJ4YlRtVI/agent.14405 1777450 * 0 users:(("ssh-agent",pid=14406,fd=3))
u_str LISTEN 0 128 /tmp/ssh-W1Tl4i8HiftZ/agent.21283 1785049 * 0 users:(("ssh-agent",pid=21284,fd=3))
real 0m0.043s
user 0m0.018s
sys 0m0.021s
In conclusion, it might seem that for some PID identification, it appears that elevated permissions are required.
Note: Not all operating systems require elevated permissions. For example, SCO Openserver 5.0.7's lsof seemed to work just fine without elevating permissions.
Caveat: This answer may fail with respect to the OP's qualification for finding "the original creator" of the socket. In the example used, no doubt PID 21283 was the originator of the socket's creation as this PID is identified in the socket name. Neither lsof nor netstat identified PID 21283 as the original creator, though clearly PID 21284 is the current maintainer.
Is there a Linux library that will enable me to tell what IP sockets are owned by what processes? I guess I'm looking for the programmatic equivalent of lsof -i. Ultimately, I want to correlate packets seen through libpcap to processes.
UPDATE: A couple of people have suggested using /proc/<pid>/net/tcp and udp, but on my system, the same data is shown for every process, so it doesn't help.
I think you first have to look through the open fds in /proc/*/fd, e.g.
4 -> socket:[11147]
and then look for the referenced sockets (by the inode) in /proc/net/tcp (or /proc/net/udp), e.g.
12: B382595D:8B40 D5C43B45:0050 01 00000000:00000000 00:00000000 00000000 1000 0 11065 1 ffff88008bd35480 69 4 12 4 -1
To determine sockets owned by a process you can just use netstat. Here's an example w/output (shortened) of netstat with options that will do what you want.
$ sudo netstat -apeen
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 127.0.0.1:8118 0.0.0.0:* LISTEN 138 744850 13248/privoxy
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 117 9612 2019/postgres
udp 0 0 127.0.0.1:51960 127.0.0.1:51960 ESTABLISHED 117 7957 2019/postgres
udp 0 0 0.0.0.0:68 0.0.0.0:* 0 7740 1989/dhclient
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 7937 2019/postgres /var/run/postgresql/.s.PGSQL.5432
unix 2 [ ACC ] STREAM LISTENING 958058 8080/emacs /tmp/emacs1000/server
unix 2 [ ACC ] STREAM LISTENING 6969 1625/Xorg /tmp/.X11-unix/X0
unix 2 [ ] DGRAM 9325 1989/dhclient
unix 3 [ ] STREAM CONNECTED 7720 1625/Xorg #/tmp/.X11-unix/X0
Make sure you run netstat as root otherwise you'll get this message:
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
An explanation of the -apeen options from the netstat manpage:
-a, --all
Show both listening and non-listening sockets. With the
--interfaces option, show interfaces that are not up
-p, --program
Show the PID and name of the program to which each socket
belongs.
-e, --extend
Display additional information. Use this option twice for
maximum detail.
--numeric , -n
Show numerical addresses instead of trying to determine symbolic host, port or user names.
--numeric-hosts
shows numerical host addresses but does not affect the resolution of port or user names.
--numeric-ports
shows numerical port numbers but does not affect the resolution of host or user names.
--numeric-users
shows numerical user IDs but does not affect the resolution of host or port names.
The /proc filesystem provides details on each process, including networking information. Open socket information is listed in /proc/net/tcp. The IPv6 sockets are listed separately in the tcp6 file. The socket information includes information such as the local and remote ports, and the socket inode number, which can be mapped back to the process by parsing the /proc/{pid}/fd/* information.
If you aren't familiar with the /proc filesystem, it is basically a virtual filesystem that allows the kernel to publish all sorts of useful information to user-space. The files are normally simple structured text files that are easy to parse.
For example, on my Ubuntu system I used netcat for testing, and ran nc -l -p 8321 to listen on port 8321. Looking at the tcp socket information:
$ cat /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:2081 00000000:0000 0A 00000000:00000000 00:00000000 00000000 1000 0 26442 1 de0c8e40 300 0 0 2 -1
1: 0100007F:0277 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7019 1 de0c84c0 300 0 0 2 -1
The first line shows it is listening on all addresses to point 8321 (0x2081). The inode number is 26442, which we can use to look up the matching pid in /proc/{pid}/fd/*, which consists of a bunch of symlinks from the file handle number to the device. So if we look up the pid for netcat, and check its fd mapping:
$ ls -l /proc/7266/fd
total 0
lrwx------ 1 gavinb gavinb 64 2009-12-31 09:10 0 -> /dev/pts/1
lrwx------ 1 gavinb gavinb 64 2009-12-31 09:10 1 -> /dev/pts/1
lrwx------ 1 gavinb gavinb 64 2009-12-31 09:10 2 -> /dev/pts/1
lrwx------ 1 gavinb gavinb 64 2009-12-31 09:10 3 -> socket:[26442]
And there we see that file descriptor 3 in this process is mapped to the socket with inode 26442, just as we expect.
So obviously to build a complete map of sockets, you will need to first enumerate all the /proc/**/fd/* files, look up the socket symlinks, then match the socket inode against the tables from /proc/net/tcp which has the endpoint information.
This is the way the lsof tool works (see lsof/dialects/linux/dsocket.c for the implementation).
Wikipedia on procfs
The Linux /proc filesystem as a Programmer's Tool
/proc/<pid>/net is equivalent to /proc/net for all processes in the same network namespace as you – in other words, it's "global" information.
You can do what lsof and fuser do, which is to iterate through both /proc/<pid>/fd/* and /proc/net/* looking for matching inodes. Quick demonstration:
#!/bin/sh
pgrep "$#" | while read pid; do
for fd in /proc/$pid/fd/*; do
name=$(readlink $fd)
case $name in
socket:\[*\])
ino=${name#*:}
for proto in tcp:10 tcp6:10 udp:10 udp6:10 unix:7; do
[[ ! -e /proc/net/${proto%:*} ]] ||
awk "
\$${proto##*:} == ${ino:1:${#ino}-2} {
print \"${proto%:*}:\", \$0
exit 1
}
" /proc/net/${proto%:*} || break
done
;;
esac
done
done
You can extend this to other protocols (I see ax25, ipx, packet, raw, raw6, udplite, udp6lite in /proc/net/ too) or rewrite in a language of your choosing.
You can read them from proc filesystem. The 'files' you probably want to look at are found in
/proc/<pid>/net (namely tcp, udp, unix)
Here's some examples on using the proc filesystem
You could try running lsof with strace and see just which files in /proc it gets data from.
I'd go to the source:
http://ubuntuforums.org/showthread.php?t=1346778