How could I redirect output to multiple targets, say stdout, file, socket and so?
say, i have a system here and connected to some network. When it fails, the guy supervises it via ssh should be able to notice it, or the GUI client should receive the error info, or, in the worst case, we can still find something in the log.
or even more targets. Atomicity may or may not need to be guaranteed.
so how to do this in bash and/or in C?
I think you are looking for the "tee" command.
You can redirect with tee to any number of files and to any commands too, like:
seq 50 | tee copy1 copy2 >((echo Original linecount: $(grep -c ''))>&2) | grep '9'
what prints:
9
19
29
39
49
Original linecount: 50 #printed to stderr
or
seq 50 | tee copy1 copy2 >((echo Original linecount: $(grep -c ''))>&2) | grep '9' | wc -l
what prints the count of numbers containing digit 9 in first 50 numbers, while make two copyes of the original sequence...
Original linecount: 50 #stderr
5
Related
I want to save 1 line from the output of top into a Bash array to later access its components:
$ timeout 1 top -d 2 | awk 'NR==8'
2436 USER 20 0 1040580 155268 91100 S 6.2 1.0 56:38.94 Xorg
Terminated
I tried:
$ gpu=($(timeout 1s top -d 2 | awk 'NR==8'))
$ mapfile -t gpu < <($(timeout 1s top -d 2 | awk 'NR==8'))
and, departing from the array requisite, even:
$ read -r gpu < <(timeout 1s top -d 2 | awk 'NR==8')
all returned a blank for either ${gpu[#]} (first two) or $gpu (last).
Edit:
As pointed out by #Cyrus and others gpu=($(top -n 1 -d 2 | awk 'NR==8')) is the obvious solution. However I want to build the cmd dynamically so top -d 2 may be replaced by other cmds such as htop -d 20 or intel_gpu_top -s 1. Only top can limit its maximum number of iterations, so that is not an option in general, and for that reason I resort to timeout 1s to kill the process in all shown attempts...
End edit
Using a shell other than Bash is not an option. Why did the above attempts fail and how can I achieve that ?
Why did the above attempts fail
Because redirection to pipe does not have terminal capabilities, top process receives SIGTTOU signal when it tries to write the terminal and take the terminal "back" from the shell. The signal causes top to terminate.
how can I achieve that ?
Use top -n 1. Generally, use the tool specific options to disable using terminal utilities by that tool.
However I want to build the cmd dynamically so top -d 2 may be replaced by other cmds such as htop -d 20 or intel_gpu_top -s 1
Write your own terminal emulation and extract the first line from the buffer of the first stuff the command displays. See GNU screen and tmux source code for inspiration.
I dont think you need the timeout there if its intended to quit top. You can instead use the -n and -b flags but feel free to add it if you need it
#!/bin/bash
arr=()
arr[0]=$(top -n 1 -b -d 2 | awk 'NR==8')
arr[1]=random-value
arr[2]=$(top -n 1 -b -d 2 |awk 'NR==8')
echo ${arr[0]}
echo ${arr[1]}
echo ${arr[2]}
output:
1 root 20 0 99868 10412 7980 S 0.0 0.5 0:00.99 systemd
random-value
1 root 20 0 99868 10412 7980 S 0.0 0.5 0:00.99 systemd
from top man page:
-b :Batch-mode operation
Starts top in Batch mode, which could be useful for sending output from top to other programs or to a
file. In this mode, top will not accept input and runs until the iterations limit you've set with the
`-n' command-line option or until killed.
-n :Number-of-iterations limit as: -n number
Specifies the maximum number of iterations, or frames, top should produce before ending.
-d :Delay-time interval as: -d ss.t (secs.tenths)
Specifies the delay between screen updates, and overrides the corresponding value in one's personal
configuration file or the startup default. Later this can be changed with the `d' or `s' interactive
commands.
bash as I can tell from the repetition of an IP within a log through a specific search?
By example:
#!/bin/bash
# Log line: [Sat Jul 04 21:55:35 2015] [error] [client 192.168.1.39] Access denied with status code 403.
grep "status\scode\s403" /var/log/httpd/custom_error_log | while read line ; do
pattern='^\[.*?\]\s\[error\]\s\[client\s(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\].*?403'
[[ $line =~ $pattern ]]
res_remote_addr="${BASH_REMATCH[1]}.${BASH_REMATCH[2]}.${BASH_REMATCH[3]}.${BASH_REMATCH[4]}"
echo "Remote Addr: $res_remote_addr"
done
I need to know the end results obtained a few times each message 403 ip, if possible sort highest to lowest.
By example output:
200.200.200.200 50 times.
200.200.200.201 40 times.
200.200.200.202 30 times.
... etc ...
This we need to create an html report from a monthly log of apache in a series of events (something like awstats).
there are better ways. following is my proposal, which should be more readable and easier to maintain:
grep -P -o '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}' log_file | sort | uniq -c | sort -k1,1 -r -n
output should be in a form of:
count1 ip1
count2 ip2
update:
filter only 403:
grep -P -o '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}(?=.*403)' log_file | sort | uniq -c | sort -k1,1 -r -n
notice that a look ahead would suffice.
If log file is in the format as mentioned in question, the best is to use awk to filter out the status code needed plus output only the IP. Then use the uniq command to count each occurence:
awk '/code 403/ {print $8}' error.log | sort | uniq -c |sort -n
In awk, we filter by regexp /code 403/ and then for matching lines we print the 8th value (values are separated by whitespace), which is the IP.
Then we need to sort the output, so that the same IPs are one after another - this is requirement of the uniq program.
uniq -c prints each unique line from input only once - and preceded by the number of occurences. Finnaly we sort this list numericaly to get the IPs sorted by count.
Sample output (first is no. of occurences, second is IP):
1 1.1.1.1
10 2.2.2.2
12 3.3.3.3
I'm using the command grep 3 times on the same line like this
ls -1F ./ | grep / | grep -v 0_*.* | grep -v undesired_result
is there a way to combine them into one command instead of having it to pipe it 3 times?
There's no way to do both a positive search (grep <something>) and a negative search (grep -v <something>) in one command line, but if your grep supports -E (alternatively, egrep), you could do ls -1F ./ | grep / | grep -E -v '0_*.*|undesired_result' to reduce the sub-process count by one. To go beyond that, you'd have to come up with a specific regular expression that matches either exactly what you want or everything you don't want.
Actually, I guess that first sentence isn't entirely true if you have egrep, but building the proper regular expression that correctly includes both the positive and negative parts and covers all possible orderings of the parts might be more frustrating than it's worth...
*/10 * * * * /usr/bin/flock -x -w 10 /tmp/craigslist.lock /usr/bin/lynx -width=120 -dump "http://sfbay.craigslist.org/search/roo/sfc?query=&srchType=A&minAsk=&maxAsk=1100&nh=6&nh=8&nh=16&nh=24&nh=17&nh=21&nh=22&nh=23&nh=27" | grep "sort by most recent" -A 53 > /home/winchell/apartments.txt
*/10 * * * * /usr/bin/flock -x -w 10 /tmp/craigslist.lock /usr/bin/php /home/winchell/apartments.php
This is a cron job. The second line php command seems to be executing even while lynx is writing to apartments.txt, and I don't see the reason. Is this correct usage assuming I'm trying to prevent read from apartments.txt while lynx/grep are writing to it? Thanks!
Your usage is not correct. Notice how your first cron job is a pipeline consisting of two commands:
/usr/bin/flock -x -w 10 /tmp/craigslist.lock /usr/bin/lynx -width=120 -dump
"http://sfbay.craigslist.org/search/roo/sfc?query=&srchType=A&minAsk=&maxAsk=1100&nh=6&nh=8&nh=16&nh=24&nh=17&nh=21&nh=22&nh=23&nh=27"
which is then piped to:
grep "sort by most recent" -A 53 > /home/winchell/apartments.txt
So the first command is locking a file but it's the second command that's writing to that file! The second command will happily execute without waiting for the lock.
One way to fix this would be to write the file while holding the lock:
lynx etc... | grep etc.. |
flock -x -w 10 /tmp/craigslist.lock tee /home/winchell/apartments.txt
The disadvantage of this approach is that lynx and grep run even if the file is locked. To prevent this, you will have to run the whole thing under the lock:
flock -x -w 10 /tmp/craigslock.lock sh -c "lynx etc... | grep etc... >thefile"
With this approach you will have to pay careful attention to quoting as the URL argument of lynx as it will require double quoting.
Finally: consider using curl or wget instead of lynx. lynx is meant for interactive usage!
My program gets executed like:
$./sort 1 < test.txt
sort is the program name
1 is the argument (argv[1])
and test.txt is the file I am inputting from
Is it possible to extract the name file from this? if so how?
The problem is I already wrote my whole program as if I could extract the name from the input line, so I need to be able to pass it into arguments.
Any help is appreciated,
Thanks!
You can't. The shell opens (open(2)) that file and sets up the redirect (most likely using dup2).
The only possible way would be for the shell to explicitly export the information in an environment variable that you could read via getenv.
But it doesn't always make sense. For example, what file name would you expect from
$ echo "This is the end" | ./sort 1
Though this can't be done portably, it's possible on Linux by calling readlink on /proc/self/fd/0 (or /proc/some_pid/fd/0).
eg, running:
echo $(readlink /proc/self/fd/0 < /dev/null)
outputs:
/dev/null
No you can't: the shell sends the content of test.txt to the standard input of your program.
Look at this:
sort << _EOF
3
1
2
_EOF
The < > | operators are processed by the shell, they alter standard input,output,error of the programs in the cmd line.
If you happen to run Solaris, you could parse pfiles output to get the file associated, if any, with stdin.
$ /usr/bin/sleep 3600 < /tmp/foo &
[1] 8430
$ pfiles 8430
8430: /usr/bin/sleep 3600
Current rlimit: 65536 file descriptors
0: S_IFREG mode:0600 dev:299,2 ino:36867886 uid:12345 gid:67890 size=123
O_RDONLY|O_LARGEFILE
/tmp/foo
1: S_IFCHR mode:0600 dev:295,0 ino:12569206 uid:12345 gid:67890 rdev:24,2
...
On most Unix platforms, you will also get the same information from lsof -p if this freeware is installed.