I used this command to convert multiple pcap log files to text using tcpdump :
$ cat /home/dalya/snort-2.9.9.0/snort_logs/snort.log.* | tcpdump -n -r - > /home/dalya/snort-2.9.9.0/snort_logs2/bigfile.txt
and it worked well.
Now I want to separate the output, each converted file in a separate output file using loop like this :
for f in /home/dalya/snort-2.9.9.0/snort_logs/snort.log.* ; do
tcpdump -n -r "$f" > /home/dalya/snort-2.9.9.0/snort_logs2/"$f.txt" ;
done
But it gave me :
bash: /home/dalya/snort-2.9.9.0/snort_logs2//home/dalya/snort-2.9.9.0/snort_logs/snort.log.1485894664.txt: No such file or directory
bash: /home/dalya/snort-2.9.9.0/snort_logs2//home/dalya/snort-2.9.9.0/snort_logs/snort.log.1485894770.txt: No such file or directory
bash: /home/dalya/snort-2.9.9.0/snort_logs2//home/dalya/snort-2.9.9.0/snort_logs/snort.log.1487346947.txt: No such file or directory
I think the problem in $f, Where did I go wrong?
If you run
for f in /home/dalya/snort-2.9.9.0/snort_logs/snort.log.* ; do
echo $f
done
You'll find that you're getting
/home/dalya/snort-2.9.9.0/snort_logs/snort.log.1485894664
/home/dalya/snort-2.9.9.0/snort_logs/snort.log.1485894770
/home/dalya/snort-2.9.9.0/snort_logs/snort.log.1487346947
You can use basename
To get only the filename, something like this:
for f in /home/dalya/snort-2.9.9.0/snort_logs/snort.log.* ; do
base="$(basename $f)"
echo $base
done
Once you're satisfied that this is working, remove the echo statement and use
tcpdump -n -r "$f" > /home/dalya/snort-2.9.9.0/snort_logs2/"$base.txt"
instead.
Edit: tcpdump -n -r "$base" > ... should have been tcpdump -n -r "$f" > ...; you only want to use $base in the context of creating the new filename, not in the context of reading the existing data.
Related
I am trying to read a directory with "ls" and do operations on it
directory example:
$ ls -1
x x
y y
z z
script file: myScript.sh
#!/bin/bash
files=(`ls -1`);
for ((i=0; i<"${#files[#]}"; i+=1 )); do
echo "${files[$i]}"
done
however, the output is
$ myScript.sh
x
x
y
y
z
z
yet if I define "files" in the following way
$ files=("x x" "y y" "z z")
$ for ((i=0; i<"${#files[#]}"; i+=1 )); do echo "${files[$i]}"; done
x x
y y
z z
How can I preserve the spaces in "files=(`ls -1`)"?
Don't.
See:
ParsingLs
BashPitfalls #1
If at all possible, use a shell glob instead.
That is to say:
files=( * )
If you need to represent filenames as a stream of text, use NUL delimiters.
That is to say, either:
printf '%s\0' *
or
find . -mindepth 1 -maxdepth 1 -print0
will emit a NUL-delimited string, which you can load into a shell array safely using (in modern bash 4.x):
readarray -d '' array < <(find . -mindepth 1 -maxdepth 1 -print0)
...or, to support bash 3.x:
array=( )
while IFS= read -r -d '' name; do
array+=( "$name" )
done < <(find . -mindepth 1 -maxdepth 1 -print0)
In either of the above, that find command potentially being on the other side of a FIFO, network stream, or other remoting layer (assuming that there's some complexity of that sort stopping you from using a native shell glob).
It seems the main conclusion is not to use ls. Back in Pleistocene age of Unix programming, they used ls; however, these days, ls is best-restricted to producing human-readable displays only. A robust script for anything that can be thrown at your script (end lines, white spaces, Chinese characters mixed with Hebrew and French, or whatever), is best achieved by some form of globbing (as recommended by others here BashPitfalls).
#!/bin/bash
for file in ./*; do
[ -e "${file}" ] || continue
# do some task, for example, test if it is a directory.
if [ -d "${file}" ]; then
echo "${file}"
fi
done
The ./ is maybe not absolutely necessary, but it may help if the file begins with a "-", clarifying which file has the return line (or lines), and likely some other nasty buggers. This is also a useful template for specific files (.e.g, ./*.pdf). For example, suppose somehow the following files are in your directory: "-t" and "<CR>t". Then (revealing other issues with ls when using nonstandard characters)
$ ls
-t ?t
$ for file in *; do ls "${file}"; done
-t ?t
?t
whereas:
$ for file in ./*; do ls "${file}"; done
./-t
./?t
also
$ for file in ./*; do echo "${file}"; done
./-t
./
t
A workaround with POSIX commands can be achieved by --
$ for file in *; do ls -- "${file}"; done # work around
-t
?t
Try this:
eval files=($(ls -Q))
Option -Q enables quoting of filenames.
Option -1 is implied (not needed), if the output is not a tty.
I have around a million files in one folder in the form xxxx_description.jpg where xxx is a number ranging from 100 to an unknown upper.
The list is similar to this:
146467_description1.jpg
146467_description2.jpg
146467_description3.jpg
146467_description4.jpg
14646_description1.jpg
14646_description2.jpg
14646_description3.jpg
146472_description1.jpg
146472_description2.jpg
146472_description3.jpg
146500_description1.jpg
146500_description2.jpg
146500_description3.jpg
146500_description4.jpg
146500_description5.jpg
146500_description6.jpg
To get the file number down in the at folder I'd like to put them all into folders grouped by the number at the start.
ie:
146467/146467_description1.jpg
146467/146467_description2.jpg
146467/146467_description3.jpg
146467/146467_description4.jpg
14646/14646_description1.jpg
14646/14646_description2.jpg
14646/14646_description3.jpg
146472/146472_description1.jpg
146472/146472_description2.jpg
146472/146472_description3.jpg
146500/146500_description1.jpg
146500/146500_description2.jpg
146500/146500_description3.jpg
146500/146500_description4.jpg
146500/146500_description5.jpg
146500/146500_description6.jpg
I was thinking to try and use command line: find | awk {} | mv command or maybe write a script, but I'm not sure how to do this most efficiently.
If you really are dealing with millions of files, I suspect that a glob (*.jpg or [0-9]*_*.jpg may fail because it makes a command line that's too long for the shell. If that's the case, you can still use find. Something like this might work:
find /path -name "[0-9]*_*.jpg" -exec sh -c 'f="{}"; mkdir -p "/target/${f%_*}"; mv "$f" "/target/${f%_*}/"' \;
Broken out for easier reading, this is what we're doing:
find /path - run find, with /path as a starting point,
-name "[0-9]*_*.jpg" - match files that match this filespec in all directories,
-exec sh -c execute the following on each file...
'f="{}"; - put the filename into a variable...
mkdir -p "/target/${f%_*}"; - make a target directory based on that variable (read mkdir's man page about the -p option)
mv "$f" "/target/${f%_*}/"' - move the file into the directory.
\; - end the -exec expression
On the up side, it can handle any number of files that find can handle (i.e. limited only by your OS). On the down side, it's launching a separate shell for each file to be handled.
Note that the above answer is for Bourne/POSIX/Bash. If you're using CSH or TCSH as your shell, the following might work instead:
#!/bin/tcsh
foreach f (*_*.jpg)
set split = ($f:as/_/ /)
mkdir -p "$split[1]"
mv "$f" "$split[1]/"
end
This assumes that the filespec will fit in tcsh's glob buffer. I've tested with 40000 files (894KB) on one command line and not had a problem using /bin/sh or /bin/csh in FreeBSD.
Like the Bourne/POSIX/Bash parameter expansion solution above, this avoids unnecessary calls to external I haven't tested that, and would recommend the find solution even though it's slower.
You can use this script:
for i in [0-9]*_*.jpg; do
p=`echo "$i" | sed 's/^\([0-9]*\)_.*/\1/'`
mkdir -p "$p"
mv "$i" "$p"
done
Using grep
for file in *.jpg;
do
dirName=$(echo $file | grep -oE '^[0-9]+')
[[ -d $dirName ]] || mkdir $dirName
mv $file $dirName
done
grep -oE '^[0-9]+' extracts the starting digits in the filename as
146467
146467
146467
146467
14646
...
[[ -d $dirName ]] returns 1 if the directory exists
[[ -d $dirName ]] || mkdir $dirName ensures that the mkdir works only if the test [[ -d $dirName ]] fails, that is the direcotry does not exists
I've a new question about a closed question from me.
In the last one I asked for help in fixing a script, which sorts files to folders by it's content. (Bash script which sorts files to folders by it's content; How to solve wildcard in variables?)
Now I have a new problem with that.
The variables had changed. The old ones where single word variables in an array, now I've multiple words with special characters as variable.
Here is my script:
#!/bin/bash
declare -a standorte;
standorte=('Zweigst: 00' 'Zweigst: 03' 'Zweigst: 08')
ls lp.3.* | while read f
do
for ort in "${standorte[#]}"; do
grep -i $ort "$f" >/dev/null 2>&1
if [ $? -eq 0 ]; then
echo Copying $f to $ort
cp "$f" $ort
fi
done
done
Now you see, the "ort" is the folder name. So the script try to copy the file lp.3.* to e.g. Zweigst: 00. But without the escape backslashes it doesn't work. Put I escape charakters into the variable, the script doesn't work, because in the file lp.3.* is no "Zweigst:\ 00".
I think, I must declare a new variable for "ort" where I put the folder names in it.
But I've no idea how to change the for loop. I must say the script, when you found Zweigst: 00 copy this file to folder "zweigst00". I'm sorry my bash script experience is not good at all. I can't change this by my own.
I have multiple (zero to unlimited) lp.3.* files (e.g. lp.3.1, lp.3.2, lp.3.5.2 and so on)
In this files is this text: http://pastebin.com/0ZzCUrpx
You just need to quote the variable:
for ort in "${standorte[#]}"; do
grep -i "$ort" "$f" >/dev/null 2>&1
# ^----^-------- quotes needed
if [ $? -eq 0 ]; then
echo Copying $f to $ort
cp "$f" "$ort"
# ^----^-------- quotes needed
fi
done
Why? Because otherwise this
grep -i $ort "$f" >/dev/null 2>&1
gets expanded as something that grep cannot understand properly:
grep -i Zweigst: 00 "$f" >/dev/null 2>&1
see that it is trying to grep Zgeigst: from file 00.
I need to get a list of files added to a master folder and copy only the new files to the respective backup folders; The paths to each folder have multiple folders, all named by numbers and only 1 level deep.
ie /tester/a/100
/tester/a/101 ...
diff -r returns typically "Only in /testing/a/101: 2093_thumb.png" per line in the diff.txt file generated.
NOTE: there is a space after the colon
I need to get the 101 from the path and filename into separate variables and copy them to the backup folders.
I need to get the lesserfolder var to get 101 without the colon
and mainfile var to get 2093_thumb.png from each line of the diff.txt and do the for loop but I can't seem to get the $file to behave. Each time I try testing to echo the variables I get all the wrong results.
#!/bin/bash
diff_file=/tester/diff.txt
mainfolder=/testing/a
bacfolder= /testing/b
diff -r $mainfolder $bacfolder > $diff_file
LIST=`cat $diff_file`
for file in $LIST
do
maindir=$file[3]
lesserfolder=
mainfile=$file[4]
# cp $mainfolder/$lesserFolder/$mainfile $bacfolder/$lesserFolder/$mainfile
echo $maindir $mainfile $lesserfolder
done
If I could just get the echo statement working the cp would work then too.
I believe this is what you want:
#!/bin/bash
diff_file=/tester/diff.txt
mainfolder=/testing/a
bacfolder= /testing/b
diff -r -q $mainfolder $bacfolder | egrep "^Only in ${mainfolder}" | awk '{print $3,$4}' > $diff_file
cat ${diff_file} | while read foldercolon mainfile ; do
folderpath=${foldercolon%:}
lesserFolder=${folderpath#${mainfolder}/}
cp $mainfolder/$lesserFolder/$mainfile $bacfolder/$lesserFolder/$mainfile
done
But it is much more reliable (and much easier!) to use rsync for this kind of backup. For example:
rsync -a /testing/a/* /testing/b/
You could try a while read loop
diff -r $mainfolder $bacfolder | while read dummy dummy dir file; do
echo $dir $file
done
Consider this simple loop:
for f in *.{text,txt}; do echo $f; done
I want to echo ONLY valid file names. Using the $f variable in a script everything works great unless there aren't any files of that extension. In the case of an empty set, $f is set to *.text and the above line echos:
*.text
*.txt
rather than echoing nothing. This creates an error if you are trying to use $f for anything that is expecting an actual real file name and instead gets *.
If there are any files that match the wildcard so that it is not an empty set everything works as I would like. e.g.
123.text
456.txt
789.txt
How can I do this without the errors and without seemingly excessive complexity of first string matching $f for an asterisk?
Set the nullglob option.
$ for f in *.foo ; do echo "$f" ; done
*.foo
$ shopt -s nullglob
$ for f in *.foo ; do echo "$f" ; done
$
You can test if the file actually exists:
for f in *.{text,txt}; do if [ -f $f ]; then echo $f; fi; done
or you can use the find command:
for f in $(find -name '*.text' -o -name '*.txt'); do
echo $f
done
Also, if you can afford the external use of ls, you can escape its results by using
for f in `ls *.txt *.text`;