zsh process file script - loops

I'm attempting to have this process a number of files but I don't want it in a look so I don't have to monitor it.
#!/usr/local/bin/zsh
X=${1-20}
for (( N=1; N<=X; N++ )); do
for p in *.xml; do
curl -X POST -H "Content-Type:application/xml" -d "#${p}" "https://url /postAPI" > "post_${p}"
sleep 1
done
done
When doing ./work.sh 5 this loops forever!
What's causing infinate loop?
Edit Based on a comment below
/tmp/tmp.KeFYeM9Z % ls -l
total 4
-rwxr-xr-x 1 naes wheel 218 Nov 20 14:42 work.sh
#!/usr/local/bin/zsh
X=${1-20}
for (( N=1; N<=X; N++ )); do
for p in /tmp/tmp.u6RnKaJ3/*.xml; do
curl -X POST -H "Content-Type:application/xml" -d "#${p}" "https://url /postAPI"
sleep 1
done
done
This still continues the infinite loop
This doesn't:
% cat work1.sh
#!/usr/local/bin/zsh
X=${1-20}
for (( N=1; N<=X; N++ )); do
date
sleep 1
done
% ./work1.sh 5
Thu Nov 20 15:22:27 PST 2014
Thu Nov 20 15:22:28 PST 2014
Thu Nov 20 15:22:29 PST 2014
Thu Nov 20 15:22:30 PST 2014
Thu Nov 20 15:22:31 PST 2014
What in my loop causes the infinite?

You are writing to the same directory you are reading from. So while reading the xml files you are writing xml files making it essentially loop forever. Although it's not really an infinite loop, it's a very large one.
Let's say you have 10 files, in that case you'll have this result:
N=1: |p| = 100
N=2: |p| = 200
N=3: |p| = 400
N=4: |p| = 800
N=5: |p| = 1600
So... it groes quite fast.
This should do the trick:
#!/usr/local/bin/zsh
X=${1-20}
OUTPUT_DIR=/tmp/output/
mkdir -p $OUTPUT_DIR
cd /tmp/tmp.u6RnKaJ3
for (( N=1; N<=X; N++ )); do
echo "Attempt $N"
for p in *.xml; do
curl -X POST -H "Content-Type:application/xml" -d "#${p}" "https://url /postAPI" > "${OUTPUT_DIR}post_${p}"
sleep 1
done
done

Related

Set Command in for-loop

I'm writing a script where I put local users in an array and use them in a for loop while parsing with set to display the last command information. I'm able to parse with just the commands inside of the "do" section successfully if run in a script by itself.
#!/bin/bash
#Automation Project (Khal)
#Author: Zacquille Joseph
#Version 1.2
#Get Users on System
user=$(getent passwd {1000..60000} | cut -d: -f1 )
#Puts all Users into Array
array=($user)
for item in ${array[*]}
do
set $(last | grep -w $item | head -n 1 )
echo
echo $*
echo "Name: $1"
echo "Date: $4 $5 $6"
echo "Time: $7"
echo "Status: $7 $8 $9 ${10}"
echo
done
I expect to only have an output of
mary pts/2 X.X.X.X Wed Apr 17 15:31 - 15:36 (00:04)
Name: mary
Date: Wed Apr 17
Time: 15:31
Status: 15:31 - 15:36 (00:04)
zac pts/0 X.X.X.X Tue Apr 30 14:10 still logged in
Name: zac
Date: Tue Apr 30
Time: 14:10
Status: 14:10 still logged in
instead I'm getting that combined with this:
BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_LINENO=([0]="0")
BASH_SOURCE=([0]="./cst1.sh")
BASH_VERSINFO=([0]="4" [1]="4" [2]="19" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
BASH_VERSION='4.4.19(1)-release'
DIRSTACK=()
EUID=1004
GROUPS=()
.....
FYI:
last | grep -w zac | head -n 1
Will return this:
zac pts/0 X.X.X.X Tue Apr 30 14:10 still logged in
Running this:
set $(last | head -n 10 | tail -n 1)
echo
echo $*
echo "Name: $1"
echo "Date: $4 $5 $6"
echo "Time: $7"
echo "Status: $7 $8 $9 ${10}"
echo
will return:
ubuntu pts/4 X.X.X.X Thu Apr 18 22:47 - 01:01 (02:13)
Name: ubuntu
Date: Thu Apr 18
Time: 22:47
Status: 22:47 - 01:01 (02:13)
try using another array instead of set command, eg :
arr2=($(last | grep -w $item | head -n 1 ))
echo
echo $*
echo "Name: ${arr2[0]} "
echo "Date: ${arr2[3]} ${arr2[4]} ${arr2[5]} "
echo "Time: ${arr2[6]}"
echo "Status: ${arr2[6]} ${arr2[7]} ${arr2[8]} ${arr2[9]}"
echo
SOLVED IT! One of the users never logged on but was created. For the purpose of my script all the users have to logon. Ideally i just replaced $item with each user and running the script (luckily we don't have to much users), then found that one specific user who never logged on gave that large output.

trying to run arbitrary commands and parse their output

here is part of code
scanf("%[^\n]%*c",command);
int pid;
pid=fork();
if (pid == 0) {
// Child process
char *argv[]={command ,NULL};
execvp(argv[0], argv);
exit (0);
}
When I give as input ls I want as output
1 copy of mysh1.c mysh1.c mysh3.c mysh.c New Folder
a.out helpmanual.desktop mysh2.c mysh4.c New File
and when i give ls -l /tmp
i'm waiting
total 12
-rw------- 1 antre antre 0 Nov 4 17:31 config-err-KT9sEZ
drwx------ 2 antre antre 4096 Nov 4 19:21 mozilla_antre0
drwx------ 2 antre antre 4096 Jan 1 1970 orbit-antre
drwx------ 2 antre antre 4096 Nov 4 17:31 ssh-HaOFtKdeIQnQ `
but i take:
1 copy of mysh1.c mysh1.c mysh3.c mysh.c New Folder
a.out helpmanual.desktop mysh2.c mysh4.c New File
It seems that you're trying to parse the output of ls -l in a C program for some reason.
That's unlikely to be the “right” thing to do. The usual mechanism is to use opendir and readdir to read the directory file, directly.
If you have some truly strange situation in which you cannot opendir (the only case that comes to mind is if you're running ls on a remote system, eg, over ssh), there is a mode in GNU ls specifically for producing an output record format that can be parsed by another program.
From the GNU coreutils info:
10.1.2 What information is listed
‘-D’
‘--dired’
With the long listing (‘-l’) format, print an additional line after
the main output:
//DIRED// BEG1 END1 BEG2 END2 ...
The BEGN and ENDN are unsigned integers that record the byte
position of the beginning and end of each file name in the output.
This makes it easy for Emacs to find the names, even when they
contain unusual characters such as space or newline, without fancy
searching.
If directories are being listed recursively (‘-R’), output a
similar line with offsets for each subdirectory name:
//SUBDIRED// BEG1 END1 ...
Finally, output a line of the form:
//DIRED-OPTIONS// --quoting-style=WORD
where WORD is the quoting style (*note Formatting the file
names::).
Here is an actual example:
$ mkdir -p a/sub/deeper a/sub2
$ touch a/f1 a/f2
$ touch a/sub/deeper/file
$ ls -gloRF --dired a
a:
total 8
-rw-r--r-- 1 0 Jun 10 12:27 f1
-rw-r--r-- 1 0 Jun 10 12:27 f2
drwxr-xr-x 3 4096 Jun 10 12:27 sub/
drwxr-xr-x 2 4096 Jun 10 12:27 sub2/
a/sub:
total 4
drwxr-xr-x 2 4096 Jun 10 12:27 deeper/
a/sub/deeper:
total 0
-rw-r--r-- 1 0 Jun 10 12:27 file
a/sub2:
total 0
//DIRED// 48 50 84 86 120 123 158 162 217 223 282 286
//SUBDIRED// 2 3 167 172 228 240 290 296
//DIRED-OPTIONS// --quoting-style=literal
Note that the pairs of offsets on the ‘//DIRED//’ line above
delimit these names: ‘f1’, ‘f2’, ‘sub’, ‘sub2’, ‘deeper’, ‘file’.
The offsets on the ‘//SUBDIRED//’ line delimit the following
directory names: ‘a’, ‘a/sub’, ‘a/sub/deeper’, ‘a/sub2’.
Here is an example of how to extract the fifth entry name,
‘deeper’, corresponding to the pair of offsets, 222 and 228:
$ ls -gloRF --dired a > out
$ dd bs=1 skip=222 count=6 < out 2>/dev/null; echo
deeper
Note that although the listing above includes a trailing slash for
the ‘deeper’ entry, the offsets select the name without the
trailing slash. However, if you invoke ‘ls’ with ‘--dired’ along
with an option like ‘--escape’ (aka ‘-b’) and operate on a file
whose name contains special characters, notice that the backslash
is included:
$ touch 'a b'
$ ls -blog --dired 'a b'
-rw-r--r-- 1 0 Jun 10 12:28 a\ b
//DIRED// 30 34
//DIRED-OPTIONS// --quoting-style=escape
If you use a quoting style that adds quote marks (e.g.,
‘--quoting-style=c’), then the offsets include the quote marks. So
beware that the user may select the quoting style via the
environment variable ‘QUOTING_STYLE’. Hence, applications using
‘--dired’ should either specify an explicit
‘--quoting-style=literal’ option (aka ‘-N’ or ‘--literal’) on the
command line, or else be prepared to parse the escaped names.
i just only needed to use strtok

unix shell : create different tar files using a string in a array

We have services that generate files named with modules names. For ex., acqDou module generates
acqDou_0001.out
acqDou_08981.out
acqDou_23423.out
The acq_cheat module generates files with almost same name, but last numbering is a bit diffrent.
My Requirment is to tar these files on the basis of module names, one tar file per module, such as acqDou.tar and acq_cheat.tar
Below is the programm I attmepted:
cd /home/swap/output/outfiles
for i in *;
do
j=`echo $i | grep -v 'out$'`
if [ -z $j ];
then continue;
fi
count1=$(echo $i | tr -d -c '_' | wc -m)
if [ $count1 -eq 2 ]
then
two=`ls -1 $i | cut -d'_' -f2 | cut -d'.' -f1 | uniq`
else
two=`ls -1 $i | cut -d'_' -f1 | cut -d'.' -f1 | uniq`
fi
_FILE="${_FILE} $two"
done
_FILE2=`echo "${_FILE[#]}" | tr ' ' '\n' | sort -u | tr '\n' ' '`
echo "${_FILE2}"
for m in "${_FILE2}"
do
ls -lrt *${m}*.x
tar -cpf $m.tar /home/swap/output/outfiles/*${m}*
done
}
In case you can distinguish the files by their name, the solution can be so simple straightforward:
tar cf acqDou.tar acqDou*.out*
tar cf acq_cheat.tar acq_cheat*.out*
If there are several such modules, you can even apply a simple for loop to process them:
for m in acqDou acq_cheat
do
tar cf $m.tar ${m}*.out*
done
Make tars for all possible module types present in the directory - in one shot.
lets say you have modules named as.. acqDou, acqDumb, acqcheat, acqfunny - In module_XXXX.out format.
something which may looks like this, as shown below. So i got 4 modules and 500 files of each of them in the directory. They all are empty, but it doesn't matter.
>Wed Oct 05|01:15:13|gaurav#[STATION]:/root/ga/scripts/temp/tmp % ls -lrtha *.out|head -7 ; echo ; ls -lrtha |tail -8
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDou_0.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDumb_0.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqcheat_0.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqfunny_0.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDou_1.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDumb_1.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqcheat_1.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDumb_498.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqcheat_498.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqfunny_498.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDou_499.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDumb_499.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqcheat_499.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqfunny_499.out
drwxrwxr-x. 2 gaurav gaurav 64K Oct 5 01:14 .
>Wed Oct 05|01:15:30|gaurav#[STATION]:/root/ga/scripts/temp/tmp %
In order to achieve the objective:
we can list the files, strip off the numbers (using sed), sort and unique the list (which gives the possible present module names in the directory) and then feed it to a while loop, to read module names one by one .. and then generate our output tarfile, module name wise and having files of that module only.
Here is the command:
ls *.out |sed 's/[0-9]//g'|sort|uniq|sed 's/_.out//g'|while read module
do
tar -cvf ${module}.tar ${module}*
done
We get something like this, once we finish.
>Wed Oct 05|01:16:43|gaurav#[STATION]:/root/ga/scripts/temp/tmp % ls -lrth *.tar
-rw-rw-r--. 1 gaurav gaurav 260K Oct 5 01:16 acqcheat.tar
-rw-rw-r--. 1 gaurav gaurav 260K Oct 5 01:16 acqDou.tar
-rw-rw-r--. 1 gaurav gaurav 260K Oct 5 01:16 acqDumb.tar
-rw-rw-r--. 1 gaurav gaurav 260K Oct 5 01:16 acqfunny.tar
>Wed Oct 05|01:16:48|gaurav#[STATION]:/root/ga/scripts/temp/tmp %
4 tar files have been created, with their module names. And it contains its respective module files. Let us see by using tar -tvf command. I will run it on all tar files and just take 2 lines as output, per file.
>Wed Oct 05|01:18:29|gaurav#[STATION]:/root/ga/scripts/temp/tmp % ls *.tar|while read file
do
echo ;echo "Looking inside file: $file"
tar -tvf $file|head -2
done
Looking inside file: acqcheat.tar
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqcheat_0.out
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqcheat_100.out
Looking inside file: acqDou.tar
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqDou_0.out
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqDou_100.out
Looking inside file: acqDumb.tar
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqDumb_0.out
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqDumb_100.out
Looking inside file: acqfunny.tar
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqfunny_0.out
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqfunny_100.out
>Wed Oct 05|01:18:42|gaurav#[STATION]:/root/ga/scripts/temp/tmp %
We do really have 500 files in each tar file. Let us just confirm that too.
>Wed Oct 05|01:18:42|gaurav#[STATION]:/root/ga/scripts/temp/tmp % ls *.tar|while read file
do
echo ; echo "File count in tar file: $file"
tar -tvf $file|wc -l
done
File count in tar file: acqcheat.tar
500
File count in tar file: acqDou.tar
500
File count in tar file: acqDumb.tar
500
File count in tar file: acqfunny.tar
500
>Wed Oct 05|01:20:03|gaurav#[STATION]:/root/ga/scripts/temp/tmp %
Cheers,
Gaurav

Bash at job not running - Array

I have built a bash script that runs fine when executed from the command line but does not work when run as batch job (with at). First I thought because of the environment but when debugging I think there is a problem with arrays I need to create. When run from command line log is created and its content is what I expected but when run with at any log is created. Any idea for what is causing this issue?
A short script with the piece of code I suppose it is not running is below
#!/bin/bash
fsol=`date +%Y%m%d`
for dia
in 0 1 2
do
var=$(date -d "$fsol +$dia days" +'%Y-%m-%d')
orto=`awk -v j=$var 'BEGIN { FS=","} $2 == j { print $3}' hora-sol.dat`
h_orto=${orto:0:2}
m_orto=${orto:2:2}
a_orto+=($h_orto $m_orto)
echo "dia $dia" $var $h_orto $m_orto >> log1.txt
done
echo ${a_orto[#]} >> log2.txt
Data in hora-sol.dat
32,2016-02-01,0711,1216,1722,10.1885659530428
33,2016-02-02,0710,1216,1723,10.2235441870822
34,2016-02-03,0709,1216,1724,10.2589836910036
35,2016-02-04,0708,1216,1725,10.2948670333624
36,2016-02-05,0707,1216,1727,10.3311771153741
37,2016-02-06,0706,1217,1728,10.3678971831004
38,2016-02-07,0705,1217,1729,10.4050108377139
39,2016-02-08,0704,1217,1730,10.4425020444393
40,2016-02-09,0703,1217,1731,10.4803551390436
41,2016-02-10,0701,1217,1733,10.5185548339287
42,2016-02-11,0700,1217,1734,10.5570862213108
43,2016-02-12,0659,1217,1735,10.5959347763989
44,2016-02-13,0658,1217,1736,10.6350863580571
45,2016-02-14,0657,1217,1737,10.6745272092687
46,2016-02-15,0655,1217,1738,10.7142439549499
47,2016-02-16,0654,1217,1740,10.7542236006922
48,2016-02-17,0653,1217,1741,10.7944535282585
49,2016-02-18,0652,1216,1742,10.8349214920733
50,2016-02-19,0650,1216,1743,10.8756156133281
51,2016-02-20,0649,1216,1744,10.9165243743526
52,2016-02-21,0648,1216,1745,10.9576366115941
53,2016-02-22,0646,1216,1746,10.9989415078031
54,2016-02-23,0645,1216,1747,11.0404285846154
55,2016-02-24,0644,1216,1749,11.0820876932144
56,2016-02-25,0642,1216,1750,11.123909005324
57,2016-02-26,0641,1215,1751,11.1658830035395
58,2016-02-27,0639,1215,1752,11.2080004711946
59,2016-02-28,0638,1215,1753,11.2502524821626
60,2016-02-29,0636,1215,1754,11.2926303895977
Running manually, it generated:
# cat log.txt
dia 0 2016-02-12 0659 1217 1735
dia 1 2016-02-13 0658 1217 1736
dia 2 2016-02-14 0657 1217 1737
06
59
06
58
06
57
Scheduling with at:
# echo "/tmp/horasol/script.sh" | at now +1 minute
warning: commands will be executed using /bin/sh
job 1 at Fri Feb 12 12:11:00 2016
It generated exactly the same:
# cat log.txt
dia 0 2016-02-12 0659 1217 1735
dia 1 2016-02-13 0658 1217 1736
dia 2 2016-02-14 0657 1217 1737
06
59
06
58
06
57
Note that warninig informing that 'at' uses /bin/sh:
warning: commands will be executed using /bin/sh
Tell us how you conclude that "does not work when run as batch job (with at)"
Tell us more about your "when debugging" moment.
Perhaps I'm reproducing here using a different proccess as you. And due to this difference it works for me.

Daemon to RSYNC data fails after few successfull attempts

I have a Daemon coded in C to copy a file from local to remote using RSYNC and update the same file after every 5 second.
Everything works fine but when the while loop has executed for say 10 to 15 times RSYNC fails.
Here is the segment of code:
#define SHELLSCRIPT "\
#! /bin/bash \n\
rsync -azh /my_daemon_test -e 'ssh -p 2222' username#domainname.com:~ \n\
"
syslog(LOG_NOTICE, "Successfully started daemon\n");
while(1) {
syslog(LOG_NOTICE, "daemon in while loop : %d\n", flag);
sys_ret = system(SHELLSCRIPT);
syslog(LOG_NOTICE, "RSYNC Executed %d\n", sys_ret);
sleep(5);
flag++;
}
Log
Dec 18 00:07:08 localhost ./mydaemon[26384]: daemon in while loop : 10
Dec 18 00:07:09 localhost ./mydaemon[26384]: RSYNC Executed 0
Dec 18 00:07:14 localhost ./mydaemon[26384]: daemon in while loop : 11
Dec 18 00:07:15 localhost ./mydaemon[26384]: RSYNC Executed 0 <==== Success
Dec 18 00:07:20 localhost ./mydaemon[26384]: daemon in while loop : 12
Dec 18 00:07:20 localhost ./mydaemon[26384]: RSYNC Executed 65280 <==== Failed
Dec 18 00:07:25 localhost ./mydaemon[26384]: daemon in while loop : 13
Dec 18 00:07:26 localhost ./mydaemon[26384]: RSYNC Executed 65280
Need help to maintain the consistency.
Thanks

Resources