Hi I have written a c program that takes 3 integers as input:
./myprogram 1 2 3
and I am aiming to pipe data from a csv file into the input of the c program. I grab each line from the c program using:
for i in $(seq 1 `wc -l "test.csv" | awk '{print $1}'`); do sed -n $i'p' "test.csv"; done;
and then would like to pipe the output of this into my c program. I have tried doing:
for i in $(seq 1 `wc -l "test.csv" | awk '{print $1}'`); do sed -n $i'p' "test.csv"; done; | ./myprogram
however I get:
Line
bash: syntax error near unexpected token `|'
how do I pipe the output into my c program?
Thanks
It helps when you really try to understand error messages the shell gives you:
Line
bash: syntax error near unexpected token `|'
If you think about it, when you chain commands together in a pipeline, there is never a ; before a |, for example:
ls | wc -l
# and not: ls; | wc -l
Whatever comes after a ; is like an independent new command, as if you typed it on a completely new, clear command line. If you type | hello on a clear command line, you'll get the exact same error, because that's the exact same situation as ; | ... in your script, for example:
$ | hello
-bash: syntax error near unexpected token `|'
Others already answered this, but I also wanted to urge you to make other improvements in your script:
Always use $() instead of backticks, for example:
for i in $(seq 1 $(wc -l "test.csv" | awk '{print $1}')); ...
You didn't need the awk there, this would work just as well:
for i in $(seq 1 $(wc -l "test.csv")); ...
You could reduce your entire script to simply this, for the same effect:
./myprogram < test.csv
In the shell, it doesn't like an explicit line termination followed by a pipe (|). The pipe already delimits the commands. So you want:
for i in $(seq 1 `wc -l "test.csv" | awk '{print $1}'`); do sed -n $i'p' "test.csv"; done | ./myprogram
Related
Why isn't this bash array populating? I believe I've done them like this in the past. Echoing ${#XECOMMAND[#]} shows no data..
DIR=$1
TEMPFILE=/tmp/dir.tmp
ls -l $DIR | tail -n +2 | sed 's/\s\+/ /g' | cut -d" " -f5,9 > $TEMPFILE
i=0
cat $TEMPFILE | while read line ;do
if [[ $(echo $line | cut -d" " -f1) == 0 ]]; then
XECOMMAND[$i]="$(echo "$line" | cut -d" " -f2)"
(( i++ ))
fi
done
When you run the while loop like
somecommand | while read ...
then the while loop is executed in sub-shell, i.e. a different process than the main script. Thus, all variable assignments that happen in the loop, will not be reflected in the main process. The workaround is to use input redirection and/or command substitution, so that the loop executes in the current process. For example if you want to read from a file you do
while read ....
do
# do stuff
done < "$filename"
or if you wan't the output of a process you can do
while read ....
do
# do stuff
done < <(some command)
Finally, in bash 4.2 and above, you can set shopt -s lastpipe, which causes the last command in the pipeline to be executed in the current process.
I think you're trying to construct an array consisting of the names of all zero-length files and directories in $DIR. If so, you can do it like this:
mapfile -t ZERO_LENGTH < <(find "$DIR" -maxdepth 1 -size 0)
(Add -type f to the find command if you're only interested in regular files.)
This sort of solution is almost always better than trying to parse ls output.
The use of process substitution (< <(...)) rather than piping (... |) is important, because it means that the shell variable will be set in the current shell, not in an ephimeral subshell.
I am trying to understand the internal working of pipes in C. I noticed if I run
int main() {
system("ls | grep d | wc");
}
Output:
3 3 53
But on running the same command with bash I get
3 3 104
Output of ls | grep d
question_1.pdf
question_2.pdf
question_2_dataset.txt
Can someone explain the cause of this discrepancy?
The same thing occurs if I use pipe via pipe() call in C.
Actually I figured out problem wasn't with ls but "grep --color=always d" which is alias of grep in my bash. The colored characters have extra length which increase the length of output.
Check out what your 'ls' command at the bash is! Try:
type ls
You probably will find that ls is an alias of some kind.
Check your bash-test again with
/bin/ls | grep d | wc
If you then get th esame result as in your C-Code you will know what went wrong.
ls is often an alias in an interactive shell.
For example, in my bash session if I do type ls I get
ls is aliased to `ls -t --group-directories-first -I .pyc -I __pycache__ -I .git --color=auto -xF'
(The alias usually comes from $HOME/.bashrc or /etc/bash.bashrc).
Now if you do:
sh -c 'ls | grep d | wc'
(or command ls| command grep d | command wc ) you should get absolutely the same result as with compiling
int main() { system("ls | grep d | wc"); }
and running it in the same directory.
I have a (let's call it original) script to parse a file (~1000 lines) line by line, generate arguments to execute a C++ program.
#!/bin/bash
i = 0
while IFS='' read -r line || [[ -n "$line" ]]; do
a="$line" | cut -c1-2
b="$line" | cut -c3-4
c="$line" | cut -c5-6
d="$line" | cut -c7-8
e="$line" | cut -c9-10
f="$line" | cut -c11-12
g="$line" | cut -c13-14
h="$line" | cut -c15-16
i="$line" | cut -c17-18
j="$line" | cut -c19-20
k="$line" | cut -c21-22
l="$line" | cut -c23-24
m="$line" | cut -c25-26
n="$line" | cut -c27-28
o="$line" | cut -c29-30
p="$line" | cut -c31-32
./a.out "$a" "$b" "$c" "$d" "$e" "$f" "$g" "$h" "$i" "$j" "$k" "$l" "$m" "$n" "$o" "$p" > $(echo some-folder/output_${i}.txt)
done < test_10.txt
I want to schedule this job in a batch, so that each run is queued and ran on separate cores.
I checked the PBS and qsub writing styles. I could write a PBS file (simple one, without all options for now. Lets call it callPBS.PBS):
#!/bin/bash
cd $PBS_O_WORKDIR
qsub ./a.out -F "my arguments"
exit 0
I can call this file instead of ./a.out -------- in original script. BUT how do I pass "my arguments"? Problem is they are not fixed.
Secondly I know qsub takes -o as an option for output file. But I want my output file name to be changed. I can pass that as an argument again, but how?
Can I do this in my original script:
callPBS.pbs > $(echo some-folder/output_${i}.txt)
I am sorry if I am missing something here. I am trying to use all that I know!
I am very new to Unix shell script and trying to get some knowledge in shell scripting. Please check my requirement and my approach.
I have a input file having data
ABC = A:3 E:3 PS:6
PQR = B:5 S:5 AS:2 N:2
I am trying to parse the data and get the result as
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
The values can be added horizontally and vertically so I am trying to use an array. I am trying something like this:
myarr=(main.conf | awk -F"=" 'NR!=1 {print $1}'))
echo ${myarr[1]}
# Or loop through every element in the array
for i in "${myarr[#]}"
do
:
echo $i
done
or
awk -F"=" 'NR!=1 {
print $1"\n"
STR=$2
IFS=':' read -r -a array <<< "$STR"
for i in "${!array[#]}"
do
echo "$i=>${array[i]}"
done
}' main.conf
But when I add this code to a .sh file and try to run it, I get syntax errors as
$ awk -F"=" 'NR!=1 {
> print $1"\n"
> STR=$2
> FS= read -r -a array <<< "$STR"
> for i in "${!array[#]}"
> do
> echo "$i=>${array[i]}"
> done
>
> }' main.conf
awk: cmd. line:4: FS= read -r -a array <<< "$STR"
awk: cmd. line:4: ^ syntax error
awk: cmd. line:5: for i in "${!array[#]}"
awk: cmd. line:5: ^ syntax error
awk: cmd. line:8: done
awk: cmd. line:8: ^ syntax error
How can I complete the above expectations?
This is the awk code to do what you want:
$ cat tst.awk
BEGIN { FS="[ =:]+"; OFS="=" }
{
print $1
for (i=2;i<NF;i+=2) {
print $i, $(i+1)
}
print ""
}
and this is the shell script (yes, all a shell script does to manipulate text is call awk):
$ awk -f tst.awk file
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
A UNIX shell is an environment from which to call UNIX tools (find, sort, sed, grep, awk, tr, cut, etc.). It has its own language for manipulating (e.g. creating/destroying) files and processes and sequencing calls to tools but it is NOT intended to be used to manipulate text. The guys who invented shell also invented awk for shell to call to manipulate text.
Read https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice and the book Effective Awk Programming, 4th Edition, by Arnold Robbins.
First off, a command that does what you want:
$ sed 's/ = /\n/;y/: /=\n/' main.conf
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
This replaces, on each line, the first (and only) occurrence of = with a newline (the s command), then turns all : into = and all spaces into newlines (the y command). Notice that
this works only because there is a space at the end of the first line (otherwise it would be a bit more involved to get the empty line between the blocks) and
this works only with GNU sed because it substitutes newlines; see this fantastic answer for all the details and how to get it to work with BSD sed.
As for what you tried, there is almost too much wrong with it to try and fix it piece by piece: from the wild mixing of awk and Bash to syntax errors all over the place. I recommend you read good tutorials for both, for example:
The BashGuide
Effective AWK Programming
A Bash solution
Here is a way to solve the same in Bash; I didn't use any arrays.
#!/bin/bash
# Read line by line into the 'line' variable. Setting 'IFS' to the empty string
# preserves leading and trailing whitespace; '-r' prevents interpretation of
# backslash escapes
while IFS= read -r line; do
# Three parameter expansions:
# Replace ' = ' by newline (escape backslash)
line="${line/ = /\\n}"
# Replace ':' by '='
line="${line//:/=}"
# Replace spaces by newlines (escape backslash)
line="${line// /\\n}"
# Print the modified input line; '%b' expands backslash escapes
printf "%b" "$line"
done < "$1"
Output:
$ ./SO.sh main.conf
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
Assume that the Pid of active processes on my machines are 1000 and 2000.
I am trying to make an array in Linux such that
The command echo ${Pid_Current[0]} gives 1000 in output
The command echo ${Pid_Current[1]} gives 2000 in output
Here is my code:
declare -a Pid_Current
Pid_Current=$(ps -aF | tail -n +2 | awk '{print $2}')
However, instead of the desired output I explained above, I receive the following output:
echo ${Pid_Current[0]} gives 1000 2000 in output
echo ${Pid_Current[1]} gives nothing in output
Would you please advise me what part of my code is incorrect?
In bash array assignment is done by enclosing the expression in parenthesis, so to use array assignment you need to write:
Pid_Current=($(ps -aF | tail -n +2 | awk '{print $2}'))
Without parenthesis the result of the expression is assigned to Pid_Current[0]