I am trying to create files from an array called columnHeaders[]
Within the array, I have test values that are currently:
id, source, EN, EN-GB, French-FR, French-DE
When I run my code I get
EN
EN.xml
EN-GB
EN-GB.xml
French-FR
French-FR.xml
French-DE
.xmlch-DE
NOTE that the FRENCH-DE filename gets morphed into .xmlch-DE
Why is this? I can't for the life of me figure out whey only that file ends up looking like this. It's driving me crazy!
Thanks for any help.
below is the snippet of my code that is causing me problems:
# take all the languages and either find the files or create new files for them. The language options
# should be stored in the columnHeader array in positions 3 - n
# cycle through all the output languages (so exclude "id, source' inputs)
# in my example, numWordsInLine is 6
c=2
while [ $c -lt $numWordsInLine ]; do
OUTPUT_LANG="${columnHeaders[$c]}"
echo "$OUTPUT_LANG"
OUTPUT_FILE="$OUTPUT_LANG".xml
# HERE'S WHERE YOU CAN SEE OUTPUT_FILE IS WRONG FOR FRENCH_DE
echo "$OUTPUT_FILE"
OUTPUT_BAK="$OUTPUT_LANG".bak
TMP_FILE="~tmp.xml"
if [ -f "$OUTPUT_BAK" ]; then
rm "$OUTPUT_BAK"
fi
# make a backup of the original language.xml file in case of program error or interruption
if [ -f "$OUTPUT_FILE" ]; then
mv "$OUTPUT_FILE" "$OUTPUT_BAK"
fi
if [ -f "$TMP_FILE" ]; then
rm "$TMP_FILE"
fi
c=$(expr $c + 1)
done
I'm betting that you are reading that line of data from a file with DOS newlines.
I'm also betting that the contents of the variable are "fine" but include a trailing carriage return.
Try printf %q\\n "$OUTPUT_FILE" or echo "$OUTPUT_FILE" | cat -v to see.
Then use something like dos2unix on the file to convert it.
Extra (unrelated) comments:
There's also no reason to use expr. ((c++)) will do what you want.
You could even turn the loop itself into for ((c=2;c < $numWordsInLine; c++)); do if you wanted to.
$numWordsInLine is also unnecessary if $columnHeaders is already split into the right "words" since you can use ${#columnHeaders} to get the length.
Related
I know this question has been asked in different manners, and I've referred to some answers on here to get to where I am now.
I'm trying to create a script to essentially watch a folder and virus scan files once they're not being written to.
The files/strings I need it to handle will sometimes contain spaces and sometimes not, as well sometimes special characters. At the moment it will actually work only on files with spaces in the name (as in actually scan and move them), but not for files without spaces. Also after each file (spaces or not) the while loop breaks thus stopping the script with the following output;
./howscan.sh: line 29: snmp.conf: syntax error: invalid arithmetic operator (error token is ".conf")
./howscan.sh: line 34: snmp.conf: syntax error: invalid arithmetic operator (error token is ".conf")
I had it working to handle file names without any spaces, but since I introduced the "${files[$i]}" method to use the array elements it only works on files with spaces and outputs the above error.
Feel free to omit the sepscan part of this, as I'm sure if I can get it working with the other tasks it'll work for that too (just wanted to show the full script for a complete understanding).
Current Script:
#!/bin/bash
set -x
workingpath='/mnt/Incoming/ToScan/'
outputpath='/mnt/Incoming/Scanned/'
logfile='/var/log/howscan.log'
faildir='/mnt/Incoming/ScanFailed/'
sepscan='/opt/Symantec/symantec_antivirus/sav manualscan -c'
# Change to working directory
cd $workingpath
# Exclude files with a given extension, in this case .aspx, and declare the remaining files as the array "files"
shopt -s extglob nullglob
# Loop the below - ie it's a watch folder
while true
do
# Exclude files with .aspx in the file name
files=( !(*.aspx) )
# If the array files is not empty then...
if [ ${#files[#]} -ne 0 ]; then
for i in ${files[*]}
# For every item in the array files process them as follows
# Declare any variables you wish to happen on files here, not globally
# Check each file to see if it's in use using fuser, do nothing and log it if its still in use, or process it and log the results
do
fileopen=`fuser "${files[$i]}" | wc -c`
# Here for 'fileopen' we are checking if the file is being writen to.
if [ $fileopen -ne 0 ]; then
echo `date` "${files[$i]}" is being used so has been ignored >> $logfile
else
echo `date` File "${files[$i]}" not being used or accessed >> $logfile
sepscan=`$sepscan "${files[$i]}" | wc -c`
if [ $sepscan = 0 ]; then
mv "${files[$i]}" $outputpath
echo `date` "${files[$i]}" has been virus scanned and moved to $outputpath >> $logfile
else
echo `date` "${files[$i]}" has been moved to $faildir as a virus or error was detected >> $logfile
fi
fi
done
fi
echo `date` 'No more files to process. Waiting 60 seconds...' >> $logfile
sleep 60
done
Let me know if I can provide anything else to help clarify my issue.
Update:
There is a file in the /mnt/Incoming/ToScan/ directory called snmp.conf by the way.
for i in ${files[*]}
should be
for i in ${!files[*]}
# or
for i in ${!files[#]}
${files[*]} expands to the contents of the array and undergoes word splitting. The above syntax expands to a list of indices of the array.
You might also need to double quote the variables, e.g.
if [ "$fileopen" -ne 0 ]; then
I have a JSON file which holds data like this: "path/to/git/directory/location": "path/to/local/location". A minimum example of the file might be this:
{
"${HOME}/dotfiles/.bashrc": "${HOME}/.bashrc",
"${HOME}/dotfiles/.atom/": "${HOME}/.atom/"
}
I have a script that systematically reads the above JSON (called locations.json) and creates an array, and then prints elements of the array that are directories. MWE:
#!/usr/bin/env bash
unset sysarray
declare -A sysarray
while IFS=: read -r field data
do
sysarray["${field}"]="${data}"
done <<< $(sed '/^[{}]$/d;s/\s*"/"/g;s/,$//' locations.json)
for file in "${sysarray[#]}"
do
if [ -d "${file}" ]
then
echo "${file}"
fi
done
However, this does not print the directory (i.e., ${HOME}/.atom).
I don't understand why this is happening, because
I have tried creating an array manually (i.e., not from a JSON) and checking if its elements are directories, and that works fine.
I have tried echoing each element in the array into a temporary file and reading each line in the file to see if it was a product of how the information was stored in the array, but no luck.
I have tried adding | tr -d "[:blank:]" | tr -d '\"' after using sed on the JSON (to see if it was a product of unintended whitespace or quotes), but no luck.
I have tried simply running [ -d "${HOME}/.atom/" ] && echo '.atom is a directory', and that works (so indeed it is a directory). I'm unsure what might be causing this.
Help on this would be great!
You could use a tool to process json files properly, which will deal with any valid json.
#!/usr/bin/env bash
unset sysarray
declare -A sysarray
while IFS="=" read -r field data
do
sysarray["${field}"]=$(eval echo "${data}")
done <<< $(jq -r 'keys[] as $k | "\($k)=\(.[$k])"' locations.json)
for file in "${sysarray[#]}"
do
if [ -d "${file}" ]
then
echo "${file}"
fi
done
Another problem is that, once the extra quote signs are properly processed, we have a literal ${HOME} that is not expanded. The only solution I came is using eval to force the expansion. It is not the nicest way, but right now I cannot find a better solution.
So I have written a bash script (named music.sh) for a Raspberry Pi to perform the following functions:
When executed, look into one single directory (Music folder) and select a random folder to look into. (Note: none of these folders here have subdirectories)
Once a folder within "Music" has been selected, then play all mp3 files IN ORDER until the last mp3 file has been reached
At this point, the script would go back to the folders in the "Music" directory and select another random folder
Then it would again play all mp3 files in that folder in order
Loop indefinitely until input from user
I have this code which does all of the above EXCEPT for the following items:
I would like to NOT play any other "album" that has been played before
Once all albums played once, then shutdown the system
Here is my code so far that is working (WITH duplicates allowed):
#!/bin/bash
folderarray=($(ls -d /home/alphekka/Music/*/))
for i in "${folderarray[#]}";
do
folderitems=(${folderarray[RANDOM % ${#folderarray[#]}]})
for j in "${folderitems[#]}";
do
echo `ls $j`
cvlc --play-and-exit "${j[#]}"
done
done
exit 0
Please note that there isn't a single folder or file that has a space in the name. If there is a space, then I face some issues with this code working.
Anyways, I'm getting close, but I'm not quite there with the entire functionality I'm looking for. Any help would be greatly appreciated! Thank you kindly! :)
Use an associative array as a set. Note that this will work for all valid folder and file names.
#!/bin/bash
declare -A folderarray
# Each folder name is a key mapped to an empty string
for d in /home/alphekka/Music/*/; do
folderarray["$d"]=
done
while [[ "${!folderarray[*]}" ]]; do
# Get a list of the remaining folder names
foldernames=( "${!folderarray[#]}" )
# Pick a folder at random
folder=${foldernames[RANDOM%${#foldernames[#]}]}
# Remove the folder from the set
# Must use single quotes; see below
unset folderarray['$folder']
for j in "$folder"/*; do
cvlc --play-and-exit "$j"
done
done
Dealing with keys that contain spaces (and possibly other special characters) is tricky. The quotes shown in the call to unset above are not syntactic quotes in the usual sense. They do not prevent $folder from being expanded, but they do appear to be used by unset itself to quote the resulting string.
Here's another solution: randomize the list of directories first, save the result in an array and then play (my script just prints) the files from each element of the array
MUSIC=/home/alphekka/Music
OLDIFS=$IFS
IFS=$'\n'
folderarray=($(ls -d $MUSIC/*/|while read line; do echo $RANDOM $line; done| sort -n | cut -f2- -d' '))
for folder in ${folderarray[*]};
do
printf "Folder: %s\n" $folder
fileArray=($(find $folder -type f))
for j in ${fileArray[#]};
do
printf "play %s\n" $j
done
done
For the random shuffling I used this answer.
One liner solution with mpv, rl (randomlines), xargs, find:
find /home/alphekka/Music/ -maxdepth 1 -type d -print0 | rl -d \0 | xargs -0 -l1 mpv
I've been trying to figure this one out for a while. I've read through multiple threads, and feel like I'm close, but the script just isn't coming together.
Scenario:
I have a media server and thousands of movie files. Each movie file has various accessory files such as the Cover artwork, Database info, Fanart, and trailer. While everything in the directory has it's coverart and database info, some files may or may not have their respective fanart or trailer. For these files I'm trying to get this script working which will create an empty "dummy" file in place of the file that should be there. Then when I actually have the time I can go back and search out just the dummy files and work to fill in the gaps where I can.
Here is what I have so far.
#!/bin/bash
find . -type f -print0 | while read -d $'\0' movie ;
do
echo $movie
moviename=${movie%\.*} #remove the extension from the string
moviename1=`echo $moviename | sed 's/\ /\\ /'` #add escaped spaces to the string
echo $moviename1 #echo the string (for debugging)
if [ ! -f $moviename-fanart* ]; #because the fanart could be .jpg, or .png, etc
then
echo "Creating $moviename-fanart.dummy"
touch "$moviename-fanart.dummy"
fi
if [ ! -f $moviename-trailer* ]; #because tralers could be .mp4, .mov, .mkv, .avi, etc
then
echo "Creating $moviename-trailer.dummy"
touch "$moviename-trailer.dummy"
fi
done
This should be pretty simple, but I think that I'm not getting the proper formating for the input string going into the test operators.
Any help would be greatly appreciated.
Thanks
Line-by-line analysis:
find . -type f -print0 | while read -d $'\0' movie; do
OK, but with bash4 you can just use shopt -s globstar to operate recursively on a directory.
moviename=${movie%\.*} #remove the extension from the string
You don't need the backslash.
moviename1=`echo $moviename | sed 's/\ /\\ /'` #add escaped spaces to the string
This line is suspect because if you quote the name, escaped spaces become doubly-escaped. You're confusing the value of the string with the representation you see of it.
if [ ! -f $moviename-fanart* ]; then #because the fanart could be .jpg, or .png, etc
Quote the string or use bash's [[ test keyword. It's a little dangerous to expand a glob inside the test expression because if it matches multiple results you'll get an error. That said, if you're sure there can be only one you can quote up to the glob. "$moviename-fanart"*.
touch "$moviename-fanart.dummy"
Here, you quote it. So essentially you're dealing with a different name now.
fi
if [ ! -f $moviename-trailer* ]; then #because tralers could be .mp4, .mov, .mkv, .avi, etc
echo "Creating $moviename-trailer.dummy"
touch "$moviename-trailer.dummy"
fi
Same thing.
done
The runtime arguments are as follows: $1 is the path to the file containing the list of files
$2 is the path to the directory containing the files
What I want to do is check that each file listed in $1 exists in the $2 directory
I'm thinking something like:
for f in 'cat $1'
do
if (FILEEXISTSIN$2DIRECTORY)
then echo '$f exists in $2'
else echo '$f is missing in $2' sleep 5 exit
fi
done
As you can see, I want it so that if any of the files listed in $1 don't exist in $2 directory, the script states this then closes. The only part I can't get my head around is the (FILEEXISTSIN$2DIRECTORY) part. I know that you can do [ -e $f ], but I don't know how you can make sure its checking that it exists in the $2 directory.
Edit: Thinking further upon this, perhaps I could use nested for loops?
If your specified input file contains a newline-separated list of files to check, then the following solution (using a while read loop) is robust enough to handle file names with spaces properly.
Generally, you should never make use of a loop of the form for i in $(command), and instead opt for a while loop. See http://mywiki.wooledge.org/DontReadLinesWithFor for more details.
while read -r file; do
if [[ -e "$2/$file" ]]; then
echo "$f exists in $2"
else
echo "$f does not exist in $2"
sleep 5
exit 1
fi
done < "$1"
Since you're dealing with a list of file names without spaces in the names (because the $(cat $1) notation will split things up like that), it is relatively straight forward:
for file in $(cat $1)
do
if [ -e "$2/$file" ]
then echo "$file exists in $2"
else echo "$file is missing in $2"; sleep 5; exit 1
fi
done
Basically, use the built-in string concatenation facilities to build the full path to the file, and use the test or [ operator to check the files existence.
The complexities arise if you have to deal with arbitrary file names, especially if one of the arbitrary characters in an arbitrary file name can be the newline character. Suffice to say, they complicate the issue sufficiently that I won't deal with it unless you say you need it dealt with, and even then, I'll negotiate on whether newlines in names need to be handled. The double-quoted variable expansion is a key part of the strategy for dealing with it. The other part of the problem is how to get the file names accurately into a variable.