Reading keys from a XML file into a shell array - arrays

Im trying to parse an xml file and get all tag values and fonts values and add them to an associative arrays. My issue is that the array doesn't seem to be having the values assigned to it properly
#!/bin/bash
GAME_NAME="."
LOCALIZATION_DIR="$GAME_NAME/assets/data/localization"
INDEX=0
OUTPUT_KEYS=()
# parse english xml for tags and font names first
for str in $(echo "cat //strings/string/#key" | xmllint --shell "$LOCALIZATION_DIR/en.xml")
do
echo "$str"
echo "--"
OUTPUT_KEYS[$index]="$str"
((INDEX++))
done
echo ${OUTPUT_KEYS[0]}
The last echo just echos the end of the tag > A little confused on how arrays should be working in shell or if there is a better way to approach this.
My XML looks like this.
<?xml version="1.0" encoding="UTF-8" ?>
<strings version="5.6051.4-en">
<!--<StarLineUI>-->
<!-- Menu -->
<string key="betProper" value="Bet" fonts="uiAccountTitle" />
<string key="linesProper" value="Lines" fonts="uiAccountTitle" />
<string key="spinsProper" value="Spins" fonts="uiAccountTitle" />
<string key="bet" value="BET" fonts="uiMenuTitle, uiAccountTitle" />
<string key="line" value="LINE" fonts="uiMessage" />
</strings>
I'm trying to build a solution that works with GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)

If you have bash 4.0 or newer:
readarray -t output_keys \
< <(xmlstarlet sel -t -m '//strings/string[#key]' -v #key -n <in.xml)
echo "${output_keys[0]}"
Otherwise:
output_keys=( )
while IFS= read -r line; do
output_keys+=( "$line" )
done < <(xmlstarlet sel -t -m '//strings/string[#key]' -v #key -n <in.xml)
In either of these, the output of xmlstarlet is just the keys you're trying to extract, as in:
betProper
linesProper
spinsProper
bet
line
...and this can be iterated over as you'd expect:
for key in "${output_keys[#]}"; do
echo "Found key: $key"
done
If you don't have xmlstarlet, you can run the XSLT equivalent to the above command line; if you have a stylesheet print-strings.xslt with the following contents:
<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:exslt="http://exslt.org/common" version="1.0" extension-element-prefixes="exslt">
<xsl:output omit-xml-declaration="yes" indent="no"/>
<xsl:template match="/">
<xsl:for-each select="//strings/string[#key]">
<xsl:call-template name="value-of-template">
<xsl:with-param name="select" select="#key"/>
</xsl:call-template>
<xsl:value-of select="'
'"/>
</xsl:for-each>
</xsl:template>
<xsl:template name="value-of-template">
<xsl:param name="select"/>
<xsl:value-of select="$select"/>
<xsl:for-each select="exslt:node-set($select)[position()>1]">
<xsl:value-of select="'
'"/>
<xsl:value-of select="."/>
</xsl:for-each>
</xsl:template>
</xsl:stylesheet>
...then you can run:
xsltproc print-strings.xslt in.xml
...to get the same output as xmlstarlet sel -t -m '//strings/string[#key]' -v #key -n <in.xml.

Related

bash script to write in specific part of xml file

My problem is that this code below puts the result from the beginning of the file but I want to put it in a specific place.
#!/bin/bash -x
USER_ID=( User1 User2 User3 )
USER_CONF=/opt/test/config.xml
for i in "${USER_ID[#]}"; do
printf '<user><id>ID</id><name><%s/name></user>\n' "$i" >> "$USER_CONF"
done
What I get now in config.xml is:
<company="external">
<enabled>true</enabled>
<users="allowed">
USER_TO_INSERT_HERE
</users>
</company>
<user><id>ID</id><name><User1/name></user>
<user><id>ID</id><name><User2/name></user>
<user><id>ID</id><name><User3/name></user>
What I want to get after the script execution in config.xml is:
<company="external">
<enabled>true</enabled>
<users="allowed">
<user><id>ID</id><name><User1/name></user>
<user><id>ID</id><name><User2/name></user>
<user><id>ID</id><name><User3/name></user>
</users>
</company>
Do you know how can I record the values from the for and write them in a variable then to just sed that var in the code?
I know how to sed it but don't know how to record in the var the values or something like that?
First of all, <users="allowed"> in an invalid XML-node. This should probably be something like <users permission="allowed">.
Please use an XML parser like xidel to edit your 'config.xml'.
With "direct element constructors":
$ xidel -s config.xml -e '
x:replace-nodes(
//users,
<users permission="allowed">{
for $user in ("User1","User2","User3") return
<user><id>ID</id><name>{$user}</name></user>
}</users>
)
' --output-node-format=xml --output-node-indent
With "computed constructors":
$ xidel -s config.xml -e '
x:replace-nodes(
//users,
function($x){
element {$x/name()} {
$x/#*,
for $user in ("User1","User2","User3") return
element user {
element id {"ID"},
element name {$user}
}
}
}
)
' --output-node-format=xml --output-node-indent
Output:
<company="external">
<enabled>true</enabled>
<users permission="allowed">
<user>
<id>ID</id>
<name>User1</name>
</user>
<user>
<id>ID</id>
<name>User2</name>
</user>
<user>
<id>ID</id>
<name>User3</name>
</user>
</users>
</company="external">
Playground
Lots of great answers over at unix.stackexchange.com.
The canonical answer for this sort of case (NOTE: not for XML in general in all cases, but for a file where there's a TOKEN on a line on it's own to be replaced - which is exactly the case you have given) is -
a) Output the part of the file "up to" the line before the line with the token
b) Output the replacement
c) Output the rest of the file "from" the line after the line with the token
e.g. here's the simple sed variant (which is no where near as elegant as the r option to sed) -
sed -e '/USER_TO_INSERT_HERE/,$ d' source.xml
cat replacement.xml
sed -e '1,/USER_TO_INSERT_HERE/ d' source.xml

XMLStartlet is not updating salesforce package.xml file

I want to create Salesforce Dynamic package.xml file. I followed the process mentioned in this apexandbeyond.wordpress.com/2017/03/15/dynamic-package-xml-generation/ blog, it is neither updating not inserting data in xml file. I am trying following way but i doesn't work for me. please help me on this.
Below code is in loop i need update xml file dynamically. Here i am getting modified file dynamically.
git diff-tree --no-commit-id --name-only --diff-filter=ACMRTUXB -t -r $PREVRSA $LCOMMIT | \
while read -r CFILE; do
case "$CFILE"
in
.cls) TYPENAME="ApexClass";;
.component) TYPENAME="ApexComponent";;
.page) TYPENAME="ApexPage";;
*) TYPENAME="UNKNOWN TYPE";;
esac
if [[ "$TYPENAME" != "UNKNOWN TYPE" ]]
then
echo $TYPENAME "," $ENTITY #example:(ApexClass,TestClass1
#example: ApexClass,Testclass2
#example: ApexPage,TestPage) # getting values Dynamically
xmlstarlet edit -L --subnode "/Package/types[name=$TYPENAME]" --type elem -n members -v
"$ENTITY" testpackage.xml > edipackage.xml
fi
done
echo Cleaning up Package.xml
xmlstarlet ed -L -i /Package -t attr -n xmlns -v "http://soap.sforce.com/2006/04/metadata" testpackage.xml > edipackage.xml
Original testpackage.xml file:
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
<types>
<members>*</members>
<name>ApexClass</name>
</types>
<types>
<members>*</members>
<name>ApexComponent</name>
</types>
<types>
<members>*</members>
<name>ApexPage</name>
</types>
<version>48.0</version>
</Package>
Expected Output:
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
<types>
<members>TestClass1</members>
<members>TestClass2</members>
<name>ApexClass</name>
</types>
<types>
<members>TestPage</members>
<name>ApexPage</name>
</types>
<version>48.0</version>
</Package>
Please help me on this.
The namespace should be defined as follows:
xml sel -N x=http://soap.sforce.com/2006/04/metadata -t -m //x:name -v . -n salesforce.xml
which returns:
ApexClass
ApexComponent
ApexPage

How can I edit my code so that I can account for output that goes against my sed command

I am writing a code to put the species named matched from a remote NCBI BLAST database, and the file the matched name came from. I want to make my code more robust so that it can deal with files that do not get a match and that go against my current sed command
#!/bin/bash
for i in ./split.contigs.Parsed/*.csv ; do
sciname=$(head -1 $i | sed -E "s/([A-Z][a-z]+ [a-z]+) .+/\1/")
contigname=$(echo $i | sed -E "s/.fa.csv//" | sed -E
"s/\.\/split.contigs.Parsed\///")
echo "$sciname,$contigname"
done
Expected
Drosophila melanogaster,contig_66:1.0-213512.0_pilon
Drosophila melanogaster,contig_67:1.0-138917.0_pilon
Drosophila sechellia,contig_67:139347.0-186625.0_pilon
Drosophila melanogaster,contig_68:3768.0-4712.0_pilon
Actual
Drosophila ananassae,contig_393:1.0-13214.0_pilon
,contig_393:13217.0-13563.0_pilon
Drosophila sp. pallidosa-like-Wau w,contig_393:14835.0-18553.0_pilon
Apteryx australis,contig_393:19541.0-21771.0_pilon
,contig_393:21780.0-22772.0_pilon
Drosophila sp. pallidosa-like-Wau w,contig_393:22776.0-31442.0_pilon
Drosophila melanogaster,contig_394:1.0-89663.0_pilon
Simply skip the loop if $sciname is null. Put this one line after defining $sciname:
[[ -z $sciname ]] && continue

How to calculate md5-content header in unix shell?

I am looking for the unix command(s) to calculate md5-content header to be used with IBM Cloud Object Storage API for deletion of multiple objects. I tried echo “request body….” | md5 | base64, however API response is - `
The Content-MD5 you specified was an invalid.
Curl CMD:
curl \
-H "Content-Type: text/plain;charset=utf-8" \
-H "Content-MD5: 75ff06f81643655397a5911ddc195ce8" \
-H "Authorization: $AuthToken" \
"https://<cos-endpoint-name>/<bucket-name>?delete" \
-d 'xml body...'
Error Response:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
<Code>InvalidDigest</Code>
<Message>The Content-MD5 you specified was an invalid.</Message>
<Resource>/ghhsa-bucket-etl-dev/</Resource>
<RequestId>aed25243-22a1-477d-9ab8-b87780625a61</RequestId>
<httpStatusCode>400</httpStatusCode>
</Error>
Appreciate any pointers on this.
The md5 builtin is kinda weak, it's a bit more straightforward using openssl for encryption if possible. Using an example from the docs:
echo -n '<?xml version="1.0" encoding="UTF-8"?><Delete><Object><Key>pasture/cow-one</Key></Object><Object><Key>pasture/cow-two</Key></Object></Delete>' | openssl dgst -md5 -binary | openssl enc -base64
This returns /Gx4aOgplRXMRI2qXXqXiQ== which is what we'd expect.

List of files located in text file to be send as parameter to WinSCP in script mode

I've to create solution that is based on WinSCP and windows batch script. Below are tasks the script has to do:
Save list of files from remote directory, to file located on machine where batch script is run.
Run WinSCP in command mode (/command) using WinSCP script passed as parameter (/script=name_of_script_file.txt) and get files (get command) from previously generated list.
The most important is to get file list, save it and pass names of files located in created file to WinSCP to get them.
How to implement this?
There's no easy way to implement this using WinSCP scripting only. It's possible, see the very end of my answer, but it may not be an ideal solution.
Why do you do this in two steps? Why don't you directly download the directory?
winscp.com /command ^
"option batch abort" ^
"option confirm off" ^
"open scp://user:password#example.com/" ^
"get /path/*" ^
"exit"
If you really need the list (e.g. for some further processing), you can instead of getting a list of files in the directory, get a list of actually downloaded files.
Enable the XML logging, and get the list from the XML log.
winscp.com /xmllog=log.xml /command ^
....
You get a log like:
<?xml version="1.0" encoding="UTF-8"?>
<session xmlns="http://winscp.net/schema/session/1.0" name="user#host" start="2015-01-30T06:45:57.008Z">
<download>
<filename value="/path/file1.txt" />
<destination value="C:\path\file1.txt" />
<result success="true" />
</download>
<download>
<filename value="/path/file2.txt" />
<destination value="C:\path\file2.txt" />
<result success="true" />
</download>
</session>
If you need a plain-text list, you can use the XSLT to convert it (e.g. download.xslt):
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:winscp="http://winscp.net/schema/session/1.0">
<xsl:output method="text" encoding="UTF-8"/>
<xsl:strip-space elements="*"/>
<xsl:template match='winscp:download[winscp:result[#success="true"]]/winscp:filename'>
<xsl:value-of select="#value"/>
<xsl:text>
</xsl:text>
</xsl:template>
</xsl:stylesheet>
To process it, use any XSLT processor:
Microsoft msxsl.exe (deprecated, but available from Internet Archive)
msxsl.exe log.xml download.xslt
Libxml2 xsltproc.exe:
xsltproc.exe download.xslt log.xml
And you get:
/path/file1.txt
/path/file2.txt
See Transforming XML Log to Text Output Using XSLT Transformation.
To answer your actual question. If you really insist on getting a file list and downloading files according to it, I suggest you use the WinSCP .NET assembly from a PowerShell script.
Use the Session.ListDirectory to retrieve a contents of a remote directory.
See the Session.ListDirectory documentation and its example code.
For recursive listing, you can also use the Session.EnumerateRemoteFiles.
Iterate the results and call the Session.GetFiles for every file.
See the Session.GetFiles documentation and its example code.
If the simple scripting is your preference:
The only way to obtaining a plain text list of remote files reliably, is to use the XML logging to capture an output of the ls scripting command:
winscp.com /xmllog=log.xml /command ^
"option batch abort" ^
"option confirm off" ^
"open scp://user:password#example.com/" ^
"ls /path" ^
"exit"
You get a log like:
<?xml version="1.0" encoding="UTF-8"?>
<session xmlns="http://winscp.net/schema/session/1.0" name="user#host" start="2015-01-30T07:08:27.749Z">
<ls>
<destination value="/path" />
<files>
<file>
<filename value="." />
<type value="d" />
<modification value="2014-07-28T07:06:49.000Z" />
<permissions value="rwxr-sr-x" />
</file>
<file>
<filename value=".." />
<type value="d" />
<modification value="2015-01-23T12:22:44.000Z" />
<permissions value="rwxr-xr-x" />
</file>
<file>
<filename value="file1.txt" />
<type value="-" />
<size value="1306091" />
<modification value="2015-01-29T23:58:12.000Z" />
<permissions value="rw-rw-rw-" />
</file>
<file>
<filename value="file2.txt" />
<type value="-" />
<size value="88" />
<modification value="2007-11-17T22:40:43.000Z" />
<permissions value="rw-r--r--" />
</file>
</files>
<result success="true" />
</ls>
</session>
Again, use the XSLT to convert the XML log to a plain-text list of files:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:winscp="http://winscp.net/schema/session/1.0">
<xsl:output method="text" encoding="UTF-8"/>
<xsl:strip-space elements="*"/>
<xsl:template match='winscp:file[winscp:type/#value="-"]/winscp:filename'>
<xsl:value-of select="#value"/>
<xsl:text>
</xsl:text>
</xsl:template>
</xsl:stylesheet>
You get:
file1.txt
file2.txt
To get WinSCP script to download files according to a plain-text list, see scripting example Uploading a list of files. It's for an upload, but just replace the put with the get and reverse an order of arguments to turn it to a download.
Note that the /command parameter does not enter a "command" (scripting?) mode. It's for passing scripting commands on a command line (as used in my answer). You are using a script file (/script). There's no point adding an empty /command parameter to it.
See the documentation of WinSCP scripting command-line parameters.

Resources