Get xmllint xpath nodes to arrays - arrays

I have NMAP-output xml-file like this:
<host><status state="down" reason="no-response" reason_ttl="0"/>
<address addr="192.168.1.1" addrtype="ipv4"/>
</host>
<taskbegin task="SYN Stealth Scan" time="1457545799"/>
<taskend task="SYN Stealth Scan" time="1457545841" extrainfo="1600 total ports"/>
<host starttime="1457545794" endtime="1457545839"><status state="up" reason="echo-reply" reason_ttl="243"/>
<address addr="192.168.1.2" addrtype="ipv4"/>
<hostnames>
<hostname name="192-168-1-2.liibalaaba.com" type="PTR"/>
</hostnames>
<ports><extraports state="closed" count="100">
<extrareasons reason="resets" count="100"/>
</extraports>
</ports>
<times srtt="6454" rttvar="163" to="100000"/>
</host>
<host starttime="1457545794" endtime="1457545837"><status state="up" reason="echo-reply" reason_ttl="51"/>
<address addr="192.168.1.3" addrtype="ipv4"/>
<hostnames>
<hostname name="192-168-1-3.liibalaaba.com" type="PTR"/>
</hostnames>
<ports><extraports state="filtered" count="98">
<extrareasons reason="no-responses" count="98"/>
</extraports>
<port protocol="tcp" portid="80"></port>
<port protocol="tcp" portid="443"></port>
</ports>
<times srtt="6378" rttvar="191" to="100000"/>
</host>
<host starttime="1457545794" endtime="1457545841"><status state="up" reason="echo-reply" reason_ttl="115"/>
<address addr="192.168.1.4" addrtype="ipv4"/>
<hostnames>
<hostname name="192-168-1-4.liibalaaba.com" type="PTR"/>
</hostnames>
<ports><extraports state="filtered" count="97">
<extrareasons reason="no-responses" count="97"/>
</extraports>
<port protocol="tcp" portid="80"></port>
<port protocol="tcp" portid="81"></port>
<port protocol="tcp" portid="443"></port>
</ports>
<times srtt="6417" rttvar="113" to="100000"/>
</host>
How I can put only the hosts that have open ports to an array and then to readable format? (Do I even need an array?)
Something like this:
address:192.168.1.3 [0,0]
open ports:80 [0,1], 443 [0,2]
address:192.168.1.4 [1,0]
open ports:80[1,1], 81 [1,2], 443 [1,3]
Now i'm using xmllint xpath combo, but it just splits out every address that has open ports to same line:
`xmllint --xpath "//*[local-name()='host']/*[local-name()='address']/#addr | //*[local-name()='ports']/*[local-name()='port']/#portid" nmap-scan.xml > result.txt`
Thanks for help!

Related

How to pass 'as' argument to gdbus?

This is a part of xml.
<method name="Mount">
<annotation name="org.chromium.DBus.Method.Kind" value="simple" />
<arg name="path" type="s" direction="in" />
<arg name="filesystem_type" type="s" direction="in" />
<arg name="options" type="as" direction="in" />
And this is what I want to implement exactly. (captured method call by dbus-monitor)
method call time=1675906347.661521 sender=:1.31 -> destination=org.chromium.CrosDisks serial=4557 path=/org/chromium/CrosDisks; interface=org.chromium.CrosDisks; member=Mount
string "/sys/devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2:1.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb1"
string ""
array [
string "rw"
string "mountlabel=MY-STORAGE"
]
And This is what I tried by gdbus.
gdbus call --system --dest org.chromium.CrosDisks --object-path /org/chromium/CrosDisks --method org.chromium.CrosDisks.Mount "/sys/devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2:1.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb1" “” [“rw” "mountlabel=MY-STORAGE"]
The I got errors.
(gdbus call:12863): GLib-CRITICAL **: 23:37:56.454: g_variant_new_string: assertion 'g_utf8_validate (string, -1, NULL)' failed
[Invalid UTF-8] Error parsing parameter 2: expected value:
\xe2\x80
^
(gdbus call:12863): GLib-CRITICAL **: 23:37:56.455: g_error_free: assertion 'error != NULL' failed
I think my expression for the type="as" is wrong.
Can someone help me?
I found the answer.
"['rw', 'mountlabel=MY-STORAGE']"

How to keep whitespace formatting while reading a file into an array using perl?

I am trying to read a text file into an array of lines while keeping the whitespace formatting of each line. Once I read the entire file into an array, if I iterate through the array printing all of the lines to STDOUT, all of the whitespace formatting is lost, as well as some of the text (look at the first line). But, if I print the lines while reading the file, and printing the array using Data::Dumper, the whitespace formatting is still there.
Since I am trying to capture the output on STDOUT while iterating through the array, please tell me what I need to do to print out the properly whitespace formatted text.
The code and output are pretty much self explanatory, as can be seen below.
I dual boot between macOS Mojave 10.14.6 and Linux Mint 20 and have the same problem on both systems with different versions of perl, versions below.
Thanks in advance for any advice!!!
macOS:
macOS Mojave 10.14.6
Darwin MBPro.lan 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64
This is perl 5, version 18, subversion 4 (v5.18.4) built for darwin-thread-multi-2level
(with 2 registered patches, see perl -V for more detail)
linux:
Linux mintMBP 5.4.0-47-generic #51-Ubuntu SMP Fri Sep 4 19:50:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
This is perl 5, version 30, subversion 0 (v5.30.0) built for x86_64-linux-gnu-thread-multi
(with 46 registered patches, see perl -V for more detail)
Here is the code:
#!/usr/bin/perl
use Data::Dumper;
my $filename = "formatted.txt";
my #lines;
open(my $FH, "<", $filename)
or die "Can't open < $filename: $!";
print "\nText printed while reading file:\n\n";
while(<$FH>) {
chomp($_);
push(#lines, $_);
print $_ . "\n";
}
close($FH);
print "\nText printed while iterating the line array:\n\n";
while(<#lines>) {
print $_ . "\n";
}
print "\nText printed using Data::Dumper :\n\n";
print Dumper(#lines);
Here is the whitespace formatted input file:
<?xml version="1.0" encoding="UTF-8"?>
<dict>
<key>Definitions</key>
<dict>
<key>src/main.c</key>
<dict>
<key>Path</key>
<string>src/main.c</string>
<key>Group</key>
<array>
<string>src</string>
</array>
</dict>
</dict>
</dict>
Here is the output:
[MBPro:Development/tmp/perl_formatted_lines] justin% ./file_test.pl
Text printed while reading file:
<?xml version="1.0" encoding="UTF-8"?>
<dict>
<key>Definitions</key>
<dict>
<key>src/main.c</key>
<dict>
<key>Path</key>
<string>src/main.c</string>
<key>Group</key>
<array>
<string>src</string>
</array>
</dict>
</dict>
</dict>
Text printed while iterating the line array:
version=1.0
<dict>
<key>Definitions</key>
<dict>
<key>src/main.c</key>
<dict>
<key>Path</key>
<string>src/main.c</string>
<key>Group</key>
<array>
<string>src</string>
</array>
</dict>
</dict>
</dict>
Text printed using Data::Dumper :
$VAR1 = '<?xml version="1.0" encoding="UTF-8"?>';
$VAR2 = '<dict>';
$VAR3 = ' <key>Definitions</key>';
$VAR4 = ' <dict>';
$VAR5 = ' <key>src/main.c</key>';
$VAR6 = ' <dict>';
$VAR7 = ' <key>Path</key>';
$VAR8 = ' <string>src/main.c</string>';
$VAR9 = ' <key>Group</key>';
$VAR10 = ' <array>';
$VAR11 = ' <string>src</string>';
$VAR12 = ' </array>';
$VAR13 = ' </dict>';
$VAR14 = ' </dict>';
$VAR15 = '</dict>';
Don't use while (<#lines>), it doesn't do what you think.*
Instead, use a for loop:
for (#lines) {
print $_, "\n";
}
*) It calls glob, i.e. it joins the contents of the array with $" and interprets the result as a glob expression, i.e. whitespace separated wildcard patterns.
Corrected OP's demo code to read lines into an array #lines
use strict;
use warnings;
use feature 'say';
use Data::Dumper;
my $fname = 'formatted.txt';
my #lines;
open my $fh, '<', $fname
or die "Couldn't open $fname";
#lines = <$fh>;
close $fh;
chomp(#lines);
say Dumper(\#lines);
say for #lines;
Input
<?xml version="1.0" encoding="UTF-8"?>
<dict>
<key>Definitions</key>
<dict>
<key>src/main.c</key>
<dict>
<key>Path</key>
<string>src/main.c</string>
<key>Group</key>
<array>
<string>src</string>
</array>
</dict>
</dict>
</dict>
Output
$VAR1 = [
'<?xml version="1.0" encoding="UTF-8"?>',
'<dict>',
' <key>Definitions</key>',
' <dict>',
' <key>src/main.c</key>',
' <dict>',
' <key>Path</key>',
' <string>src/main.c</string>',
' <key>Group</key>',
' <array>',
' <string>src</string>',
' </array>',
' </dict>',
' </dict>',
'</dict>'
];
<?xml version="1.0" encoding="UTF-8"?>
<dict>
<key>Definitions</key>
<dict>
<key>src/main.c</key>
<dict>
<key>Path</key>
<string>src/main.c</string>
<key>Group</key>
<array>
<string>src</string>
</array>
</dict>
</dict>
</dict>

How to use XQuery to simulate STRING_AGG() (grouped string concatenation)?

I have XML data which I need to convert to relational form. I use XQuery cause I don't know the number of address nodes. I'd like to get the whole address/adresses separeted by a comma. I guess I need to use LET clause but I'm still receiving an error.
Here's my code:
declare #xml as xml = '<root>
<Row>
<proceeding>
<signatures>V GU 86/18</signatures>
<signatures>V GUp 9/19</signatures>
<signatures>V GUp 8/19</signatures>
</proceeding>
<entity>
<info>
<cleaned_name>Kate Smith</cleaned_name>
</info>
<address>
<town>London </town>
<house_number>1 </house_number>
<flat_number>1</flat_number>
<street>Downing Street</street>
<zip_code>00-001</zip_code>
</address>
</entity>
<entity>
<info>
<cleaned_name>John Smith</cleaned_name>
</info>
<address>
<town>Washington </town>
<house_number>1</house_number>
<flat_number>1</flat_number>
<street>Pennsylvania Avenue</street>
<zip_code>00-001</zip_code>
</address>
</entity>
</Row>
</root>'
select
isnull(STUFF(a.x.query('for $s in entity/info/cleaned_name return <x>{concat(",",$s)}</x>').value('.','varchar(max)'),1,1,''),'') as 'Nazwa podmiotu'
,isnull(STUFF(a.x.query('for $s in proceeding/signatures return <x>{concat(",",$s)}</x>').value('.','varchar(max)'),1,1,''),'') as 'Sygnatura'
--,isnull(STUFF(a.x.query('for $s in entity let $L := $s/entity/address return <x>{concat(",",Address="{$s/Address}")}</x>').value('.','varchar(max)'),1,1,''),'')
from #xml.nodes('/root/Row') as a(x)
My desired outcome
Are you looking for this?
select
isnull(STUFF(a.x.query('for $s in entity/info/cleaned_name return <x>{concat(",",$s)}</x>').value('.','varchar(max)'),1,1,''),'') as 'Nazwa podmiotu'
,isnull(STUFF(a.x.query('for $s in proceeding/signatures return <x>{concat(",",$s)}</x>').value('.','varchar(max)'),1,1,''),'') as 'Sygnatura'
,isnull(STUFF(a.x.query('for $s in entity
return <x>
{
concat(", ",($s/address/zip_code/text())[1]," "
,($s/address/town/text())[1]," "
,($s/address/street/text())[1]," "
,($s/address/house_number/text())[1],"/"
,($s/address/flat_number/text())[1]
)
}
</x>').value('.','varchar(max)'),1,2,''),'')
from #xml.nodes('/root/Row') as a(x);
The result
Nazwa podmiotu Sygnatura AllAdresses
Kate Smith,John Smith V GU 86/18,V GUp 9/19,V GUp 8/19 00-001 London Downing Street 1 /1, 00-001 Washington Pennsylvania Avenue 1/1
UPDATE Multiple addresses and identical data
You can try this (according to your comment)
Your test data with one second address and one copied address:
declare #xml as xml = '<root>
<Row>
<proceeding>
<signatures>V GU 86/18</signatures>
<signatures>V GUp 9/19</signatures>
<signatures>V GUp 8/19</signatures>
</proceeding>
<entity>
<info>
<cleaned_name>Kate Smith</cleaned_name>
</info>
<address>
<town>London </town>
<house_number>1 </house_number>
<flat_number>1</flat_number>
<street>Downing Street</street>
<zip_code>00-001</zip_code>
</address>
<address>
<town>Yorkshire </town>
<house_number>1 </house_number>
<flat_number>1</flat_number>
<street>Morning Street</street>
<zip_code>00-999</zip_code>
</address>
</entity>
<entity>
<info>
<cleaned_name>John Smith</cleaned_name>
</info>
<address>
<town>Washington </town>
<house_number>1</house_number>
<flat_number>1</flat_number>
<street>Pennsylvania Avenue</street>
<zip_code>00-001</zip_code>
</address>
<address>
<town>Washington </town>
<house_number>1</house_number>
<flat_number>1</flat_number>
<street>Pennsylvania Avenue</street>
<zip_code>00-001</zip_code>
</address>
</entity>
</Row>
</root>'
--The query
select
isnull(STUFF(a.x.query('for $s in entity/info/cleaned_name return <x>{concat(",",$s)}</x>').value('.','varchar(max)'),1,1,''),'') as 'Nazwa podmiotu'
,isnull(STUFF(a.x.query('for $s in proceeding/signatures return <x>{concat(",",$s)}</x>').value('.','varchar(max)'),1,1,''),'') as 'Sygnatura'
,isnull(STUFF(a.x.query('for $s in entity/address
return
<x>{concat(", ",($s/zip_code/text())[1]," "
,($s/town/text())[1]," "
,($s/street/text())[1]," "
,($s/house_number/text())[1],"/"
,($s/flat_number/text())[1]
)}</x>')
.query('for $a in distinct-values(/x/text()) return $a').value('.','varchar(max)'),1,2,''),'')
from #xml.nodes('/root/Row') as a(x);
The idea in short:
We use the first XQuery to create a simple XML fragment like this
<x>, 00-001 London Downing Street 1 /1</x>
<x>, 00-999 Yorkshire Morning Street 1 /1</x>
<x>, 00-001 Washington Pennsylvania Avenue 1/1</x>
<x>, 00-001 Washington Pennsylvania Avenue 1/1</x>
With this we can use a second XQuery and place distinct-values() there.

when use isql,i got this: [IM002][unixODBC][Driver Manager]Data source name not found, and no default driver specified

when i type fllow command: isql dsnOracle -v
i got :
[IM002][unixODBC][Driver Manager]Data source name not found, and no default driver specified
[ISQL]ERROR: Could not SQLConnect
my config file:
[root#localhost lib]# cat /etc/odbc.ini
[dsnOracle]
Description = data source to oracle
Driver = Oracle
Servername = PARA_ORACLE
port = 1521
[root#localhost lib]# cat /etc/odbcinst.ini
[Oracle]
Description = ODBC for Oracle 
Driver  = /usr/lib/libsqora.so.11.1
[root#localhost lib]# cat $TNS_ADMIN/tnsnames.ora
PARA_ORACLE =
(DESCRIPTION =
(ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 172.100.2.13)(PORT = 1521)) )
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl)
)
)
[root#localhost lib]# rpm -qa |grep oracle
oracle-instantclient11.2-odbc-11.2.0.1.0-1.x86_64
oracle-instantclient11.2-basic-11.2.0.1.0-1.x86_64
[root#localhost lib]# rpm -qa |grep ODBC
unixODBC-2.2.14-12.el6_3.x86_64
i have try for a long time ,but i always get :data source name not found .
(it's there any relation with x86 or x64?).
any suggestion is fine for me.
finally i change my /etc/odbc.ini like this and it works.
[dsnOracle]
Application Attributes=T
Attributes=W
BatchAutocommitMode=IfAllSuccessful
BindAsFLOAT=F
CloseCursor=F
DisableDPM=F
DisableMTS=T
Driver=Oracle
DSN=OracleODBC-11g
EXECSchemaOpt=
EXECSyntax=T
Failover=T
FailoverDelay=10
FailoverRetryCount=10
FetchBufferSize=64000
ForceWCHAR=F
Lobs=T
Longs=T
MaxLargeData=0
MetadataIdDefault=F
QueryTimeout=T
ResultSets=T
SQLGetData extensions=F
Translation DLL=
Translation Option=0
DisableRULEHint=T
StatementCache=F
CacheBufferSize=20
UseOCIDescribeAny=F
UserID=userid
ServerName=PARA_ORACLE
Password=password

Bash Pass Arrays to Function

I am unclear what I am doing wrong here or is there some type of bash issue?
I am declaring some static sting arrays and interactively building up another array of string values, all to be passed to a bash function. I get corruption of the array values it seems inside the function itself.
add_firewall_rich_rules() {
node_ips="${1}"
swarm_tcp_ports="${2}"
swarm_udp_ports="${3}"
echo "in: ${node_ips[#]}"
echo "in: ${swarm_tcp_ports[#]}"
echo "in: ${swarm_udp_ports[#]}"
for ip in "${node_ips[#]}"
do
for tcp_port in "${swarm_tcp_ports[#]}"
do
rule="'rule family=\"ipv4\" source address=\"$ip\" port protocol=\"tcp\" port=\"$tcp_port\" accept'"
cmd="firewall-cmd --permanent --zone=public --add-rich-rule=$rule"
echo "$cmd"
done
for udp_port in "${swarm_udp_ports[#]}"
do
rule="'rule family=\"ipv4\" source address=\"$ip\" port protocol=\"udp\" port=\"$udp_port\" accept'"
cmd="firewall-cmd --permanent --zone=public --add-rich-rule=$rule"
echo "$cmd"
done
done
}
declare -a swarm_tcp_ports=('2377' '7946' '4789' '8500' '4000')
declare -a swarm_udp_ports=('2377' '7946' '4789')
declare -a node_ips=()
echo "init: ${node_ips[#]}"
echo "init: ${swarm_tcp_ports[#]}"
echo "init: ${swarm_udp_ports[#]}"
node_ip=""
last_node_ip=""
while read -e -p "Enter ip of additional node in the cluster (hit enter twice to stop adding values): " -i "`echo $node_ip |sed 's/[^.]*$//'`" node_ip; do
if [ "$node_ip" == "$last_node_ip" ]; then
break
fi
if [[ $node_ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
node_ips=("${node_ips[#]}" $node_ip)
else
echo "hit return again to stop adding values"
fi
last_node_ip=$node_ip
echo "${swarm_tcp_ports[#]}"
done
if [ "${#node_ips[#]}" -gt 0 ]; then
echo "out: ${node_ips[#]}"
echo "out: ${swarm_tcp_ports[#]}"
echo "out: ${swarm_udp_ports[#]}"
add_firewall_rich_rules ${node_ips[#]} ${swarm_tcp_ports[#]} ${swarm_udp_ports[#]}
fi
From the Terminal:
# ./firewall_add_rich_rule.sh
init:
init: 2377 7946 4789 8500 4000
init: 2377 7946 4789
Enter ip of additional node in the cluster (hit enter twice to stop adding values): 192.168.1.105
2377 7946 4789 8500 4000
Enter ip of additional node in the cluster (hit enter twice to stop adding values): 192.168.1.106
2377 7946 4789 8500 4000
Enter ip of additional node in the cluster (hit enter twice to stop adding values): 192.168.1.
hit return again to stop adding values
2377 7946 4789 8500 4000
Enter ip of additional node in the cluster (hit enter twice to stop adding values): 192.168.1.
out: 192.168.1.105 192.168.1.106
out: 2377 7946 4789 8500 4000
out: 2377 7946 4789
in: 192.168.1.105 192.168.1.106
in: 192.168.1.106 7946 4789 8500 4000
in: 2377 7946 4789
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.105" port protocol="tcp" port="192.168.1.106" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.105" port protocol="tcp" port="7946" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.105" port protocol="tcp" port="4789" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.105" port protocol="tcp" port="8500" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.105" port protocol="tcp" port="4000" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.105" port protocol="udp" port="2377" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.105" port protocol="udp" port="7946" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.105" port protocol="udp" port="4789" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.106" port protocol="tcp" port="192.168.1.106" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.106" port protocol="tcp" port="7946" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.106" port protocol="tcp" port="4789" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.106" port protocol="tcp" port="8500" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.106" port protocol="tcp" port="4000" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.106" port protocol="udp" port="2377" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.106" port protocol="udp" port="7946" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.106" port protocol="udp" port="4789" accept'
It seems like the first element from the first array passed into the function is replacing the first element of the next array passed after the function call is made (and i have noticed sometimes the next array as well but not this time).
When you pass your arrays to the function, it expands the values.
arr1=( "1" "2" "3" )
arr2=( "a" "b" "c" )
someFunc "${arr1[#]}" "${arr2[#]}"
#same as someFunc "1" "2" "3" "a" "b" "c" "d"
So in your function you should take the name of the arrays rather than expanding them and declaring them as a new array inside your functions.
arrayTest(){
declare -a localArr1=("${!1}")
declare -a localArr2=("${!2}")
//process arrays
}
arr1=( "1" "2" "3" )
arr2=( "a" "b" "c" )
arrayTest "arr1[#]" "arr2[#]"

Resources