Parsing stdout chunks into arrays in bash or ruby - arrays

Question

I am trying to find the most efficient way to turn stdout log entries from racadm (dell chassis/idrac) into individual arrays or json arrays so I can evaluate each entry one at a time. The output always has the same fields. The output below is pretty typical
$ racadm chassislog view -c Storage -b PDR
SeqNumber = 11700
Message ID = PDR17
Category = Storage
AgentID = CMC
Severity = Information
Timestamp = 2020-03-21 00:02:06
Message Arg 1 = Physical Disk 0:0:15
FQDD = Disk.Bay.15:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1
Message = Global hot spare assigned to Physical Disk 0:0:15.
--------------------------------------------------------------------------------
SeqNumber = 11699
Message ID = PDR26
Category = Storage
AgentID = CMC
Severity = Information
Timestamp = 2020-03-21 00:02:04
Message Arg 1 = Physical Disk 0:0:3
FQDD = Disk.Bay.3:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1
Message = Physical Disk 0:0:3 is online.
--------------------------------------------------------------------------------
SeqNumber = 11696
Message ID = PDR71
Category = Storage
AgentID = CMC
Severity = Information
Timestamp = 2020-03-21 00:02:01
Message Arg 1 = Physical Disk 0:0:15
Message Arg 2 = Physical Disk 0:0:3
FQDD = Disk.Bay.15:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1
Message = Copyback completed from Physical Disk 0:0:15 to Physical Disk 0:0:3.
--------------------------------------------------------------------------------
SeqNumber = 11670
Message ID = PDR70
Category = Storage
AgentID = CMC
Severity = Information
Timestamp = 2020-03-20 21:45:47
Message Arg 1 = Physical Disk 0:0:15
Message Arg 2 = Physical Disk 0:0:3
FQDD = Disk.Bay.15:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1
Message = Copyback started from Physical Disk 0:0:15 to Physical Disk 0:0:3.
--------------------------------------------------------------------------------
SeqNumber = 11667
Message ID = PDR8
Category = Storage
AgentID = CMC
Severity = Information
Timestamp = 2020-03-20 21:45:44
Message Arg 1 = Physical Disk 0:0:3
FQDD = Disk.Bay.3:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1
Message = Physical Disk 0:0:3 is inserted.
--------------------------------------------------------------------------------
i'd really love to read the entire output into an associative array so I could step through
each entry in a for loop for events. Looking for guidance in ruby(chef) or bash.

Answer 1

This perl one-liner converts input like the above into an array of JSON objects which you can then process in any JSON-aware tool.
racadm chassislog view -c Storage -b PDR | \
perl -MJSON::PP -lne 'if (/([^=]*?)\s*=\s*(.*)/) { $obj{$1} = $2 }
elsif (/^-+$/) { push #records, { %obj }; undef %obj }
END { push #records, { %obj } if defined %obj;
print encode_json(\#records) }'
outputs (After pretty-printing):
[
{
"Timestamp": "2020-03-21 00:02:06",
"Message ID": "PDR17",
"Category": "Storage",
"Message": "Global hot spare assigned to Physical Disk 0:0:15.",
"AgentID": "CMC",
"Severity": "Information",
"SeqNumber": "11700",
"FQDD": "Disk.Bay.15:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1",
"Message Arg 1": "Physical Disk 0:0:15"
},
{
"Category": "Storage",
"Message ID": "PDR26",
"Timestamp": "2020-03-21 00:02:04",
"SeqNumber": "11699",
"Message": "Physical Disk 0:0:3 is online.",
"Severity": "Information",
"AgentID": "CMC",
"Message Arg 1": "Physical Disk 0:0:3",
"FQDD": "Disk.Bay.3:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1"
},
{
"FQDD": "Disk.Bay.15:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1",
"Message Arg 2": "Physical Disk 0:0:3",
"Message Arg 1": "Physical Disk 0:0:15",
"Severity": "Information",
"AgentID": "CMC",
"Message": "Copyback completed from Physical Disk 0:0:15 to Physical Disk 0:0:3.",
"SeqNumber": "11696",
"Timestamp": "2020-03-21 00:02:01",
"Category": "Storage",
"Message ID": "PDR71"
},
{
"Message Arg 1": "Physical Disk 0:0:15",
"FQDD": "Disk.Bay.15:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1",
"Message Arg 2": "Physical Disk 0:0:3",
"SeqNumber": "11670",
"Message": "Copyback started from Physical Disk 0:0:15 to Physical Disk 0:0:3.",
"Severity": "Information",
"AgentID": "CMC",
"Category": "Storage",
"Message ID": "PDR70",
"Timestamp": "2020-03-20 21:45:47"
},
{
"Timestamp": "2020-03-20 21:45:44",
"Message ID": "PDR8",
"Category": "Storage",
"Message": "Physical Disk 0:0:3 is inserted.",
"AgentID": "CMC",
"Severity": "Information",
"SeqNumber": "11667",
"FQDD": "Disk.Bay.3:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1",
"Message Arg 1": "Physical Disk 0:0:3"
}
]

Answer 2

Not bash, since shell is for handling files and launching commands, but using GNU awk which is often falsely percieved as part of the shell, it's simple yet powerfull programming language. step through each entry in a for loop for events is not really a requirement so here is a small sample:
$ gawk -v item="Message Arg 2" ' # queried item as parameter
BEGIN {
RS="\n-+$\n" # record is separated by a bunch of -:s
FS="\n" # a line is a field within a record
}
{
for(nf=1;nf<=NF;nf++) { # loop all lines in a record
split($nf,t,/ *= */) # split lines by = and surrounding space
a[NR][t[1]]=t[2] # hash to a 2 dimensional array indexed by
} # record no. and the item, value as value
}
END { # after lines are hashed, make queries
for(nr in a) # for each record in hash
if(item in a[nr]) # if queried item is found in it
printf "%d: %s = %s\n", nr,item,a[nr][item] # output
}' file
Output for query item Message Arg 2:
3: Message Arg 2 = Physical Disk 0:0:3
4: Message Arg 2 = Physical Disk 0:0:3
Here is an alternate ending for match a condition im looking for in "Message" I would like to reference the corresponding FQDD:
$ gawk -v item=Message -v cond=started -v output=FQDD
BEGIN {
RS="\n-+$\n" # record is separated by a bunch of -:s
FS="\n" # a line is a field within a record
}
{
for(nf=1;nf<=NF;nf++) { # loop all lines in a record
split($nf,t,/ *= */) # split lines by = and surrounding space
a[NR][t[1]]=t[2] # hash to a 2 dimensional array indexed by
} # record no. and the item, value as value
}
END {
for(nr in a)
if((item in a[nr]) && a[nr][item]~cond)
printf "%d: %s = %s\n", nr,output,a[nr][output]
}
Output now:
4: FQDD = Disk.Bay.15:Enclosure.Internal.0-0:RAID.ChassisIntegrated.1-1
ie. if variable item is found in a[nr][item] and that array elemets value matches with cond print the value of a[nr]["FQDD"] in the same record.
In SQL that would be SELECT output FROMfileWHERE item LIKE '%cond%'

Answer 3

Based on Shawns one liner as a pattern, a colleague ended up finding a python 2.7 compatible way to do exactly what we want, code is below and offers the exact functionality I need.
import re
import json
from pprint import pprint
regex_string_1 = '([^=]*?)\s*=\s*(.*)'
regex_string_2 = '^-+$'
regex1 = re.compile(regex_string_1)
regex2 = re.compile(regex_string_2)
current_entry = {}
entries = []
lines = test.split('\n')
for line in lines:
if regex1.match(line):
key, value = [element.strip() for element in line.split('=')]
current_entry[key] = value
elif regex2.match(line):
entries.append(current_entry)
current_entry = {}
pprint(entries)