Extracting data from file to populate database with bash script - database

Question

I have a dataset of many files. Each file contains many reviews of the type separated by a blank line:
<Author>bigBob
<Content>definitely above average! we had a really nice stay there last year when I and...USUALLY OVER MANY LINES
<Date>Jan 2, 2009
<img src="http://cdn.tripadvisor.com/img2/new.gif" alt="New"/>
<No. Reader>-1
<No. Helpful>-1
<Overall>4
<Value>4
<Rooms>4
<Location>4
<Cleanliness>5
<Check in / front desk>4
<Service>3
<Business service>4
<Author>rickMN... next review goes on
For every review I need to extract the data after the tag and put it in something like this (which I plan write to a .sql file so when I do ".read" it will populate my database):
INSERT INTO [HotelReviews] ([Author], [Content], [Date], [Image], [No_Reader], [No_Helpful], [Overall], [Value], [Rooms], [Location], [Cleanliness], [Check_In], [Service], [Business_Service]) VALUES ('bigBob', 'definitely above...', ...)
My question is how can I extract the data after each tag and put it in an insert statement using bash?
EDIT
Text after <Content> tag is usually a paragraph with a number of lines

Answer 1

This is the right approach for what you're trying to do:
$ cat tst.awk
NF {
if ( match($0,/^<img\s+src="([^"]+)/,a) ) {
name="Image"
value=a[1]
}
else if ( match($0,/^<([^>"]+)>(.*)/,a) ) {
name=a[1]
value=a[2]
sub(/ \/.*|\./,"",name)
gsub(/ /,"_",name)
}
names[++numNames] = name
values[numNames] = value
next
}
{ prt() }
END { prt() }
function prt() {
printf "INSERT INTO [HotelReviews] ("
for (nameNr=1; nameNr<=numNames; nameNr++) {
printf " [%s]", names[nameNr]
}
printf ") VALUES ("
for (nameNr=1; nameNr<=numNames; nameNr++) {
printf " \047%s\047", values[nameNr]
}
print ""
numNames = 0
delete names
delete values
}
.
$ awk -f tst.awk file
INSERT INTO [HotelReviews] ( [Author] [Content] [Date] [Image] [No_Reader] [No_Helpful] [Overall] [Value] [Rooms] [Location] [Cleanliness] [Check_in] [Service] [Business_service]) VALUES ( 'bigBob' 'definitely above average! we had a really nice stay there last year when I and...USUALLY OVER MANY LINES' 'Jan 2, 2009' 'http://cdn.tripadvisor.com/img2/new.gif' '-1' '-1' '4' '4' '4' '4' '5' '4' '3' '4'
INSERT INTO [HotelReviews] ( [Author]) VALUES ( 'rickMN... next review goes on'
The above uses GNU awk for the 3rd arg to match(). Massage to get the precise formatting/output you want.

Answer 2

Example:
#!/bin/bash
while IFS= read -r line; do
[[ $line =~ ^\<Author\>(.*) ]] && Author="${BASH_REMATCH[1]}"
[[ $line =~ ^\<Content\>(.*) ]] && Content="${BASH_REMATCH[1]}"
# capture lines not starting with < and append to variable Content
[[ $line =~ ^[^\<] ]] && Content+="$line"
# match an empty line
[[ $line =~ ^$ ]] && echo "${Author}, ${Content}"
done < file
Output with your file:
bigBob, definitely above average! we had a really nice stay there last year when I and ...
=~: match to a regex (string left, regex right without quotes)
^: match start of line
\< or \>: match < or >
.*: here match rest of line
(.*): capture rest of line to first element of array BASH_REMATCH
See: The Stack Overflow Regular Expressions FAQ