Parsing XML in T-SQL - ABS data - sql-server

Question

I'm after some assistance parsing an xml file provided by the Australian Bureau of Statistics. I've read over everything I can find here and online, and am not having much luck reading this data.
This is a sample of the XML:
<?xml version='1.0' encoding='UTF-8'?>
<message:MessageGroup xmlns:message="http://www.w3.org/2001/XMLSchema" xmlns="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/generic" xmlns:common="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/common" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/generic http://www.sdmx.org/docs/2_0/SDMXGenericData.xsd http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message http://www.sdmx.org/docs/2_0/SDMXMessage.xsd">
<Header xmlns="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message">
<ID>none</ID>
<Test>false</Test>
<Truncated>false</Truncated>
<Prepared>2018-02-14T22:41:03</Prepared>
<Sender id="ABS">
<Name xml:lang="en">Australian Bureau of Statistics</Name>
<Name xml:lang="fr">Australian Bureau of Statistics</Name>
</Sender>
</Header>
<DataSet keyFamilyURI="http://stat.data.abs.gov.au/restsdmx/sdmx.ashx/GetKeyFamily/RES_PROP_INDEX">
<KeyFamilyRef>RES_PROP_INDEX</KeyFamilyRef>
<Series>
<SeriesKey>
<Value concept="MEASURE" value="1" />
<Value concept="PROP_TYPE" value="3" />
<Value concept="ASGS_2011" value="3GBRI" />
<Value concept="FREQUENCY" value="Q" />
</SeriesKey>
<Attributes>
<Value concept="TIME_FORMAT" value="P3M" />
</Attributes>
<Obs>
<Time>2015-Q1</Time>
<ObsValue value="112.7" />
</Obs>
<Obs>
<Time>2015-Q2</Time>
<ObsValue value="113.7" />
</Obs>
</Series>
<Series>
<SeriesKey>
<Value concept="MEASURE" value="1" />
<Value concept="PROP_TYPE" value="3" />
<Value concept="ASGS_2011" value="5GPER" />
<Value concept="FREQUENCY" value="Q" />
</SeriesKey>
<Attributes>
<Value concept="TIME_FORMAT" value="P3M" />
</Attributes>
<Obs>
<Time>2015-Q1</Time>
<ObsValue value="114.4" />
</Obs>
<Obs>
<Time>2015-Q2</Time>
<ObsValue value="113.4" />
</Obs>
</Series>
<Annotations>
<common:Annotation>
<common:AnnotationTitle>Statistical usage warning</common:AnnotationTitle>
<common:AnnotationText>ABS.Stat beta is continuing to be developed. Data will be updated as soon as possible following its 11:30 am release on the ABS website.</common:AnnotationText>
</common:Annotation>
</Annotations>
</DataSet>
</message:MessageGroup>
The result set should return 3 values:
SeriesKey Value where Concept = 'ASGS_2011'
Obs "Time" Value
Obs "Obsvalue" Value
e.g the first record would return a row like:
3GBRI | 2015-Q1 | 112.7
This sample would return 4 rows of data like this.
I've tried inserting the XML into an XML variable or a column in a DB with an XML datatype, and then using SQL's XML functionality, but I'm an absolute novice at this and am having trouble with the correct coding/approach.
Any advice or sample code would be greatly appreciated.

Answer 1

I'd suggest do get rid of the <?xml blah ?> declaration. This is only useful when you store an XML to a file. Within SQL-Server any XML's encoding is fixed to unicode / UCS-2. This can lead to encoding problems...
There are namespaces involved...
DECLARE #xml XML=
'<?xml version=''1.0'' encoding=''UTF-8''?>
<message:MessageGroup xmlns:message="http://www.w3.org/2001/XMLSchema" xmlns="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/generic" xmlns:common="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/common" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/generic http://www.sdmx.org/docs/2_0/SDMXGenericData.xsd http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message http://www.sdmx.org/docs/2_0/SDMXMessage.xsd">
<Header xmlns="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message">
<ID>none</ID>
<Test>false</Test>
<Truncated>false</Truncated>
<Prepared>2018-02-14T22:41:03</Prepared>
<Sender id="ABS">
<Name xml:lang="en">Australian Bureau of Statistics</Name>
<Name xml:lang="fr">Australian Bureau of Statistics</Name>
</Sender>
</Header>
<DataSet keyFamilyURI="http://stat.data.abs.gov.au/restsdmx/sdmx.ashx/GetKeyFamily/RES_PROP_INDEX">
<KeyFamilyRef>RES_PROP_INDEX</KeyFamilyRef>
<Series>
<SeriesKey>
<Value concept="MEASURE" value="1" />
<Value concept="PROP_TYPE" value="3" />
<Value concept="ASGS_2011" value="3GBRI" />
<Value concept="FREQUENCY" value="Q" />
</SeriesKey>
<Attributes>
<Value concept="TIME_FORMAT" value="P3M" />
</Attributes>
<Obs>
<Time>2015-Q1</Time>
<ObsValue value="112.7" />
</Obs>
<Obs>
<Time>2015-Q2</Time>
<ObsValue value="113.7" />
</Obs>
</Series>
<Series>
<SeriesKey>
<Value concept="MEASURE" value="1" />
<Value concept="PROP_TYPE" value="3" />
<Value concept="ASGS_2011" value="5GPER" />
<Value concept="FREQUENCY" value="Q" />
</SeriesKey>
<Attributes>
<Value concept="TIME_FORMAT" value="P3M" />
</Attributes>
<Obs>
<Time>2015-Q1</Time>
<ObsValue value="114.4" />
</Obs>
<Obs>
<Time>2015-Q2</Time>
<ObsValue value="113.4" />
</Obs>
</Series>
<Annotations>
<common:Annotation>
<common:AnnotationTitle>Statistical usage warning</common:AnnotationTitle>
<common:AnnotationText>ABS.Stat beta is continuing to be developed. Data will be updated as soon as possible following its 11:30 am release on the ABS website.</common:AnnotationText>
</common:Annotation>
</Annotations>
</DataSet>
</message:MessageGroup>';
--The query will declare the needed namespaces and then use .nodes() to get the <Series> and a second call to .nodes() to get the nested <Obs>:
WITH XMLNAMESPACES(DEFAULT 'http://www.SDMX.org/resources/SDMXML/schemas/v2_0/generic'
,'http://www.w3.org/2001/XMLSchema' AS message)
SELECT ser.value(N'(SeriesKey/Value[#concept="ASGS_2011"]/#value)[1]',N'nvarchar(max)') AS Concept
,obs.value(N'(Time/text())[1]',N'nvarchar(max)') AS [Time]
,obs.value(N'(ObsValue/#value)[1]',N'decimal(10,4)') AS ObsValue
FROM #xml.nodes(N'/message:MessageGroup/DataSet/Series') AS A(ser)
OUTER APPLY A.ser.nodes(N'Obs') AS B(obs);
The result
Concept Time ObsValue
3GBRI 2015-Q1 112.7000
3GBRI 2015-Q2 113.7000
5GPER 2015-Q1 114.4000
5GPER 2015-Q2 113.4000