Parse XML file using SQL into Specific Columns - sql-server

I have an XML file that looks like this:
<?xml version="1.0" encoding = "utf-8"?>
<CIM CIMVERSION="2.0" DTDVERSION="2.0">
<MESSAGE ID="1111" PROTOCOLVERSION="1.0">
<SIMPLEREQ>
<VALUE.NAMEDINSTANCE>
<INSTANCENAME CLASSNAME="DCIM_ControllerView">
<KEYBINDING NAME="InstanceID">
<KEYVALUE VALUETYPE="string">RAID.Integrated.1-1</KEYVALUE>
</KEYBINDING>
</INSTANCENAME>
<INSTANCE CLASSNAME="DCIM_ControllerView">
<PROPERTY NAME="LastUpdateTime" TYPE="string">
<VALUE>20170223172914.000000+000</VALUE>
<DisplayValue>2017-02-23T17:29:14</DisplayValue>
</PROPERTY>
<PROPERTY NAME="LastSystemInventoryTime" TYPE="string">
<VALUE>20170223172914.000000+000</VALUE>
<DisplayValue>2017-02-23T17:29:14</DisplayValue>
</PROPERTY>
<PROPERTY NAME="RealtimeCapability" TYPE="uint32">
<VALUE>6</VALUE>
<DisplayValue>Capable</DisplayValue>
</PROPERTY>
<PROPERTY NAME="SupportControllerBootMode" TYPE="uint8">
<VALUE>1</VALUE>
<DisplayValue>Supported</DisplayValue>
</PROPERTY>
<PROPERTY NAME="SupportEnhancedAutoForeignImport" TYPE="uint8">
<VALUE>1</VALUE>
<DisplayValue>Supported</DisplayValue>
</PROPERTY>
<PROPERTY NAME="MaxAvailablePCILinkSpeed" TYPE="string">
<VALUE>Generation 3</VALUE>
<DisplayValue>Generation 3</DisplayValue>
</PROPERTY>
<PROPERTY NAME="MaxPossiblePCILinkSpeed" TYPE="string">
<VALUE>Generation 3</VALUE>
<DisplayValue>Generation 3</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PatrolReadState" TYPE="uint8">
<VALUE>1</VALUE>
<DisplayValue>Stopped</DisplayValue>
</PROPERTY>
<PROPERTY NAME="DriverVersion" TYPE="string">
<DisplayValue>Not Applicable</DisplayValue>
</PROPERTY>
<PROPERTY NAME="CacheSizeInMB" TYPE="uint32">
<VALUE>0</VALUE>
<DisplayValue>0 MB</DisplayValue>
</PROPERTY>
<PROPERTY NAME="SupportRAID10UnevenSpans" TYPE="uint8">
<VALUE>1</VALUE>
<DisplayValue>Supported</DisplayValue>
</PROPERTY>
<PROPERTY NAME="T10PICapability" TYPE="uint8">
<VALUE>1</VALUE>
<DisplayValue>Supported</DisplayValue>
</PROPERTY>
<PROPERTY NAME="SlicedVDCapability" TYPE="uint8">
<VALUE>1</VALUE>
<DisplayValue>Sliced Virtual Disk creation supported</DisplayValue>
</PROPERTY>
<PROPERTY NAME="CachecadeCapability" TYPE="uint8">
<VALUE>0</VALUE>
<DisplayValue>Cachecade Virtual Disk not supported</DisplayValue>
</PROPERTY>
<PROPERTY NAME="KeyID" TYPE="string">
<DisplayValue/>
</PROPERTY>
<PROPERTY NAME="EncryptionCapability" TYPE="uint8">
<VALUE>1</VALUE>
<DisplayValue>Local Key Management Capable</DisplayValue>
</PROPERTY>
<PROPERTY NAME="EncryptionMode" TYPE="uint8">
<VALUE>0</VALUE>
<DisplayValue>None</DisplayValue>
</PROPERTY>
<PROPERTY NAME="SecurityStatus" TYPE="uint32">
<VALUE>1</VALUE>
<DisplayValue>Encryption Capable</DisplayValue>
</PROPERTY>
<PROPERTY NAME="SASAddress" TYPE="string">
<VALUE>1111</VALUE>
<DisplayValue>1111</DisplayValue>
</PROPERTY>
<PROPERTY NAME="ProductName" TYPE="string">
<VALUE>PERC Mini</VALUE>
<DisplayValue>PERC Mini</DisplayValue>
</PROPERTY>
<PROPERTY NAME="DeviceCardSlotType" TYPE="string">
<VALUE>Unknown</VALUE>
<DisplayValue>Unknown</DisplayValue>
</PROPERTY>
<PROPERTY NAME="DeviceCardSlotLength" TYPE="uint8">
<VALUE>2</VALUE>
<DisplayValue>Unknown</DisplayValue>
</PROPERTY>
<PROPERTY NAME="DeviceCardDataBusWidth" TYPE="string">
<VALUE>Unknown</VALUE>
<DisplayValue>Unknown</DisplayValue>
</PROPERTY>
<PROPERTY NAME="DeviceCardManufacturer" TYPE="string">
<VALUE>DELL</VALUE>
<DisplayValue>DELL</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PCISubDeviceID" TYPE="string">
<VALUE>1111</VALUE>
<DisplayValue>1F47</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PCIDeviceID" TYPE="string">
<VALUE>1</VALUE>
<DisplayValue>1</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PCISubVendorID" TYPE="string">
<VALUE>1</VALUE>
<DisplayValue>1</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PCIVendorID" TYPE="string">
<VALUE>1</VALUE>
<DisplayValue>1</DisplayValue>
</PROPERTY>
<PROPERTY NAME="Function" TYPE="string">
<VALUE>0</VALUE>
<DisplayValue>0</DisplayValue>
</PROPERTY>
<PROPERTY NAME="Device" TYPE="string">
<VALUE>0</VALUE>
<DisplayValue>0</DisplayValue>
</PROPERTY>
<PROPERTY NAME="Bus" TYPE="string">
<VALUE>2</VALUE>
<DisplayValue>2</DisplayValue>
</PROPERTY>
<PROPERTY NAME="ControllerFirmwareVersion" TYPE="string">
<VALUE>00.00.00.00</VALUE>
<DisplayValue>00.00.00.00</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PCISlot" TYPE="uint8">
<VALUE>0</VALUE>
<DisplayValue>0</DisplayValue>
</PROPERTY>
<PROPERTY NAME="RollupStatus" TYPE="uint32">
<VALUE>1</VALUE>
<DisplayValue>OK</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PrimaryStatus" TYPE="uint32">
<VALUE>1</VALUE>
<DisplayValue>OK</DisplayValue>
</PROPERTY>
<PROPERTY NAME="DeviceDescription" TYPE="string">
<VALUE>Integrated RAID Controller 1</VALUE>
<DisplayValue>Integrated RAID Controller 1</DisplayValue>
</PROPERTY>
<PROPERTY NAME="FQDD" TYPE="string">
<VALUE>RAID.Integrated.1-1</VALUE>
<DisplayValue>RAID.Integrated.1-1</DisplayValue>
</PROPERTY>
<PROPERTY NAME="InstanceID" TYPE="string">
<VALUE>RAID.Integrated.1-1</VALUE>
<DisplayValue>RAID.Integrated.1-1</DisplayValue>
</PROPERTY>
</INSTANCE>
</VALUE.NAMEDINSTANCE>
</SIMPLEREQ>
</MESSAGE>
</CIM>
I am able to bring into SQL and read as an XML object.
Currently,I bring the XML text in a single string using SSIS.
After the file is read in SSIS, it bring it into my database into a staging table like this:
Now that the files can be brought into my staging table, I want to parse them into specific columns using a trigger.
Please note that the XML string are in UTF-8.
This data comes from a Dell Hardware inventory XML file.
Their document outlines their XML schema in this diagram:
I need to get that XML data into a column format that mimics that diagram. So the headers would be:
CIM | Messgae | Classname | InstanceID | PropertyName | Value | DisplayValue
These columns would contain their respective data in the XML
Also, as I mentioned it is into utf-8 format.
To read it as an XML object in SQL, I am using this code:
declare #XML xml
set #XML = (SELECT Top 1 REPLACE([XML], 'utf-8', 'utf-16')FROM [StagingTable])
SELECT #XML
Also, I have multiple instances of INSTANCENAME, with more sets INSTANCE, PROPERTY, VALUE, DISPLAYVALUE, etc.
Here is the XML schema as per the Dell Document:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema targetNamespace="http://schemas.dell.com/HWinventory/1/0/events"
elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:evt="http://schemas.dell.com/HWinventory/1/0/events">
<xs:element name="CIM">
<xs:complexType>
<xs:sequence>
<xs:element name="MESSAGE">
<xs:complexType>
<xs:sequence>
<xs:element name="SIMPLEREQ">
<xs:complexType>
<xs:sequence>
<xs:element maxOccurs="unbounded" name="VALUE.NAMEDINSTANCE">
<xs:complexType>
<xs:sequence>
<xs:element name="INSTANCENAME">
<xs:complexType>
<xs:sequence>
<xs:element name="KEYBINDING">
<xs:complexType>
<xs:sequence>
<xs:element name="KEYVALUE">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="VALUETYPE" type="xs:string" use="required" />
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:sequence>
<xs:attribute name="NAME" type="xs:string" use="required" />
</xs:complexType>
</xs:element>
</xs:sequence>
<xs:attribute name="CLASSNAME" type="xs:string" use="required" />
</xs:complexType>
</xs:element>
<xs:element name="INSTANCE">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" name="PROPERTY.ARRAY">
<xs:complexType>
<xs:sequence>
<xs:element name="VALUE.ARRAY">
<xs:complexType>
<xs:sequence>
<xs:element name="VALUE" type="xs:string" />
<xs:element name="DisplayValue" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
<xs:attribute name="NAME" type="xs:string" use="required" />
<xs:attribute name="TYPE" type="xs:string" use="required" />
</xs:complexType>
</xs:element>
<xs:element maxOccurs="unbounded" name="PROPERTY">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" name="VALUE" type="xs:string" />
<xs:element name="DisplayValue" type="xs:string" />
</xs:sequence>
<xs:attribute name="NAME" type="xs:string" use="required" />
<xs:attribute name="TYPE" type="xs:string" use="required" />
</xs:complexType>
</xs:element>
</xs:sequence>
<xs:attribute name="CLASSNAME" type="xs:string" use="required" />
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
<xs:attribute name="ID" type="xs:int" use="required" />
<xs:attribute name="PROTOCOLVERSION" type="xs:decimal" use="required" />
</xs:complexType>
</xs:element>
</xs:sequence>


The following query will retrieve various data from your XML. It won't be exactly what you need, but it should give you a template for any value you might want to pull out:
DECLARE #xml XML=
N'<CIM CIMVERSION="2.0" DTDVERSION="2.0">
<MESSAGE ID="4711" PROTOCOLVERSION="1.0">
<SIMPLEREQ>
<VALUE.NAMEDINSTANCE xmlns:fo="http://www.w3.org/1999/XSL/Format">
<INSTANCENAME CLASSNAME="DCIM_ControllerView">
<KEYBINDING NAME="InstanceID">
<KEYVALUE VALUETYPE="string">RAID.Slot.1-1</KEYVALUE>
</KEYBINDING>
</INSTANCENAME>
<INSTANCE CLASSNAME="DCIM_ControllerView">
<PROPERTY NAME="DriverVersion" TYPE="string">
<DisplayValue />
</PROPERTY>
<PROPERTY NAME="KeyID" TYPE="string">
<DisplayValue />
</PROPERTY>
<PROPERTY NAME="SASAddress" TYPE="string">
<VALUE>5782BCB00C577600</VALUE>
<DisplayValue>5782BCB00C577600</DisplayValue>
</PROPERTY>
<PROPERTY NAME="ProductName" TYPE="string">
<VALUE>PERC H310 Adapter</VALUE>
<DisplayValue>PERC H310 Adapter</DisplayValue>
</PROPERTY>
<PROPERTY NAME="DeviceCardSlotType" TYPE="string">
<VALUE>PCI Express x8</VALUE>
<DisplayValue>PCI Express x8</DisplayValue>
</PROPERTY>
<PROPERTY NAME="DeviceCardManufacturer" TYPE="string">
<VALUE>DELL</VALUE>
<DisplayValue>DELL</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PCISubDeviceID" TYPE="string">
<VALUE>1F4E</VALUE>
<DisplayValue>1F4E</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PCIDeviceID" TYPE="string">
<VALUE>73</VALUE>
<DisplayValue>73</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PCISubVendorID" TYPE="string">
<VALUE>1028</VALUE>
<DisplayValue>1028</DisplayValue>
</PROPERTY>
<PROPERTY NAME="PCIVendorID" TYPE="string">
<VALUE>1000</VALUE>
<DisplayValue>1000</DisplayValue>
</PROPERTY>
<PROPERTY NAME="Function" TYPE="string">
<VALUE>0</VALUE>
<DisplayValue>0</DisplayValue>
</PROPERTY>
<PROPERTY NAME="Device" TYPE="string">
<VALUE>0</VALUE>
<DisplayValue>0</DisplayValue>
</PROPERTY>
<PROPERTY NAME="Bus" TYPE="string">
<VALUE>1</VALUE>
<DisplayValue>1</DisplayValue>
</PROPERTY>
<PROPERTY NAME="ControllerFirmwareVersion" TYPE="string">
<VALUE>20.10.1-0066</VALUE>
<DisplayValue>20.10.1-0066</DisplayValue>
</PROPERTY>
</INSTANCE>
</VALUE.NAMEDINSTANCE>
</SIMPLEREQ>
</MESSAGE>
</CIM>';
--The query
SELECT #xml.value(N'(/CIM/#CIMVERSION)[1]',N'nvarchar(max)') AS CIM_version
,#xml.value(N'(/CIM/MESSAGE/#ID)[1]',N'nvarchar(max)') AS Message_Id
,#xml.value(N'(/CIM/MESSAGE/SIMPLEREQ/VALUE.NAMEDINSTANCE/INSTANCENAME/KEYBINDING/KEYVALUE/text())[1]',N'nvarchar(max)') AS Keybinding_Value
,prp.value(N'#NAME',N'nvarchar(max)') AS Prop_Name
,prp.value(N'(VALUE/text())[1]',N'nvarchar(max)') AS Prop_Value
,prp.value(N'(DisplayValue/text())[1]',N'nvarchar(max)') AS Prop_DisplayValue
FROM #xml.nodes(N'/CIM/MESSAGE/SIMPLEREQ/VALUE.NAMEDINSTANCE/INSTANCE/PROPERTY') AS A(prp);
The result
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| CIM_version | Message_Id | Keybinding_Value | Prop_Name | Prop_Value | Prop_DisplayValue |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | DriverVersion | NULL | NULL |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | KeyID | NULL | NULL |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | SASAddress | 5782BCB00C577600 | 5782BCB00C577600 |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | ProductName | PERC H310 Adapter | PERC H310 Adapter |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | DeviceCardSlotType | PCI Express x8 | PCI Express x8 |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | DeviceCardManufacturer | DELL | DELL |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | PCISubDeviceID | 1F4E | 1F4E |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | PCIDeviceID | 73 | 73 |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | PCISubVendorID | 1028 | 1028 |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | PCIVendorID | 1000 | 1000 |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | Function | 0 | 0 |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | Device | 0 | 0 |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | Bus | 1 | 1 |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+
| 2.0 | 4711 | RAID.Slot.1-1 | ControllerFirmwareVersion | 20.10.1-0066 | 20.10.1-0066 |
+-------------+------------+------------------+---------------------------+-------------------+-------------------+

Related

How to parse XML which is stored in a column in SQL Server

I want to parse XML which is stored in a column in SQL Server:
Table column with XML in this picture
and the XML is:
<?xml version="1.0" encoding="UTF-8"?>
<propertylist id="root">
<property expanded="Y" id="createsdi" selectedindex="0" type="collection">
<collection>
<propertylist id="1305619640064">
<property expanded="Y" id="columnvalues" selectedindex="0" type="collection">
<collection>
<propertylist id="1396440519721" />
</collection>
</property>
<property expanded="Y" id="tests" selectedindex="0" type="collection">
<collection>
<propertylist id="p1602059752707" sequence="1000000">
<property id="id" type="simple"><![CDATA[item1]]></property>
<property id="workitemid" type="simple"><![CDATA[SOURAVTESTER|1]]></property>
</propertylist>
</collection>
</property>
<property expanded="Y" id="specs" selectedindex="0" type="collection">
<collection>
<propertylist id="p1602059825237" sequence="1000000">
<property id="id" type="simple"><![CDATA[item1]]></property>
<property id="specid" type="simple"><![CDATA[EMSpec|1]]></property>
</propertylist>
</collection>
</property>
</propertylist>
</collection>
</property>
<property expanded="Y" id="workorder" type="propertylist">
<propertylist id="root_0">
<property expanded="Y" id="graceperiod" type="propertylist">
<propertylist id="root_0_workorder_0">
<property expanded="Y" id="graceperiod" type="propertylist">
<propertylist id="root_0_workorder_0_graceperiod_0" />
</property>
<property expanded="Y" id="deviation" type="propertylist">
<propertylist id="root_0_workorder_0_graceperiod_0" />
</property>
</propertylist>
</property>
</propertylist>
</property>
</propertylist>
How can I parse the values of workitemid, specid?
I have written the following code
SELECT
ID,
ParentPropertyId = xc.value('(../#id)[1]', 'varchar(100)'),
WorkItemId = xc.value('(.)[1]', 'varchar(100)')
FROM
scheduleplanitem
CROSS APPLY
valuetree.nodes('//propertylist/property[#id="workitemid"]') xt(xc)
but I get an error
The XMLDT method 'nodes' can only be invoked on columns of type xml
The result should be:
workitemid SOURAVTESTER
specid EMSpec

Please try the following solution.
It does the following:
Removes XML prolog with the UTF-8 encoding. SQL Server implicitly converts any XML internally to UTF-16.
Converts NVARCHAR() data type to XML data type. After that XQuery
methods to handle XML become available.
SQL
-- DDL and sample data population, start
DECLARE #tbl TABLE (ID INT IDENTITY PRIMARY KEY, col NVARCHAR(MAX));
INSERT INTO #tbl (col) VALUES
(N'<?xml version="1.0" encoding="UTF-8"?>
<propertylist id="root">
<property expanded="Y" id="createsdi" selectedindex="0" type="collection">
<collection>
<propertylist id="1305619640064">
<property expanded="Y" id="columnvalues" selectedindex="0"
type="collection">
<collection>
<propertylist id="1396440519721"/>
</collection>
</property>
<property expanded="Y" id="tests" selectedindex="0"
type="collection">
<collection>
<propertylist id="p1602059752707" sequence="1000000">
<property id="id" type="simple"><![CDATA[item1]]></property>
<property id="workitemid" type="simple"><![CDATA[SOURAVTESTER|1]]></property>
</propertylist>
</collection>
</property>
<property expanded="Y" id="specs" selectedindex="0"
type="collection">
<collection>
<propertylist id="p1602059825237" sequence="1000000">
<property id="id" type="simple"><![CDATA[item1]]></property>
<property id="specid" type="simple"><![CDATA[EMSpec|1]]></property>
</propertylist>
</collection>
</property>
</propertylist>
</collection>
</property>
<property expanded="Y" id="workorder" type="propertylist">
<propertylist id="root_0">
<property expanded="Y" id="graceperiod" type="propertylist">
<propertylist id="root_0_workorder_0">
<property expanded="Y" id="graceperiod" type="propertylist">
<propertylist id="root_0_workorder_0_graceperiod_0"/>
</property>
<property expanded="Y" id="deviation" type="propertylist">
<propertylist id="root_0_workorder_0_graceperiod_0"/>
</property>
</propertylist>
</property>
</propertylist>
</property>
</propertylist>')
, (NULL);
-- DDL and sample data population, end
--DECLARE #startPos INT = LEN('<?xml version="1.0" encoding="utf-8"?>') + 1;
DECLARE #prolog NVARCHAR(100) = '<?xml version="1.0" encoding="utf-8"?>';
;WITH rs AS
(
SELECT ID
--, TRY_CAST(SUBSTRING(col
-- , #startPos
-- , LEN(col)) AS XML) AS xmldata
, TRY_CAST('<root>' + REPLACE(col, #prolog, '') + '</root>' AS XML) AS xmldata FROM #tbl
)
SELECT ID
, c.value('(./text())[1]', 'VARCHAR(100)') AS Item
FROM rs
CROSS APPLY xmldata.nodes('//property[#id=("specid","workitemid")]') AS t(c);
Output
+----+----------------+
| ID | Item |
+----+----------------+
| 1 | SOURAVTESTER|1 |
| 1 | EMSpec|1 |
+----+----------------+

Is there any Solution to download attachment from gmail in WSO2 ESB / WSO2 EI?

I am working on wso2 esb since last 6 months. I want to download attachment from gmail in wso2 esb/ei and save them into local folder.Google it and get solution only for sending attachment using gmail connector, not downloading attachment.
Note: Already seen,but can't able to get solution by referring the following link-
WSO2 esb get attach files from email
Code:
ProxyService
<proxy name="GmailConfigAttachment-TaskProxy" startOnLoad="true"
transports="http https" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<log level="custom">
<property name="Log Text"
value="Inside GmailConfigAttachment-TaskProxy Service"></property>
</log>
<property name="FORCE_SC_ACCEPTED" scope="axis2" type="STRING"
value="true" />
<property description="serviceName" name="serviceName"
scope="operation" type="STRING"
value="TaskScheduler_ASG_Read_Email_Body_Dummy_Service" />
<property description="messageId" expression="get-property('MessageID')"
name="messageId" scope="operation" type="STRING" />
<sequence key="Load_Gmail_Configuration" />
<gmail.init>
<userId>{$ctx:userId}</userId>
<accessToken>{$ctx:accessToken}</accessToken>
<apiUrl>{$ctx:apiUrl}</apiUrl>
<clientId>{$ctx:clientId}</clientId>
<clientSecret>{$ctx:clientSecret}</clientSecret>
<refreshToken>{$ctx:refreshToken}</refreshToken>
<accessTokenRegistryPath>{$ctx:registryPath}</accessTokenRegistryPath>
</gmail.init>
<gmail.listAllMails>
<labelIds>INBOX</labelIds>
<q>is:unread</q>
</gmail.listAllMails>
<log level="full" />
<iterate description="MailIterator" expression="//jsonObject/messages" sequential="true" id="listUnread">
<target>
<sequence>
<log description="Iterate Logger" level="custom" separator=",**, ">
<property expression="fn:concat('GmailConfigAttachment-TaskProxy_Service ESB-MessageId:',get-property('operation','messageId'))" name="LogText" />
<property expression="//id/text()" name="Gmail-MessageId" />
</log>
<property description="emailMsgId" expression="//id/text()" name="emailMsgId" scope="default" type="STRING" />
<property description="emailSubject" expression="//headers/name[text()='Subject']/../value/text()" name="emailSubject" scope="default" type="STRING"/>
<property description="emailDate" expression="//headers/name[text()='Date']/../value/text()" name="emailDate" scope="operation" type="STRING"/>
<property description="toAddress" expression="//headers/name[text()='To']/../value/text()" name="toAddress" scope="operation" type="STRING"/>
<header expression="fn:concat('Bearer ', $ctx:uri.var.gmail.accessToken)" name="Authorization" scope="transport" />
<gmail.readMail>
<id>{$ctx:emailMsgId}</id>
</gmail.readMail>
<payloadFactory media-type="xml">
<format>
<htmlProcessEmailBody xmlns="">
<emailMessageId>$1</emailMessageId>
<emailSubject>$2</emailSubject>
<emailBody>$3</emailBody>
</htmlProcessEmailBody>
</format>
<args>
<arg evaluator="xml" expression="get-property('emailMsgId')"/>
<arg evaluator="xml" expression="get-property('emailSubject')"/>
<arg evaluator="xml" expression="//payload"/>
</args>
</payloadFactory>
<log level="custom">
<property description="emailMsgId" expression="get-property('emailMsgId')" name="emailMsgId" scope="default" type="STRING" />
<property description="emailSubject" expression="//headers/name[text()='Subject']/../value/text()" name="emailSubject" scope="default" type="STRING"/>
<property description="emailDate" expression="//headers/name[text()='Date']/../value/text()" name="emailDate" scope="default" type="STRING"/>
<property description="toAddress" expression="//headers/name[text()='To']/../value/text()" name="toAddress" scope="default" type="STRING"/>
</log>
<!-- here i need further process to download attachment from gmail -->
</sequence>
</target>
</iterate>
</inSequence>
<outSequence />
<faultSequence />
</target>
Output of ESB
So Can anyone help me to provide solution?
Awaiting for your response.
Thank you.

Finally i found the solution for my question.
Idea :
Iterating each parts then getting email attachment id which will be passed to gmail api in order to get base64 encoded format.
Code snippet:
<iterate continueParent="true" description="MailIterator" expression="//parts" id="listUnread" sequential="true">
<target>
<sequence>
<property expression="//filename[text()!=' ']" name="AttachedFileName" scope="default" type="STRING"/>
<property expression="substring-after(get-property('AttachedFileName'),'.')" name="Attachmentextension" scope="default" type="STRING"/>
<filter description="check emailSubject" regex="jpg|jpeg|png|gif|webp|tiff|tif|psd|raw|bmp|dib|heif|heic|indd|ind|jp2" source="lower-case(get-property('Attachmentextension'))">
<then>
<property description="emailAttachmentId" expression="//attachmentId/text()" name="uri.var.attachmentId" scope="default" type="STRING"/>
<log level="custom">
<property expression="get-property('AttachedFileName')" name="=====ValidAttachmentFileName===="/>
<property expression="get-property('uri.var.attachmentId')" name="====emailAttachmentId====="/>
</log>
<header expression="fn:concat('Bearer ', $ctx:uri.var.gmail.accessToken)" name="Authorization" scope="transport"/>
<call>
<endpoint>
<http method="get" uri-template="{+uri.var.gmail.apiUrl}/{+uri.var.gmail.apiVersion}/users/{+uri.var.gmail.userId}/messages/{+uri.var.id}/attachments/{+uri.var.attachmentId}"/>
</endpoint>
</call>
<property description="emailAttachment" expression="//data/text()" name="emailAttachment" scope="default" type="STRING"/>
<!-- ==========Script for Base64 url to Base64 Encoding Format ========== -->
<script language="js"><![CDATA[var log=mc.getServiceLog();
var emailAttachment = mc.getProperty('emailAttachment');
emailAttachment=emailAttachment.replaceAll("_","/").replaceAll("-","+");
mc.setProperty("modifiedemailAttachment",emailAttachment)]]></script>
<log level="custom">
<property expression="get-property('modifiedemailAttachment')" name="========modifiedemailAttachment========"/>
</log>

why CAMEL_MESSAGEPROCESSED created in SQL server db

I want to run Camel built application as stanalone. I am using maven to create standalone jar and execute route.When I run MainApp.java main method from eclipse or run using camel run then code runs as expected. But when I run using java -jar then 3 tables CAMEL_MESSAGEPROCESSED, CAMEL_MESSAGETRACED,hibernate_sequence are created into db. I do not want these tables created. Please help me find what I am doing wrong.
Camel, camel-jpa version 2.19.1
hibernate-entitymanager version 5.2.7.Final
Camel-context.xml
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:prop="http://camel.apache.org/schema/placeholder"
xmlns:camel="http://camel.apache.org/schema/spring"
xsi:schemaLocation="http://www.springframework.org/schema/beans classpath:org/springframework/beans/factory/xml/spring-beans-4.3.xsd
http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">
<bean id="properties"
class="org.apache.camel.component.properti
es.PropertiesComponent">
<property name="location" value="classpath:db.properties" />
</bean>
<bean id="bridgePropertyPlaceholder"
class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfigurer">
<property name="location" value="classpath:db.properties" />
</bean>
<camelContext id="camel5" xmlns="http://camel.apache.org/schema/spring">
<routeBuilder ref="myBuilder" />
</camelContext>
<bean id="myBuilder"
class="com.aexp.gsnt.insight.updateCentralData.InsightRouterBuilder" />
<bean id="jpa" class="org.apache.camel.component.jpa.JpaComponent">
<property name="entityManagerFactory" ref="entityManagerFactory" />
<property name="transactionManager" ref="jpaTxManager" />
</bean>
<bean id="jpaTxManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>
<bean id="entityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceUnitName" value="persistenceUnit" />
<property name="dataSource" ref="dataSource" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="databasePlatform" value="org.hibernate.dialect.SQLServerDialect" />
<property name="showSql" value="${showSQL}" />
<property name="generateDdl" value="${generateDdl}" />
</bean>
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.format_sql">${formatSQL}</prop>
</props>
</property>
</bean>
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver" />
<property name="url" value="${jdbc.url}" />
<property name="username" value="${username}" />
<property name="password" value="${password}" />
</bean>
</beans>
Persistence.xml
<?xml version="1.0" encoding="UTF-8" ?>
<persistence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"
version="2.0" xmlns="http://java.sun.com/xml/ns/persistence">
<persistence-unit name="persistenceUnit"
transaction-type="RESOURCE_LOCAL">
</persistence-unit>
</persistence>
MainApp.java
import org.springframework.context.support.ClassPathXmlApplicationContext;
public class MainApp {
private static ClassPathXmlApplicationContext classPathXmlApplicationContext;
/* Invoked from Jar */
public static void main(String args[]) throws Exception {
System.out.println("Started Central Data Extractor");
System.setProperty("jsse.enableSNIExtension", "false");
classPathXmlApplicationContext = new ClassPathXmlApplicationContext("classpath:/META-INF/spring/camel-context.xml");
classPathXmlApplicationContext.start();
}
}

BreezeJS unable to parse EDM metadata with more than one namespace?

I am writing a web application with Breeze, OData, WebAPI, and Angular (BOWA) using the integrated WebAPI ASP Identity schema. When breeze consumes the $metadata it is unable to finish and gives a 'null' error and fails. I disabled the ASP Identity to test and it works fine. Is Breeze unable to address more than a single namespace? The namespace is posted below. Are there any known work arounds to this? (OData v3)
<Schema xmlns="http://schemas.microsoft.com/ado/2009/11/edm" Namespace="Microsoft.AspNet.Identity.EntityFramework">
<ComplexType Name="IdentityUserRole">
<Property Name="UserId" Type="Edm.String"/>
<Property Name="RoleId" Type="Edm.String"/>
</ComplexType>
<EntityType Name="IdentityUserClaim">
<Key>
<PropertyRef Name="Id"/>
</Key>
<Property Name="Id" Type="Edm.Int32" Nullable="false"/>
<Property Name="UserId" Type="Edm.String"/>
<Property Name="ClaimType" Type="Edm.String"/>
<Property Name="ClaimValue" Type="Edm.String"/>
</EntityType>
<ComplexType Name="IdentityUserLogin">
<Property Name="LoginProvider" Type="Edm.String"/>
<Property Name="ProviderKey" Type="Edm.String"/>
<Property Name="UserId" Type="Edm.String"/>
</ComplexType>
</Schema>

Take a look at the section "Correcting the namespace for EdmBuilder" on this page: http://www.getbreezenow.com/documentation/odata-server

2 rows in qrtz_scheduler_state for each cluster node

I have quartz setup with clustering. Each of the quartz cluster node servers have two distinct entries in the qrtz_scheduler_state table. I can't understand it. Has anyone seen this?
More info:
Original configuration that has issue
Spring 3.0.3
Quartz 1.8.5
Weblogic 10.3.5
SQL Server 2005
New configuration that also has the issue
Spring 3.1.1
Quartz 2.1.5
Weblogic 10.3.5
SQL Server 2005
quarts-config.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:jee="http://www.springframework.org/schema/jee"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/jee
http://www.springframework.org/schema/jee/spring-jee-2.0.xsd">
<!-- Defining this bean starts the triggers -->
<bean id="scheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="triggers">
<list>
<ref bean="pendingTransactionJob" />
<ref bean="assetJob" />
</list>
</property>
<property name="dataSource" ref="wfbisDataSource" />
<property name="transactionManager" ref="wfbisTransactionManager" />
<property name="overwriteExistingJobs" value="true" />
<property name="autoStartup" value="false" />
<property name="schedulerContextAsMap">
<map>
<entry key="pendingTransactionService" value-ref="pendingTransactionService" />
<entry key="assetService" value-ref="assetService" />
</map>
</property>
<property name="quartzProperties">
<props>
<prop key="org.quartz.scheduler.instanceName">ClusteredScheduler</prop>
<prop key="org.quartz.scheduler.instanceId">AUTO</prop>
<prop key="org.quartz.threadPool.class">org.quartz.simpl.SimpleThreadPool</prop>
<prop key="org.quartz.threadPool.threadCount">6</prop>
<prop key="org.quartz.jobStore.clusterCheckinInterval">300000</prop>
<prop key="org.quartz.threadPool.threadPriority">5</prop>
<prop key="org.quartz.jobStore.misfireThreshold">60000</prop>
<prop key="org.quartz.jobStore.class">org.quartz.impl.jdbcjobstore.JobStoreTX</prop>
<prop key="org.quartz.jobStore.driverDelegateClass">org.quartz.impl.jdbcjobstore.MSSQLDelegate</prop>
<prop key="org.quartz.jobStore.tablePrefix">QRTZ_</prop>
<prop key="org.quartz.jobStore.isClustered">true</prop>
<prop key="org.quartz.jobStore.useProperties">false</prop>
</props>
</property>
</bean>
<bean id="pendingTransactionJob" class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
<property name="jobDetail" ref="pendingTransactionJobDetail" />
<property name="cronExpression" value="0 0/5 * * * ?" />
</bean>
<bean name="pendingTransactionJobDetail" class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
<property name="jobClass" value="com.wellsfargo.trust.message.ScheduledJob" />
<property name="jobDataAsMap">
<map>
<entry key="serviceName" value="pendingTransactionService" />
</map>
</property>
</bean>
<bean id="assetJob" class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
<property name="jobDetail" ref="assetJobDetail" />
<property name="cronExpression" value="0 0/5 * * * ?" />
</bean>
<bean id="assetJobDetail" class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
<property name="jobClass" value="com.wellsfargo.trust.message.ScheduledJob" />
<property name="jobDataAsMap">
<map>
<entry key="serviceName" value="assetService" />
</map>
</property>
</bean>

Delete table entries, restart the cluster and see if double entries are created again. It could happen that those entries were created but not deleted by some errors.

Resources