I have a Problem with saving the XML nodes in different Tables.
I want to save the Listnodes in one Table(ListsSet) and the Columnsnodes in another Table(ColumnsSet) which is referencing the ListTable. But in the Columnsnode is also a DependentLookupField childnode. I want to save the value from the DependentLookupField in a third Table(DependentLookupFieldsSet), which is referencing the ColumnTable.
My XML:
<Lists>
<List title="test">
<ListUrl>fsasa</ListUrl>
<ListTitle>gsdfgsg</ListTitle>
<ListDesc>jasdh</ListDesc>
<Columns>
<Column action="modify">
<InternalName>Title</InternalName>
<DisplayNameOrigin>Titel</DisplayNameOrigin>
<DisplayName>Name</DisplayName>
</Column>
<Column action="new">
<Required>true</Required>
<FieldType>Choice</FieldType>
<InternalName>anrede</InternalName>
<DisplayName>Anrede</DisplayName>
</Column>
<Column action="add" type="sitecolumn">
<Required>true</Required>
<InternalName>Bank</InternalName>
<DisplayName>Bank</DisplayName>
<MultipleValues>0</MultipleValues>
<DependentLookupFields>
<DependentLookupField internalName="Title">My Value</DependentLookupField>
</DependentLookupFields>
</Column>
</Columns>
</List>
<List>
.
.
.
etc
</List>
</Lists>
My Code of the Stored Procedure:
INSERT INTO ListsSet
SELECT
List.value('ListUrl[1]','NVARCHAR(100)') AS ListURL,
List.value('ListTitle[1]','NVARCHAR(100)') AS ListTitle,
List.value('ListDesc[1]','NVARCHAR(100)') AS ListDesc,
FROM
#xml.nodes('/Lists/List')AS TEMPTABLE(List)
INSERT INTO ColumnsSet
SELECT
l.ListsID,
c.value('FieldType[1]','NVARCHAR(100)') AS FieldType,
c.value('InternalName[1]','NVARCHAR(100)') AS InternalName,
c.value('DisplayName[1]','NVARCHAR(100)') AS DisplayName,
c.value('Required[1]','NVARCHAR(100)') AS Required,
c.value('DisplayNameOrigin[1]','NVARCHAR(100)') AS DisplayNameOrigin,
c.value('MultipleValues[1]','NVARCHAR(100)') AS MultipleValues,
FROM
ListsSet AS l
cross apply #xml.nodes('/Lists/List/Columns/Column[../../ListUrl=sql:column("l.ListUrl")]') AS TEMPTABLE(c)
INSERT INTO DependentLookupFieldsSet
SELECT
c.ColumnsID,
d.value('DependentLookupField[1]','NVARCHAR(100)') AS DependetLookupField
FROM
ColumnsSet AS c
cross apply #xml.nodes('/Lists/List/Columns/Column/DependentLookupFields/DependentLookupField[../../InternalName=sql:column("c.InternalName")]') AS TEMPTABLE(d)
The Code for saving the Listsnodes and Columnsnode is working fine. But the Code for the DependentLookupField don't save the value. In the Table "DependentLookupFieldsSet" I get the Rows which are referencing the ColumsTable(that is working), but the saved value is always null in each row.
Any help is appreciated!
Related
I stored my data in XML format in SQL server, in this way
**<Column Name="GROSS" DataType="float" Value="939760" />**
but somehow one column name (GROSS) in my XML data is stored twice, now I want to remove/rename one of them.
Below are screenshots of my database view:
Table view
XML view
This it what I tried, but it only changed the value, it did not rename the column name.
update Aquara7bc772839.EmpTransaction set TransactionFieldDetails.modify('replace value of (/PayDetails/Column[#Name="LEV_ENCASHRATE"]/#Value)[1] with "796.00"') WHERE Id = 276620;
I have highlighted my column names in above image link please check
I want to remove or rename one column.
You can use the XML.modify() method with a delete node instruction...
declare #EmpTransaction table (
Id int not null,
TransactionFieldDetails xml
);
insert #EmpTransaction values (
276620,
N'<PayDetails>
<Column Name="GROSS" DataType="float" Value="939760" />
<Column Name="GROSS" DataType="float" Value="939760" />
</PayDetails>'
);
update #EmpTransaction
set TransactionFieldDetails.modify('delete /PayDetails/Column[#Name="GROSS"][2]')
where Id = 276620;
select * from #EmpTransaction;
Which gives you...
<PayDetails>
<Column Name="GROSS" DataType="float" Value="939760" />
</PayDetails>
Note that the node index is 1-based, i.e.: Column[#Name="GROSS"][1] would remove the first GROSS node, Column[#Name="GROSS"][2] removes the second GROSS node.
As to how you got two GROSS values in the first place ... whatever created the XML for you probably lists the GROSS column twice.
For your next question: Please do not poste pictures. Best was, to provide a MCVE (a stand-alone sample to reproduce your issue).
You can try something along this:
This is such a stand-alone sample
DECLARE #mockupTable TABLE(ID INT IDENTITY, YourXml XML);
INSERT INTO #mockupTable VALUES
(N'<PayDetails>
<Column Name="blah" DataType="string" Value="Some blah" />
<Column Name="GROSS" DataType="float" Value="1.1" />
<Column Name="Another" DataType="string" Value="One more" />
<Column Name="GROSS" DataType="float" Value="1.2" />
</PayDetails>')
,(N'<PayDetails>
<Column Name="blah" DataType="string" Value="Some blah" />
<Column Name="GROSS" DataType="float" Value="2.0" />
<Column Name="Another" DataType="string" Value="One more" />
<Column Name="GROSS" DataType="float" Value="2.0" />
</PayDetails>');
--This query will first check, if there are different values for GROSS within one XML. This might be a reason to look a bit closer before deleting them.
SELECT t.ID
,t.YourXml
FROM #mockupTable t
WHERE t.YourXml.value('count(distinct-values(/PayDetails/Column[#Name="GROSS"]/#Value))','int')>1;
--And this statement will return each <Column> just once per name (always the first of its kind)
UPDATE #mockupTable SET YourXml=YourXml.query(N'
<PayDetails>
{
for $elmt in distinct-values(/PayDetails/Column/#Name)
return /PayDetails/Column[#Name=$elmt][1]
}
</PayDetails>
');
--Check the output
SELECT * FROM #mockupTable
The idea in short:
With the Xpath /PayDetails/Column[#Name="GROSS"] we are reducing the observed set to columns, where the attribute Name equals GROSS. distinct-values() is a XQuery-function returning each value in a list just once. So we can use count() to check, if there are differing values for GROSS within one XML.
The UPDATE uses XQuery's FLWOR abilities. We use again distinct-values to get all values for Name within <Column>. Then we return just the first (see the [1]) for each name.
UPDATE: Check for doubled elements
With this query you can run through your whole table to search for any non-unique column name per XML:
SELECT t.YourXml.query(N'
for $elmt in distinct-values(/PayDetails/Column/#Name)
return <NameCount Name="{$elmt}" Count="{count(/PayDetails/Column[#Name=$elmt])}"/>
').query('/NameCount[#Count>1]')
FROM #mockupTable t;
I'm trying to import an XML document using SSIS that is exported using a Microsoft Office Excel format with 17 columns. I have an XML task that is removing the multiple namespaces, but now I have a document that is formatted like the sample code below. I can load each Cell into it's own record in the database, but since there are no tags inside the row or cell sections, I have a database table with one column for each Cell. I don't have any row numbers so not sure if I can do some sort of pre-sort, or if I'm going to have to do a bunch of SQL based on the row number being a multiple of 17 and STUFF FOR XML PATH the rows via temp tables which seems messy.
<Worksheet>
<Table>
<Column/>
<Row>
<Cell StyleID="s62">
<Data Type="String">City</Data>
</Cell>
<Cell StyleID="s62">
<Data Type="String">State</Data>
</Cell>
<Cell StyleID="s62">
<Data Type="String">Zip</Data>
</Cell>
</Row>
</Table>
</Worksheet>
I do not know, if I understood this correctly...
Assuming your XML is in a variable DECLARE #xml XML you might get the Cells by calling them with their position within the tree
SELECT R.value('Cell[1]/Data[1]','varchar(max)') AS City
,R.value('Cell[2]/Data[1]','varchar(max)') AS State
,R.value('Cell[3]/Data[1]','varchar(max)') AS Zip
--add more
FROM #xml.nodes('/Worksheet/Table/Row') AS A(R)
or you might think about pivot like this
SELECT p.*
FROM
(
SELECT 'Cell_' + CAST(ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS VARCHAR(10)) AS ColumnName
,Cell.value('Data[1]','varchar(max)') AS Data
FROM #xml.nodes('/Worksheet/Table/Row/Cell') AS A(Cell)
) AS tbl
PIVOT
(
MIN(Data) FOR ColumnName IN(Cell_1,Cell_2,Cell_3,Cell_4 /*add as many as you need*/)
) AS p;
The second could be transfered into dynamic SQL to analyse the existing Cell nodes, get their names and number and return a resultset fitting to the XML's data...
The last you would need, if the XML's data is not the same in all calls...
The only solution I found was to open the file with script task using interop and save it in excel format.
I need to export data from SQL Server 2012 based on a view. While testing the export for a downstream system, I was manually extracting the data out of the table that the view is based on and the BIT data type columns were reporting as 1/0.
However, once I setup the view against the table, I noticed that the BIT data type columns reported as TRUE/FALSE. This happens whether I perform a select against the view or export from it.
Why does this happen and how can I maintain the same results in the view as the data table (1/0)?
The bit data type is interpreted by clients differently. SSMS, will report back a 1 or 0 for a bit while the same 1/0 is interpreted by an SSIS's Data Flow as True or False.
Whether the source is a table or a view makes no matter for SSIS unless you explicitly change the data type.
For setup, I created 2 tables and a view
CREATE TABLE dbo.BaseTable
(
SomeBit bit NOT NULL
, RowDescription varchar(50) NOT NULL
);
CREATE TABLE dbo.TargetTable
(
SomeBit bit NOT NULL
, RowDescription varchar(50) NOT NULL
, SourcePackage nvarchar(100) NOT NULL
);
GO
CREATE VIEW dbo.MyView
AS
SELECT
BT.SomeBit
, BT.RowDescription
FROM
dbo.BaseTable AS BT;
GO
INSERT INTO
dbo.BaseTable
(
SomeBit
, RowDescription
)
VALUES
(CAST(0 AS bit), 'Falsification')
, (CAST(1 AS bit), 'True dat');
GO
At this point, if I use SSMS and query either dbo.BaseTable or dbo.MyView, I will get back a 1 and 0. But again, these are just artifacts of presentation. In C, 0 is false and any numeric value that isn't 0 is true. Excel will present it as FALSE and TRUE. Every client will interpret the value into whatever the local representation of a boolean value is. SSIS chose True and False.
I built out a simple package that pulls data from BaseTable or MyView and writes it to a text file and a table.
The basic control flow looks thus
The data flow looks complex but it's not.
I select from either my table or view, add a description for my target table, use a multicast so I can send the same data to multiple destinations and then write to a file and table.
If I query SSMS for my sources and destinations, you'll see that the destination libraries handle the translation between the local and foreign representation of the data type.
There is no such translation available for a flat file because there's no "standard" for the representation of a boolean. I might like Y/N. Even so, the
I tried a number of things to coerce a 1/0 to be written to the flat file. I set my data types to
Boolean DT_BOOL
Single byte signed int DT_I1
Four byte signed int DT_I4
String DT_STR
but it never mattered (which actually seems odd given how persnickety SSIS is about data types) --- my output was always the same
False,Falsification
True,True dat
Ultimately, if I wanted a 0 or a 1 in that output file, I needed to change my data type: either in the source query with an explicit cast or through a Derived Column component using the ternary operator SomeBit ? (DT_I1)1 : (DT_I1)0. Use DT_I1/I2/I4/I8 as you see fit
Fun trivia note: if you chose to use the Data Conversion component you're going to get 0 for False, -1 for True or if you use a lazy cast in the Derived Component (DT_I1) SomeBit It seems they follow the C interpretation of boolean values.
Biml it
No need to take my word for it. Using the above table definitions and population of values, if you install the free addon BIDS Helper you can generate the same code for any version of SSIS.
After installing BIDS Helper, right click on an SSIS project and in the context menu, select Add Biml file. Replace the contents of that file with the below code; save and then right-click to generate a new package.
You will need to edit the values for the Flat File Connection to point to valid locations as well as point the ole db connection string to wherever you spin up your tables.
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<FlatFileConnection FilePath="C:\ssisdata\so_29244868.table.csv" FileFormat="FFF_table" Name="FF_Table" />
<FlatFileConnection FilePath="C:\ssisdata\so_29244868.view.csv" FileFormat="FFF_table" Name="FF_View" />
<OleDbConnection Name="CM_OLE" ConnectionString="Data Source=localhost\dev2014;Initial Catalog=tempdb;Provider=SQLNCLI11.0;Integrated Security=SSPI;" />
</Connections>
<FileFormats>
<FlatFileFormat
Name="FFF_table" IsUnicode="false" CodePage="1252"
FlatFileType="RaggedRight">
<Columns>
<Column Name="SomeBit" DataType="Boolean" Delimiter="," />
<Column Name="RowDescription" DataType="AnsiString" Length="50" Delimiter="CRLF"/>
</Columns>
</FlatFileFormat>
</FileFormats>
<Packages>
<Package ConstraintMode="Parallel" Name="so_29244868">
<Tasks>
<Dataflow Name="DFT Table example">
<Transformations>
<OleDbSource ConnectionName="CM_OLE" Name="OLE_SRC dbo_BaseTable">
<ExternalTableInput Table="dbo.BaseTable" />
</OleDbSource>
<DerivedColumns Name="DER Package name">
<Columns>
<Column DataType="String" Name="SourcePackage" Length="100">"DFT Table example"</Column>
</Columns>
</DerivedColumns>
<Multicast Name="MC Dupe">
<OutputPaths>
<OutputPath Name="FF" />
<OutputPath Name="Table" />
</OutputPaths>
</Multicast>
<FlatFileDestination ConnectionName="FF_Table" Name="FF_DST table">
<InputPath OutputPathName="MC Dupe.FF" />
</FlatFileDestination>
<OleDbDestination
ConnectionName="CM_OLE"
Name="OLE_DST Table"
TableLock="false">
<InputPath OutputPathName="MC Dupe.Table" />
<ExternalTableOutput Table="[dbo].[TargetTable]"></ExternalTableOutput>
</OleDbDestination>
</Transformations>
</Dataflow>
<Dataflow Name="DFT View example">
<Transformations>
<OleDbSource ConnectionName="CM_OLE" Name="OLE_SRC dbo_MyView">
<ExternalTableInput Table="dbo.MyView" />
</OleDbSource>
<DerivedColumns Name="DER Package name">
<Columns>
<Column DataType="String" Name="SourcePackage" Length="100">"DFT View example"</Column>
</Columns>
</DerivedColumns>
<Multicast Name="MC Dupe">
<OutputPaths>
<OutputPath Name="FF" />
<OutputPath Name="Table" />
</OutputPaths>
</Multicast>
<FlatFileDestination ConnectionName="FF_View" Name="FF_DST view">
<InputPath OutputPathName="MC Dupe.FF" />
</FlatFileDestination>
<OleDbDestination
ConnectionName="CM_OLE"
Name="OLE_DST view"
TableLock="false"
>
<InputPath OutputPathName="MC Dupe.Table" />
<ExternalTableOutput Table="[dbo].[TargetTable]"></ExternalTableOutput>
</OleDbDestination>
</Transformations>
</Dataflow>
</Tasks>
</Package>
</Packages>
</Biml>
I've run into the same problem using Entity Framework.
Try casting the bit field to a bit.
I have the XML data below that represents one row but can't seem to get this imported into sql server without completely rewriting the xml file to look like the second block of code at the very bottom. I have thousands of small xml files in the first format I need to start processing and importing into a sql table. Ideally if I don't use some custom scripting to rewrite the xml I'm thinking I could import into a temp table and use pivot or transposing to get the attributes into cells in a row.
<ping>
<feed Scale="4.0" resolution="67.58859099656746">
<beta name="my_misc_beta" totalRecords="1">
<row>
<column>
<key>CUSTOMER ID</key>
<value>123456</value>
</column>
<column>
<key>CUSTOMER NAME</key>
<value>Johnys Bike Shop</value>
</column>
<column>
<key>REGION NAME</key>
<value>Cool Area</value>
</column>
<column>
<key>CUSTOMER CATEGORY</key>
<value>Bike Shop</value>
</column>
<column>
<key>CUSTOMER DESCRIPTION</key>
<value>coolest bike shop</value>
</column>
<column>
<key>CUSTOMER STATUS</key>
<value>Current</value>
</column>
<column>
<key>CUSTOMER CONTACT</key>
<value>johnny#bikeshop.net</value>
</column>
</row>
</beta>
</feed>
</ping>
This xml below easily imports into sql server using ssis and looping through the directory. But I had to hand rewrite this by hand for testing. Is there a way with maybe c# or another language to take the inside text and write them to element tags etc in ssis. Compare the two xml docs and you'll see that the above code will not import diectly to a row.
<ping>
<feed Scale="4.0" resolution="67.58859099656746">
<beta name="my_misc_beta" totalRecords="1">
<row>
<column>
<CUSTOMER_ID>123456</CUSTOMER_ID>
<CUSTOMER_NAME>Johnys Pedal Shop</CUSTOMER_NAME>
<REGION_NAME>Cool Area</REGION_NAME>
<CUSTOMER_CATEGORY>Bike Shop</CUSTOMER_CATEGORY>
<CUSTOMER_DESCRIPTION>coolest bike shop</CUSTOMER_DESCRIPTION>
<CUSTOMER_STATUS>Current</CUSTOMER_STATUS>
<CUSTOMER_CONTACT>johnny#bikeshop.net</CUSTOMER_CONTACT>
</column>
</row>
</beta>
</feed>
</ping>
Don't know anything about what XML capabilities there are in SSIS but you can do what you want using a regular query.
First you need to shred the XML on the row node using nodes() Method (xml Data Type) to get one row in the resul for each row in the XML.
Then you use value() Method (xml Data Type) to extract the column value you want. You could fetch the values by column number but it is safer to check the key value in a predicate before fetching the value.
select T.X.value('(column[key/text() = "CUSTOMER ID"]/value/text())[1]', 'int') as CustomerID,
T.X.value('(column[key/text() = "CUSTOMER NAME"]/value/text())[1]', 'nvarchar(100)') as CustomerName,
T.X.value('(column[key/text() = "REGION NAME"]/value/text())[1]', 'nvarchar(100)') as RegionName,
T.X.value('(column[key/text() = "CUSTOMER CATEGORY"]/value/text())[1]', 'nvarchar(100)') as CustomerCategory,
T.X.value('(column[key/text() = "CUSTOMER DESCRIPTION"]/value/text())[1]', 'nvarchar(100)') as CustomerDescription,
T.X.value('(column[key/text() = "CUSTOMER STATUS"]/value/text())[1]', 'nvarchar(100)') as CustomerStatus,
T.X.value('(column[key/text() = "CUSTOMER CONTACT"]/value/text())[1]', 'nvarchar(100)') as CustomerContact
from #XML.nodes('/ping/feed/beta/row') as T(X)
SQL Fiddle
Trying to bulk insert lots of rows into a table.
My SQL statement:
INSERT INTO [NCAATreasureHunt-dev].dbo.CatalinaCodes(Code)
SELECT (Code)
FROM OPENROWSET(BULK 'C:\Users\Administrator\Desktop\NCAATreasureHunt\10RDM.TXT',
FORMATFILE='C:\Users\Administrator\Desktop\NCAATreasureHunt\formatfile.xml') as t1;
10RDM.TXT:
DJKF61TGN7
Q9TVM16Z6Z
X44T4169FN
JQ2PT1ZXZK
C7NW71QPNG
SFJRR1FWKZ
TYZJW1ZPFY
9MR3M1J3N5
QJ6R217JTK
TVJVW19TYT
formatfile.xml
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="C1" xsi:type="CharTerm" TERMINATOR="\r\n"/>
</RECORD>
<ROW>
<COLUMN SOURCE="C1" NAME="Code" xsi:type="SQLNVARCHAR" />
</ROW>
</BCPFORMAT>
This is the error I'm getting:
Cannot insert the value NULL into column 'Claimed', column does not allow nulls. INSERT fails.
I'm trying to skip the Claimed column. What am I doing wrong in my format file?
See if this answer helps.
With an XML format file, you cannot skip a column when you are
importing directly into a table by using a bcp command or a BULK
INSERT statement. However, you can import into all but the last column
of a table. If you have to skip any but the last column, you must
create a view of the target table that contains only the columns
contained in the data file. Then, you can bulk import data from that
file into the view.
To use an XML format file to skip a table column by using
OPENROWSET(BULK...), you have to provide explicit list of columns in
the select list and also in the target table, as follows:
INSERT ... SELECT FROM OPENROWSET(BULK...)