I am receiving data files in XML format from a public agency. The structure of the files and the notations within make me think that what I am seeing is a bulk table dump from SQL Server. Each XML file starts with a schema definition. This is followed by multiple elements, each element looking like it contains one table row of data. The structure of the files makes me think that they were created by some facility within SQL Server itself.
I have looked at the SQL Server docs and online articles, but I cannot find information on how this would be accomplished. I am thinking that if there is a built in facility to export the data in this format, there must be a built in facility to import the data back into SQL Server.
A snippet of one XML file showing the opening schema section and a couple of data elements can be found here;
sample data XML file
Am I correct in thinking that these files are SQL Server dumps? If so, who are these exported and then imported?
You can use xml functionality of SQL Server. Something like this.
Import
declare #x xml = N'--your xml here'
; with xmlnamespaces('urn:schemas-microsoft-com:sql:SqlRowSet1' as p)--required as xml has xmlns attribute
--insert mytable
select t.v.value('p:pool_idn[1]','int') pool_idn, --note p: prefix
t.v.value('p:eff_dte[1]','datetime') eff_dte,
t.v.value('p:pool_nam[1]','varchar(50)') pool_nam
--and so on
from #x.nodes('root/p:pool') t(v)
Export
select * from mytable
for xml auto, elements, root, xmlschema('urn:my:schema')
Related
I am working on a vb.net application, the management wants me to change the applications data source from SQL Server to XML.
I have a class called WebData.vb in the old application I need to somehow find a way to replace the stored procedures in it and make it read xml. So I was thinking of getting the xml structure from the returning result set of the stored procedure. I looked online and they said that for normal select statement you can do something like this:
FOR xml path ('Get_Order'),ROOT ('Get_Orders')
I am looking for something like
EXEC dbo.spMML_GET_ORDERS_FOR_EXPORT
FOR xml path ('Get_Order'),ROOT ('Get_Orders')
so now that I have the structure I can pass that data to a datatable and then return that datatable to the method.
Also if there is an alternative way in creating a XML stored procedure please let me know thanks coders.
Assuming you can't modify the stored proc (due to other dependencies or some other reason) to have the SELECT within the proc have the FOR XML syntax, you can use INSERT/EXEC to insert the results of the stored proc into a temp table or table variable, then apply your FOR XML onto a query of those results.
Something like this should work:
DECLARE #Data TABLE (...) -- Define table to match results of stored proc
INSERT #Data
EXEC dbo.spMML_GET_ORDERS_FOR_EXPORT
SELECT * FROM #Data FOR xml path ('Get_Order'),ROOT ('Get_Orders')
There are a few methods, one adding namespaces using WITH XMLNAMESPACES(<STRING> AS <NAMESPACE string>). XMLNAMESPACES can embed appropriate XML markers to your tables for use with other applications (which hopefully is a factor here), making documentation a little easier.
Depending on your application use, you can use FOR XML {RAW, PATH, AUTO, or EXPLICIT} in your query, as well as XQUERY methods...but for your needs, stick to the simpler method like XML PATH or XML AUTO.
XML PATH is very flexible, however you lose the straightforward identification of the column datatypes.
XMLNAMESPACE
WITH XMLNAMESPACES('dbo.MyTableName' AS SQL)
SELECT DENSE_RANK() OVER (ORDER BY Name ASC) AS 'Management_ID'
, Name AS [Name]
, Began AS [Team/#Began]
, Ended AS [Team/#Ended]
, Team AS [Team]
, [Role]
FROM dbo.SSIS_Owners
FOR XML PATH, ELEMENTS, ROOT('SQL')
XML AUTO
Because you might want to return to the database, I suggest using XML AUTO with XMLSCHEMA, where the sql datatypes are kept in the XML.
SELECT DENSE_RANK() OVER (ORDER BY Name ASC) AS 'Management_ID'
, Name AS [Name]
, Began AS [Team/#Began]
, Ended AS [Team/#Ended]
, Team AS [Team]
, [Role]
FROM dbo.SSIS_Owners
FOR XML AUTO, ELEMENTS, XMLSCHEMA('SSIS_Owners')
Downside is XMLNAMESPACES is not an option, but you can get around this through solutions like XML SCHEMA COLLECTIONS or in the query itself as I showed.
You can also just use XML PATH directly without the namespace, but again, that depends on your application use as you are transforming everything to XML files.
Also note how I defined the embedded attributes. A learning point here, but think about the query in the same order that the XML would appear. That is why I defined the variable attributes first before I then stated what the text for that node was.
Lastly, I think you'll find Paparazzi has a question on this topic that covers quite. TSQL FOR XML PATH Attribute On , Type
I'm trying to use SSIS to extract XML representation of a query result set to a text file. My query is currently successfully extracting the exact XML output I need when I run it in SSMS. I've tried every trick I can find to use this result set in a SSIS package to create a file.
Using a dataflow to port a OLE Source to a Flat file doesn't work because the output of a XML query is treated as TEXT and SSIS can't push TEXT, NTEXT or IMAGE to a file destination.
I've tried to then Execute SQL Task to fill a user variable and then use a Script Task (written using C#) to write the contents of this user variable to a file output, but the user variable is always empty. I don't know, but I suspect this is, again, because the XML is treated as TEXT or IMAGE and the user variable doesn't handle this.
The query is in this form:
SELECT *
FROM dataTable
WHERE dataTable.FIELD = 'Value'
FOR XML AUTO, ROOT('RootVal')
The resulting dataset is well formed XML, but I can't figure out how to get it from result set to file.
It's a relatively easy task for me to write a console app to do this in C# 4.0, but restrictions require me to at least prove it CAN'T be done with SSIS before I write the console app and a scheduler.
Sorry to spoil, but there's an SSIS option for you: Export Column Transformation.
I defined an OLE DB query with
SELECT
*
FROM
(
SELECT * FROM dbo.spt_values FOR XML AUTO, ROOT('RootVal')
) D (xml_node)
CROSS APPLY
(
SELECT 'C:\ssisdata\so_xmlExtract.xml'
) F (fileName)
This results in 1 row and 2 columns in the dataflow. I then attached the Export Column Transformation and wired it up with xml_node as Extract Column and fileName as the File Path Column
Mostly truncated results follow
<RootVal>
<dbo.spt_values name="rpc" number="1" type="A " status="0"/>
<dbo.spt_values name="dist" number="8" type="A " status="0"/>
<dbo.spt_values name="deferred" number="8192" type="V " low="0" high="1" status="0"/>
</RootVal>
A more detailed answer, with pictures, is available on this Q&A Export Varbinary(max) column with ssis
BillInKC's answer is the best I've ever seen, but SQL can be simplified (no need for cross apply):
SELECT X.*, 'output.xml' AS filename
FROM (SELECT * FROM #t FOR XML PATH('item'), ROOT('itemList')) AS X (xml_node)
It will output the same structure:
xml_node filename
-------------------------------------------------- ----------
<itemList><item><num>1000</num></item></itemlist> output.xml
(1 row(s) affected)
I am working on an import script, the idea being to import multiple workbooks into one table. I have made progress, so I am able to import one workbook successfully into my table. What I want to do is create a query that will loop a folder read the file names and import the data into my database in Microsoft SQL Server Management Studio.
--Creating the TABLE--
CREATE TABLE BrewinDolphinHoldings
(
recordID INT AUTO_NUMBER
FUNDNAME VARCHAR(25),
SEDOL VARCHAR(7),
ISIN VARCHAR(11),
NAME VARCHAR(20),
WEIGHT INTEGER(3)
)
constraint[pk_recordID]PRIMARYKEY
(
[recordID] ASC
)
INSERT INTO BrewinDolphinHoldings
VALUES
("HoldingsData', GB125451241, DavidsHoldings, 22)
--SELECTING THE SHEET--
SELECT/UPDATE? *
FROM OPENROWSET('Microsoft.JET.OLEDB.4.0',
'Excel 8.0;Database=X:\CC\sql\DEMO\SpreadsheetName + '.xlsx',
'SELECT * FROM [Sheet1$]') AS HoldingsData
So essentially my question is, I want to create a loop a loop that will read the file name in a directory, and the import will read that name every time it loops and import the relevant spreadsheets? so,for example:
DECLARE SpreadsheetName as STRING
DECLARE FileExtension as '.xlsx'
FOR EACH ITEM IN DIRECTORY
X=1
Y=MAX
FILENAME THAT LOOP READS = SpreadsheetName
SELECT * FROM
OPENROWSET('Microsoft.JET.OLEDB.12.0',
'Excel 8.0;Database=X:\CC\sql\DEMO\SpreadsheetName + fileExtension.xls
END LOOP
So, I'm thinking maybe something like this? Although I don't know if the loop will overwrite my database? maybe instead of UPDATE I should use INSERT?
I don't want to use SSIS, preferably a query, although if anyone can recommend anything I could look into, or, help me with this loop It would greatly help
I'm open to new ideas from you guys, so if anyone can try and fix my code, or give me a few examples of imports for multiple excel sheets, would be greatly appreciated!
I'm new to SQL Server, I do have some previous programming experience!
Thanks!
You can use bcp to do what you are talking about to import any type of delimited text file, such as csv or text tab delimited. If it is possible generate/save the spreadsheets as csv and use this method. See these links.
Import Multiple CSV Files to SQL Server from a Folder
http://www.databasejournal.com/features/mssql/article.php/3325701/Import-multiple-Files-to-SQL-Server-using-T-SQL.htm
If it has to be excel, then you can't use bcp, but these should still help you with the logic for the loops on the file folders. I have never used the excel openrowset before, but if you have it working like you said, it should be able to insert in just the same. You can still use the xp_cmdshell/xp_dirtree to look at the files and generate the path even though you can't import it with bcp.
How to list files inside a folder with SQL Server
I would then say it would be easiest to do a insert from a select statement from the openrowset to put it into the table.
http://www.w3schools.com/sql/sql_insert_into_select.asp
Make sure xp_cmdshell is enabled on your sql server instance as well.
https://msdn.microsoft.com/en-us/library/ms190693(v=sql.110).aspx
I have a set of xml files that I want to parse the data of and import in to a sql server 2012 database. The provided xml files will be validated against a schema.
I am looking as to what is the best method of doing this is. I have found this: http://msdn.microsoft.com/en-us/library/ms171878.aspx
I am wondering if this is the best way or if there are others?
You have several options:
SSIS XML Source. This does not validate against the schema. If you want to detect and properly handle invalid XML files, create a script task to validate the schema in C#.
Parse the XML in a stored procedure.
Insert the entire XML file in one column. Depending on your schema validation requirements, you can use an untyped or typed XML column. (Or both)
Parse the XML using XPath functions. This is actually very fast.
INSERT INTO SomeTable (Column1, Column2, Column3)
SELECT
YourXmlColumn.value('(/root/col1)[1]','int'),
YourXmlColumn.value('(/root/col2)[1]','nvarchar(10)'),
YourXmlColumn.value('(/root/col3)[1]','nvarchar(2000)'),
YourXmlColumn.value('(/root/col4)[1]','datetime2(0)')
FROM YourXmlTable
How can I import CSV file data into SQL Server 2000 table? I need to insert data from CSV file to table twice a day. Table has more then 20 fields but I only need to insert value into 6 fields.
i face same problem before i can suggest start reading here. The author covers:"This is very common request recently – How to import CSV file into SQL Server? How to load CSV file into SQL Server Database Table? How to load comma delimited file into SQL Server? Let us see the solution in quick steps."
I need to insert data from CSV file to table twice a day.
Use DTS to perform the import, then schedule it.
For SQL 2000, I would use DTS. You can then shedule this as a job when your happy with it.
Below is a good Microsoft link explaining how to use it.
Data Transformation Services (DTS)
You describe two distinct problems:
the CSV import, and
the extraction of data into only those 6 fields.
So break your solution down into two steps:
import the CSV into a raw staging table, and
then insert into your six 'live' fields from that staging table.
There is a function for the first part, called BULK INSERT, the syntax looks like this:
BULK INSERT target_staging_table_in_database
FROM 'C:\Path_to\CSV_file.csv'
WITH
(
DATAFILETYPE = 'CHAR'
,FIRSTROW = 2
,FIELDTERMINATOR = ','
,ROWTERMINATOR = '\n'
);
Adjust to taste, and consult the docs for more options. You might also want to TRUNCATE or DELETE FROM your staging table before doing the bulk insert so you don't have any old data in there.
Once you get the information into the database, doing an UPDATE or INSERT into those six fields should be straightforward.
You can make of use SQL Server Integration services(SSIS). It's jusy one time task to create the Package. Next time onwards just run that package.
You can also try Bulk Insert as daniel explained.
You can also try Import export wizard in SQL Server 2000.