I'm trying insert a large xml document (about 10MB) into a Sql Server 2008 table, the XML document is built at run-time.
Is there a better way to perform this insert. I'm using a simple insert command with one parameter of type string, but it doesn't work. In the table, the field is showing a NULL value.
Is there a way to do this with a single insert?
Try to use BULK INSERT expression to increase performance.
http://msdn.microsoft.com/ru-ru/library/ms188365.aspx
Related
I store csv strings in a datatable. I later need to create a temporary table from that csv, but BULK INSERT only offers a filename as datasource.
Is there any possibility to import from a string?
Thank you and regards
Gabriel
In general, it is not desirable to store unnormalized CSV data in a SQL Server table. It makes it very hard to query and work with the data. That being said, sometimes we have to live with bad design decisions. That having been said, you could try writing your CSV column to file. From the query menu of SSMS choose SQLCMD mode, and then type the following:
:OUT c:\path\to\your\file.csv
SET NOCOUNT ON;SELECT csv_column FROM dbo.yourTable
Now that you have a bona fide CSV file, you should be able to use BULK INSERT. Note that I have assumed here that the CSV data which you want to import is contained within a single column csv_column, and that the data is well formed (e.g. each record has the same number of commas etc.).
I have a table with a VARBINARY(MAX) field (SQL Server 2008 with FILESTREAM)
My requirement is that when I go to deploy to production, I can only supply my IT team with a group of SQL scripts to be executed in a certain order. A new table I am making in production has this VARBINARY(MAX) field. Usually with new tables, I will script out the CREATE TABLE script. And, if I have data I need to go with it, I will then script out the INSERT scripts. Not too complicated.
But with VARBINARY(MAX), the Stored Procedure I was using to generate the INSERT statements fails on that table. I tried selecting that field, printing it, copying it, converting to hex, etc. The main issue I have with that is that it doesn't select all the data in the field. I do a check DATALENGTH([FileColumn]) and if the source row contains 1,004,382 bytes, the max I can get the copied or selected data when inserting again is 8000. So basically it is truncated (i.e. invalid) data.....
How can I do this better? I tried Googling this like crazy but I must be missing something. Remember, I can't access the filesystem. This has to be all scripted.
If this is a one time (or seldom) thing to do, you can try scripting the data out from the SSMS Wizard as described here:
http://sqlblog.com/blogs/eric_johnson/archive/2010/03/08/script-data-in-sql-server-2008.aspx
Or, if you need to do this frequently and want to automate it, you can try the SQL# SQLCLR library (which I wrote and while most of it is free, the function you need here is not). The function to do this is DB_DumpData and it also generates INSERT statements.
But again, if this is a one time or infrequent task, then try the data export wizard that is built into Management Studio. That should allow you to then create the SQL script that you can run in Production. I just tested this on a table with a VARBINARY(MAX) field containing 3,365,964 bytes of data and the Generate Scripts wizard generated an INSERT statement with the entire hex string of 6.73 million characters for that one value.
UPDATE:
Another quick and easy way to do this in a manner that would allow you to copy / paste the entire INSERT statement into a SQL script and not have to bother with BCP or SSMS Export Wizard is to just convert the value to XML. First you would CONVERT the VARBINARY to VARCHAR(MAX) using the optional style of "1" which gives you a hex string starting with "0x". Once you have the hex string of the binary data you can concatenate that into an INSERT statement and that entire thing, when converted to XML, can contain the entire VARBINARY field. See the following example:
DECLARE #Binary VARBINARY(MAX) = CONVERT(VARBINARY(MAX),
REPLICATE(
CONVERT(NVARCHAR(MAX), 'test string'),
100000)
)
SELECT 'INSERT INTO dbo.TableName (ColumnName) VALUES ('+
CONVERT(VARCHAR(MAX), #Binary, 1) + ')' AS [Insert]
FOR XML RAW;
Don't script from SSMS
bcp the data out/in, or use something like SSMS tools to generate INSERT statements
It more than a bit messed up, but in the past and on the web I've seen this done using a base64-encoded string. You use an xml value to wrap the string and from there you can convert it to a varbinary. Here's an example:
http://blogs.msdn.com/b/sqltips/archive/2008/06/30/converting-from-base64-to-varbinary-and-vice-versa.aspx
I can't speak personally to how effective or performant this is, though, especially for large values. Because it is at best an ugly hack, I'd tuck it away inside a UDF somewhere, so that if a better method is found you can update it easily.
I have never tried anything like this before, but from the documentation for SQL Server 2008 R2, it sounds like using SUBSTRING will work to get the entire varbinary value, although you may have to work with it in chunks, using UPDATEs with the .WRITE clause to append the data.
Updating Large Value Data Types
Use the .WRITE (expression, #Offset, #Length) clause to perform a partial or full update of varchar(max), nvarchar(max), and varbinary(max) data types. For example, a partial update of a varchar(max) column might delete or modify only the first 200 characters of the column, whereas a full update would delete or modify all the data in the column.
For best performance, we recommend that data be inserted or updated in chunk sizes that are multiples of 8040 bytes.
Hope this helps.
I have couple hundred rows of data of about 100 bytes each and need the field values validated against fairly large (milions of rows) table in SQL server. The query itself on SQL is very quick, but running single query per row from the script is not slow, probably because of the connection set-up and teardown. I can't add any stored procs to the server. Is there any way how to pass a dataset/table to a query from the script and return results rather then iterate the dataset row-by-row? Of course PowerShell code samples would be welcome ;)
There won't be a way via powershell (or any other method I am aware of) as SQL Server can't accept a table as an input parameter in a query. Any method you use for this will at some point be passing a row/specific values to SQL.
What you COULD do is create a table from your array and compare that temp table to your large table for validation. Determining if this is faster will need some testing, though.
I've got a column in a SQL Server table that is type XML. I want to change multiple values in the XML in a stored proc. I've already found the XML.Modify(...) command and have it working for a single value. Can I change multiple values in the XML in one update command or do I need to do it as multuple update commands?
You will need to use multiple (xmlcol).modify calls - you cannot update multiple bits at once, as far as I know.
Is there a direct route that is pretty straight forward? (i.e. can SQL Server read XML)
Or, is it best to parse the XML and just transfer it in the usual way via ADO.Net either as individual rows or perhaps a batch update?
I realize there may be solutions that involve large complex stored procs--while I'm not entirely opposed to this, I tend to prefer to have most of my business logic in the C# code. I have seen a solution using SQLXMLBulkLoad, but it seemed to require fairly complex SQL code.
For reference, I'll be working with about 100 rows at a time with about 50 small pieces of data for each (strings and ints). This will eventually become a daily batch job.
Any code snippets you can provide would be very much appreciated.
SQL Server 2005 and up have a datatype called "XML" which you can store XML in - untyped or typed with a XSD schema.
You can basically fill columns of type XML from an XML literal string, so you can easily just use a normal INSERT statement and fill the XML contents into that field.
Marc
You can use the function OPENXML and stored procedure sp_xml_preparedocument to easily convert your XML into rowsets.
If you are using SQL Server 2008 (or 2005), it has an xml native datatype. You can associate an XSD schema with xml variables, and Insert directly into columns of type xml.
Yes, SQL Server 2005 and above can parse XML out of the box.
You use the nodes, value and query methods to break it down how you want, whether values or attributes
Some shameless plugging:
Importing XML into SQL Server
Search XML Column in SQL
Xml data and Xml document could have different meaning.
When xml type is good for data, it doesn't save formatting (white spaces removed), so in some cases (e.g. cofiguration files) the best option is nvarchar.