Related
This question already has an answer here:
For XML length limitation
(1 answer)
Closed 10 years ago.
I have a query that returns XML which I want to convert to varchar. My query returns 93,643 characters of XML. When I try to cast my xml result as varchar, I only get 43,679 characters when I copy the result set to a text editor. When I do len(xmlString), I get 93,643 characters.
I know from this post that varchar(max) can have up to 2^31 characters and 1 byte = 1 character, but it seems to be cutting off my data.
Do XML characters count as more than 1 byte? Why am I not able to select all the data from my xml result?
CAST((SELECT COLUMNS FROM TABLE FOR XML PATH('Name'), TYPE) AS VARCHAR(MAX)
This is just a limitation of the Managementstudio.
With a testquery on a bigger table I get described 43,679 characters.
The same Query deliveres 267089 characters in a application via ADO.
Not sure why you need to cast you xml data to varchar(max) but if you just want to copy all data don't cast it at all. In this case in the result window you will see one cell with a clickable value (just like a web link). Click it and all your data will be opened in a new window, then you will be able to save it like a file or just copy it. Hope it helps.
Note: I'm running under SQL Server 2008 R2...
I've taken the time to read dozens of posts on this site and other sites on how to execute dynamic SQL where the query is more than 4000 characters. I've tried more than a dozen solutions proposed. The consensus seems to be to split the query into 4000-character variables and then do:
EXEC (#SQLQuery1 + #SQLQuery2)
This doesn't work for me - the query is truncated at the end of #SQLQuery1.
Now, I've seen samples how people "force" a long query by using REPLICATE a bunch of spaces, etc., but this is a real query - but it gets a little more sophisticated than that.
I have SQL View with a name of "Company_A_ItemView".
I have 10 companies that I want to create the same exact view, with different names, e.g.
"Company_B_ItemView"
"Company_C_ItemView"
..etc.
If you offer help, please don't ask why there are multiple views - just accept that I need to do it this way, OK?
Each company has its own set of tables, and the CREATE VIEW statement references several tables by name. Here's BRIEF sample, but remember, the total length of the query is around 6000 characters:
CREATE view [dbo].[Company_A_ItemView] as
select
WE.[Item No_],
WE.[Location Code],
LOC.[Bin Number],
[..more fields, etc.]
from
[Company_A_Warehouse_Entry] WE
left join
[Company_A_Location] LOC
...you get the idea
So, what I am currently doing is:
a. Pulling the contents of the CREATE VIEW statement into 2 Declared Variables, e.g.
Set #SQLQuery1 = (select text
from syscomments
where ID = 1382894081 and colid = 1)
Set #SQLQuery2 = (select
from syscomments
where ID = 1382894081 and colid = 2)
Note that this is how SQL stores long definitions - when you create the view, it stores the text into multiple syscomments records. In my case, the view is split into a text chunk of 3591 characters into the first syscomment record and the rest of the text is in the second record. I have no idea why SQL doesn't use all 4000 characters in the syscomment field. And the statement is broken in the middle of a word.
Please note in all my examples, all #SQLQueryxxx variables are declared as varchar(max). I've also tried declaring them as nvarchar(max) and varchar(8000) and nvarchar(8000) with the same results.
b. I then do a "Search and Replace" for "Company_A" and replace it with "Company_B". In the code below, the variable "#CompanyID" is first set to "Company_B":
SET #SQLQueryNew1 = #SQLQuery1
SET #SQLQueryNew1 = REPLACE(#SQLQueryNew1, 'Company_A', #CompanyID)
SET #SQLQueryNew2 = #SQLQuery2
SET #SQLQueryNew2 = REPLACE(#SQLQueryNew2, 'Company_A',#CompanyID)
c. I then try:
EXEC (#SQLQueryNew1 + #SQLQueryNew2)
The message returned indicates that it's trying to execute the statement truncated at the end of #SQLQueryNew1, e.g. 80% (approx) of the query's text.
I've tried CAST'ing the final result into a new varchar(max) and nvarchar(max) - no luck
I've tried CAST'ing the original query a new varchar(max) and nvarchar(max)- no luck
I've looked at the result of retrieving the original CREATE VIEW statement, and it's fine.
I've tried various other ways of retrieving the original CREATE VIEW statement, such as:
Set #SQLQuery1 = (select VIEW_DEFINITION)
FROM [MY_DATABASE].[INFORMATION_SCHEMA].[VIEWS]
where TABLE_NAME = 'Company_A_ItemView')`
This one returns only the first 4000 characters of the CREATE VIEW
Set #SQLQuery1 = (SELECT (OBJECT_DEFINITION(#ObjectID))
If I do a
SELECT LEN(OBJECT_DEFINITION(#ObjectID))
it returns the correct length of the query (e.g. 5191), but if I look at #SQLQuery1, or try to
EXEC(#SQLQuery1), the statement is still truncated.
c. There are some references that state that since I'm manipulating the text of the query after retrieving it, the resulting variables are then truncated to 4000 characters. I've tried CAST'ing the result as I do the REPLACE, e.g.
SET #SQLQueryNew1 = SELECT (CAST(REPLACE(#SQLQueryNew1,
'Company_A',
#CompanyID) AS varchar(max))
Same result.
I know there are other methods, such as creating stored procedures for creating the views. But the views are being developed and are somewhat "in flux", so placing the text of the CREATE VIEW inside a stored proc is cumbersome. My goal is to be able to take Company_A's view and replicate it exactly - multiple times, except reference Company_B's view name and table names, Company_C's view name and table names, etc.
I'm wondering if there is anyone out there who has done this type of manipulation of a long SQL "CREATE VIEW" statement and try to execute it.
Just use VARCHAR(MAX) or NVARCHAR(MAX). They work fine for EXEC(string).
FYI,
Note that this is how SQL stores long definitions - when you create
the view, it stores the text into multiple syscomments records.
This is not correct. This is how it used to be done on SQL Server 2000. Since SQL Server 2005 and higher they are saved as NVARCHAR(MAX) in a single entry in sys.sql_modules.
syscomments is still around, but it is retained read-only solely for compatibility.
So all you should need to do is to change your #SQLQuery1,2,etc. variables to a single NVARCHAR(MAX) variable, and pull your View code from the [definition] column of the sys.sql_modules table instead.
Note that you should be careful with your string manipulations as there are certain functions that will revert to (N)VARCHAR(4000) output if all of their input arguments are not (N)VARCHAR(MAX). (Sorry, I do not know which ones, but REPLACE() may be one). In fact, this may be what has been causing so much confusion in your tests.
declare your sql variables (#SQLQuery1...) as nvarchar(4000)
be sure each sql part did't exceed 4000 byte (copy each part to a text file and test the file size in bytes)
Software I'm working with uses a text field to store XML. From my searches online, the text datatype is supposed to hold 2^31 - 1 characters. Currently SQL Server is truncating the XML at 65,535 characters every time. I know this is caused by SQL Server, because if I add a 65,536th character to the column directly in Management Studio, it states that it will not update because characters will be truncated.
Is the max length really 65,535 or could this be because the database was designed in an earlier version of SQL Server (2000) and it's using the legacy text datatype instead of 2005's?
If this is the case, will altering the datatype to Text in SQL Server 2005 fix this issue?
that is a limitation of SSMS not of the text field, but you should use varchar(max) since text is deprecated
Here is also a quick test
create table TestLen (bla text)
insert TestLen values (replicate(convert(varchar(max),'a'), 100000))
select datalength(bla)
from TestLen
Returns 100000 for me
MSSQL 2000 should allow up to 2^31 - 1 characters (non unicode) in a text field, which is over 2 billion. Don't know what's causing this limitation but you might wanna try using varchar(max) or nvarchar(max). These store as many characters but allow also the regular string T-SQL functions (like LEN, SUBSTRING, REPLACE, RTRIM,...).
If you're able to convert the column, you might as well, since the text data type will be removed in a future version of SQL Server. See here.
The recommendation is to use varchar(MAX) or nvarchar(MAX). In your case, you could also use the XML data type, but that may tie you to certain database engines (if that's a consideration).
You should have a look at
XML Support in Microsoft SQL Server
2005
Beginning SQL Server 2005 XML
Programming
So I would rather try to use the data type appropriate for the use. Not make a datatype fit your use from a previous version.
Here's a little script I wrote for getting out all data
SELECT #data = N'huge data';
DECLARE #readSentence NVARCHAR (MAX) = N'';
DECLARE #dataLength INT = ( SELECT LEN (#data));
DECLARE #currIndex INT = 0;
WHILE #data <> #readSentence
BEGIN
DECLARE #temp NVARCHAR (MAX) = N'';
SET #temp = ( SELECT SUBSTRING (#data, #currIndex, 65535));
SELECT #temp;
SET #readSentence += #temp;
SET #currIndex += 65535;
END;
In this live SQL Server 2008 (build 10.0.1600) database, there's an Events table, which contains a text column named Details. (Yes, I realize this should actually be a varchar(MAX) column, but whoever set this database up did not do it that way.)
This column contains very large logs of exceptions and associated JSON data that I'm trying to access through SQL Server Management Studio, but whenever I copy the results from the grid to a text editor, it truncates it at 43679 characters.
I've read on various locations on the Internet that you can set your Maximum Characters Retrieved for XML Data in Tools > Options > Query Results > SQL Server > Results To Grid to Unlimited, and then perform a query such as this:
select Convert(xml, Details) from Events
where EventID = 13920
(Note that the data is column is not XML at all. CONVERTing the column to XML is merely a workaround I found from Googling that someone else has used to get around the limit SSMS has from retrieving data from a text or varchar(MAX) column.)
However, after setting the option above, running the query, and clicking on the link in the result, I still get the following error:
Unable to show XML. The following error happened:
Unexpected end of file has occurred. Line 5, position 220160.
One solution is to increase the number of characters retrieved from the server for XML data. To change this setting, on the Tools menu, click Options.
So, any idea on how to access this data? Would converting the column to varchar(MAX) fix my woes?
SSMS only allows unlimited data for XML data. This is not the default and needs to be set in the options.
One trick which might work in quite limited circumstances is simply naming the column in a special manner as below so it gets treated as XML data.
DECLARE #S varchar(max) = 'A'
SET #S = REPLICATE(#S,100000) + 'B'
SELECT #S as [XML_F52E2B61-18A1-11d1-B105-00805F49916B]
In SSMS (at least versions 2012 to current of 18.3) this displays the results as below
Clicking on it opens the full results in the XML viewer. Scrolling to the right shows the last character of B is preserved,
However this does have some significant problems. Adding extra columns to the query breaks the effect and extra rows all become concatenated with the first one. Finally if the string contains characters such as < opening the XML viewer fails with a parsing error.
A more robust way of doing this that avoids issues of SQL Server converting < to < etc or failing due to these characters is below (credit Adam Machanic here).
DECLARE #S varchar(max)
SELECT #S = ''
SELECT #S = #S + '
' + OBJECT_DEFINITION(OBJECT_ID) FROM SYS.PROCEDURES
SELECT #S AS [processing-instruction(x)] FOR XML PATH('')
I was able to get this to work...
SELECT CAST('<![CDATA[' + LargeTextColumn + ']]>' AS XML) FROM TableName;
One work-around is to right-click on the result set and select "Save Results As...". This exports it to a CSV file with the entire contents of the column. Not perfect but worked well enough for me.
Did you try this simple solution? Only 2 clicks away!
At the query window,
set query options to "Results to Grid", run your query
Right click on the results tab at the grid corner, save results as any files
You will get all the text you want to see in the file!!! I can see 130,556 characters for my result of a varchar(MAX) field
The simplest workaround I found is to backup the table and view the script. To do this
Right click your database and choose Tasks > Generate Scripts...
"Introduction" page click Next
"Choose Objects" page
Choose the Select specific database objects and select your table.
Click Next
"Set Scripting Options" page
Set the output type to Save scripts to a specific location
Select Save to file and fill in the related options
Click the Advanced button
Set General > Types of data to script to Data only or Schema and Data and click ok
Click Next
"Summary Page" click next
Your sql script should be generated based on the options you set in 4.2. Open this file up and view your data.
The data type TEXT is old and should not be used anymore, it is a pain to select data out of a TEXT column.
ntext, text, and image (Transact-SQL)
ntext, text, and image data types
will be removed in a future version of
Microsoft SQL Server. Avoid using
these data types in new development
work, and plan to modify applications
that currently use them. Use
nvarchar(max), varchar(max), and
varbinary(max) instead.
you need to use TEXTPTR (Transact-SQL) to retrieve the text data.
Also see this article on Handling The Text Data Type.
It sounds like the Xml may not be well formed. If that is the case, then you will not be able to cast it as Xml and given that, you are limited in how much text you can return in Management Studio. However, you could break up the text into smaller chunks like so:
With Tally As
(
Select ROW_NUMBER() OVER ( ORDER BY s1.object_id ) - 1 As Num
From sys.sysobjects As s1
Cross Join sys.sysobjects As s2
)
Select Substring(T1.textCol, T2.Num * 8000 + 1, 8000)
From Table As T1
Cross Join Tally As T2
Where T2.Num <= Ceiling(Len(T1.textCol) / 8000)
Order By T2.Num
You would then need to manually combine them again.
EDIT
It sounds like there are some characters in the text data that the Xml parser does not like. You could try converting those values to entities and then try the Convert(xml, data) trick. So something like:
Update Table
Set Data = Replace(Cast(Data As varchar(max)),'<','<')
(I needed to cast to varchar(max) because the replace function will not work on text columns. There should not be any reason you couldn't convert those text columns to varchar(max).)
You are out of luck, I think. THe problem is not a SQL level problem as all other answers seem to focus on, but simply one of the user interface. Management Studio is not meant to be a general purpose / generic data access interface. It is not there to be your interface, but your administrative area, and it has serious limitations handling binary data and large test data - because people using it within the specified usage profile will not run into this problem.
Presenting large text data is simply not the planned usage.
Your only choice would be a table valued function that takes the text input and cuts it rows for every line, so that Management Studio gets a list of rows, not a single row.
I prefer this simple XML hack which makes columns clickable in SSMS on a cell-by-cell basis. With this method, you can view your data quickly in SSMS’s tabular view and click on particular cells to see the full value when they are interesting. This is identical to the OP’s technique except that it avoids the XML errors.
SELECT
e.EventID
,CAST(REPLACE(REPLACE(e.Details, '&', '&'), '<', '<') AS XML) Details
FROM Events e
WHERE 1=1
AND e.EventID BETWEEN 13920 AND 13930
;
Starting from SSMS 18.2, you can now view up to 2 million characters in the grid results. Source
Allow more data to be displayed (Result to Text) and stored in cells
(Result to Grid). SSMS now allows up to 2M characters for both.
I verified this with the code below.
DECLARE #S varchar(max) = 'A'
SET #S = REPLICATE(#S,2000000) + 'B'
SELECT #S as a
declare #takeOver table(details nvarchar(max))
declare #json_auto nvarchar(max)
select #json_auto = (select distinct
From table_1 cg
inner join table_2 c
on cg.column_1= c.column_1and cg.isDeleted =0 and c.isdeleted = 0
inner join table_3 d
on c.column_2= d.column_2 and d.isdeleted = 0
where cg.Id= 1017
for Json Auto)
insert into #takeOver
values(#json_auto)
select * from #takeOver
In SQL Server 2005 I am trying to query a varchar(MAX) column which has some rows with text data that exceed the 8192. Yet, In Management Studio I have under Tools --> Options --> Query Results --> Results to Text --> Max numbers of characters displayed in each column = 8192, which is a maximum. Accordingly, it seems the truncation on these rows occurs only due to the limitation imposed by text output.
The only thing I see to get around this is to use a SUBSTRING function to grab say the first 8000 chars, then the next 8000 chars etc. etc. But this is ugly and error prone.
I should mention that SSIS and BCP are not options for me.
Does anyone have a better suggestion? Thanks!
You can export the data to a flat file which will not be truncated. To do this:
Right click the Database
Click Tasks -> Export Data
Select your Data Source (defaults should be fine)
Choose "Flat File Destination" for the Destination type.
Pick a file name for the output.
On the "Specify Table Copy or Query", choose "Write a query to specify the data to transfer"
Paste in your query
Remaining steps should be self explanatory. This will output the file to text and you can open it in your favorite text editor.
I also use XML but a slightly different method that gets around most of the issues with XML entitisation.
declare #VeryLongText nvarchar(max) = '';
SELECT top 100 #VeryLongText = #VeryLongText + '
' + OBJECT_DEFINITION(object_id)
FROM sys.all_objects
WHERE type='P' and is_ms_shipped=1
SELECT LEN(#VeryLongText)
SELECT #VeryLongText AS [processing-instruction(x)] FOR XML PATH('')
PRINT #VeryLongText /*WILL be truncated*/
Make sure that the "XML data" limit in SSMS is set sufficiently high!
Did you try this simple solution? Only 2 clicks away!
At the query window,
set query options to "Results to Grid", run your query
Right click on the results tab at the grid corner, save results as any files
You will get all the text you want to see in the file!!! I can see 130,556 characters for my result of a varchar(MAX) field
My solution was a bit round-about but got me there (as long as the output is less than 65535 characters):
In SQL Management Studio, set the limit for grid results to 65535 (Tools > Options > Query Results > SQL Server > Results to Grid > Non XML data)
Run the query, output to grid
Right-click the results, choose "Save Results As..." and save the results to a file
Open the file in notepad or similar to get the output
UPDATE: To demonstrate that this works, here's some SQL that selects a single 100,000 character column. If I save the grid output to a csv file, all 100,000 characters are there with no truncation.
DECLARE #test nvarchar(MAX), #i int, #line nvarchar(100)
SET #test = ''; SET #i = 100
WHILE #i < 100000
BEGIN
SET #test = #test + STUFF(REPLICATE('_', 98) + CHAR(13) + CHAR(10), 1, LEN(CAST(#i AS nvarchar)), CAST(#i AS nvarchar))
SET #i = #i + 100
END
SELECT #test
Notes:
It doesn't seem to make any difference what the character length setting is, as I orignally thought.
I'm using SQL 2008 R2 (both the server and Management Studio)
It doesn't seem to make a difference if the long column is stored in a local variable (as in this example), or selected from an actual table
I ran in to this trying to export XML. This is the solution I used:
Select the Result to Grid option, right click the link that shows up in the Results pane, then select Save Results As, choose the All Files file type, give the file a name and click Save. All the xml data is saved correctly to a file.
I'm using SSMS 10, and I could't get Torre's solution to work. The export wizard kept thinking the input column was an image:
The data type for "input column "XML_F52E2B61-18A1-11d1-B105-00805F49916B" (26)" is DT_IMAGE
In SSMS if you select data from a row it is limited to a small number of characters, but if you Edit data from a row, the full value will be there. It might not always be there but if you ctrl-a, ctrl-c then past it in an editor it will all be there.
If given a choice I would have the query return the data as "For XML Auto" or "For XML Raw" or "For XML explicit" that way the limitations are much higher and you can do much more with the outputed results.
I usually use XML to get huge debug string as output (using test harness from Luke):
declare #test nvarchar(max), #i int, #line nvarchar(100)
set #test = ''; set #i = 100
while #i < 100000
begin
set #test = #test + stuff(replicate('_', 98) + char(13) + char(10), 1, len(cast(#i as nvarchar)), cast(#i as nvarchar))
set #i = #i + 100
end
-- ctrl+d for "results to grid" then click the xml output
--select cast('<root>' + #test + '</root>' as xml)
-- revised
select #test for xml path(''), type;
Another workaround , use HeidiSql for this tricky queries. It does not have the limits in the field lenght.
The truncation you are talking about only happens in Management Studio. If you pull the column into another app, it will not be truncated.
There's no way you're using Query Analyzer to talk to SQL Server 2005. Do you mean Management Studio?