How to trim a large table with more than 300 columns dynamically. when i tried with this code i was getting an error because the variable #sql is of nvarchar(max) and i have like 300 columns so all the column names were not coming when i run this query can any of the guys help me in this ?
DECLARE #SQL nVARCHAR(MAX)
DECLARE #TableName NVARCHAR(128)
SET #TableName = 'MYTAbleName'
SELECT #SQL = COALESCE(#SQL + ',[', '[') +
COLUMN_NAME + ']=LTRIM(RTRIM([' + COLUMN_NAME + ']))'
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = #TableName
SET #SQL = 'UPDATE [' + #TableName + '] SET ' + #SQL
PRINT #SQL
EXECUTE #SQL
While you have defined #SQL nvarchar(max), it all boils down to how a query is sent to the SQL server engine.
Quote from MSDN maximum capacity specifications.
"Network Packet Size is the size of the tabular data stream (TDS) packets used to communicate between applications and the relational Database Engine. The default packet size is 4 KB, and is controlled by the network packet size configuration option."
The Max batch size is 64 * 4 or 250 MB for a query to be submitted to the engine. This is alot smaller than the (max) = 2GB.
Add a debugging line to see if you are coming close to this limitation. Comment out printing the t-sql and executing the query.
-- Show how long the dynamic t-sql is
-- PRINT #SQL
PRINT 'LEN OF TSQL'
PRINT LEN(#SQL)
-- EXECUTE #SQL
Please report back if this is the issue!
I do not have a test case like yours.
-- Use msdb
USE MSDB
GO
-- Largest column count = 62, [sysutility_ucp_instances]
SELECT TABLE_NAME, MAX(ORDINAL_POSITION) as MaxCols
FROM INFORMATION_SCHEMA.COLUMNS
GROUP BY TABLE_NAME
order by MAX(ORDINAL_POSITION) desc
Since your TSQL is getting cut off, it must be casting to a varchar() which has a max of 8000 bytes.
Try casting each piece of the update to varchar(max).
SET #SQL = cast('UPDATE [' as varchar(max)) + cast(#TableName as varchar(max)) + cast('] SET ' as varchar(max)) + #SQL;
Use varchar() instead of nvarchar to get one power of 2 back.
I hope this fixes your issue.
Related
We have a View in SQL Server that constantly evolving.
We want to show it in a report as it is (If we add/remove a field in the View, we don't have to modify the report and add/remove manuelly the field).
A sort of a table/matrix that is refreshing by itself.
Thank you by advance for your help.
Without using a table or matrix? Why not? You can't really do this without using one of these controls...
Ignoring that for a moment though, the problem you will face is that the dataset query has to always return the same structure every time it is run so you can't point it directly at a query that is constantly changing.
The only way you might be able to do this is to write a query that unpivots your table/view into another structure and then report on that. By using a matrix, you could reconstruct the table in the report.
There are drawbacks to this approach. All value data needs to be cast to a constant datatype so if each row has a mix of text and numeric values, they would all have to be converted to text.
This approach also assumes there is a key column on the table/view.
Below is a simple example of the kind of thing I mean. This is based on the sample 'AdventureWorksDW2016' database in case you want to test it.
DECLARE #Schema sysname = 'dbo' -- Schema where table/view resides
DECLARE #Table sysname = 'DimGeography' -- name of table or view to read from
DECLARE #KeyColumn sysname = 'GeographyKey' -- name of keycolumn, assumed to be INT in this exmaple
SELECT
COLUMN_NAME, ORDINAL_POSITION, DATA_TYPE
into #t
FROM INFORMATION_SCHEMA.COLUMNS t
WHERE TABLE_SCHEMA = #Schema AND TABLE_NAME = #Table
AND COLUMN_NAME != #KeyColumn
DECLARE #OrdPos int
DECLARE #ColName sysname
DECLARE #sql varchar(max) = ''
CREATE TABLE #result (KeyID int, ColumnName sysname, ColumnPosition int, ColumnValue varchar(75)) -- <= Update 75 to suit maximum column length
WHILE EXISTS(SELECT * FROM #t)
BEGIN
SELECT TOP 1 #OrdPos = ORDINAL_POSITION, #ColName = COLUMN_NAME FROM #t
SET #SQL = 'INSERT INTO #result SELECT ' + #KeyColumn + ', ''' + #ColName + ''', ' + CAST(#OrdPos as varchar(10)) + ', CAST(' + #ColName + ' as varchar(200)) FROM ' + QUOTENAME(#Schema) + '.' + QUOTENAME(#Table)
EXEC (#sql)
DELETE FROM #t WHERE ORDINAL_POSITION = #OrdPos
END
SELECT * FROM #result
If we take a look at the results (just for 2 keyid vales for simplicity) we can see we have a consistent structure.
SELECT * FROM #result where keyid in (207,208) order by KeyID, ColumnPosition
Now, you can build a simple report using a Matrix, have a row group that groups by KeyID and have a column group that groups by ColumnName. The column group sorting can be set to ColumnPosition and the matrix 'data' cell set to ColumnValue.
This whole process will effectively recreate the table/view and be dynamic.
Since I am flushing in a lot of data into a table on daily basis, I am trying to calculate the index and data Bytes per row size, and based on research below is the script that I have come across to identify the data byte size per row:
declare #table nvarchar(128);
declare #idcol nvarchar(128);
declare #sql nvarchar(max);
--initialize those two values
set #table = '[Person].[AddressType]'
set #idcol = 'AddressTypeID, Name'
set #sql = 'select ' + #idcol + ' , (0'
-- This select statement collects all columns of a table and calculate datalength
select #sql = #sql + ' + isnull(datalength(' + name + '), 1)'
from sys.columns where object_id = object_id(#table)
set #sql = #sql + ') as RowSize from ' + #table + ' order by rowsize desc'
-- Execute sql query
exec (#sql)
Th above script suffices the requirement of calculating the data bytes per row. However I am not sure how to calculate the bytes per row for an index.
These are the indexes defined on the table:
For non clustered index you can use this:
https://learn.microsoft.com/en-us/sql/relational-databases/databases/estimate-the-size-of-a-nonclustered-index?view=sql-server-ver15
For clustered index you can use this:
https://learn.microsoft.com/en-gb/sql/relational-databases/databases/estimate-the-size-of-a-clustered-index?view=sql-server-ver15
I have a stored procedure that dynamically generates a table to hold staging data for imports. This routine was running fine until Wednesday. I have tracked this down to one particular area. Code that's causing an issue is:
DECLARE #strSQL NVARCHAR(MAX) = N'';
SELECT #strSQL = #strSQL + N',[' + CustomerField + N'] NVARCHAR(MAX)' + CHAR(10)
FROM dbo.WebServices
WHERE CallType = 'customer'
AND IsPrimaryTable = 1
AND Source = 'clientName'
ORDER BY TagOrder;
SET #strSQL = STUFF(#strSQL, 1, 1, 'CREATE TABLE ' + 'TableName' + CHAR(10) + '(')+ N')';
PRINT #strSQL
NB. Table its pulling from holds a list of columns and the data hasn't changed either. There are about 34 rows of data with no special characters, and it isn't exposed to the clients.
This gives me an output of:
CREATE TABLE TableName
([CUSACNR2] NVARCHAR(MAX)
)
Running this on exactly the same data, even in the same window on SSMS, with an additional TOP filter gives me the results I was expecting i.e.
DECLARE #strSQL NVARCHAR(MAX) = N'';
SELECT TOP (99999) #strSQL = #strSQL + N',[' + CustomerField + N'] NVARCHAR(MAX)' + CHAR(10)
FROM dbo.WebServices
WHERE CallType = 'customer'
AND IsPrimaryTable = 1
AND Source = 'clientName'
ORDER BY TagOrder;
SET #strSQL = STUFF(#strSQL, 1, 1, 'CREATE TABLE ' + 'TableName' + CHAR(10) + '(')+ N')';
PRINT #strSQL
Generates the dynamic SQL:
CREATE TABLE TableName
([TAG] NVARCHAR(MAX)
,[CUSACNR] NVARCHAR(MAX)
.
.
.
,[SHIPPING_POSTAL_CODE] NVARCHAR(MAX)
,[CUSACNR2] NVARCHAR(MAX)
)
Specific column names removed for security
While I have tracked this down in one specific stored proc, we use this technique for building dynamic SQL a lot and I am seeing errors popping up in multiple places. I'm assuming that something has changed on the server as this has started happening in multiple databases at one time.
Does anyone know of anything that may have caused this issue? Its a legacy system that has been running fine for a long time and re-coding every instance of this type of activity is not an option.
I have found the answer. Sort of. I now know what had changed, so I've been able to correct it, but it should not have made any difference.
There was an index on the table that was causing the issue. Before you say that shouldn't make any difference, and you're right it shouldn't, I've managed to reliably solve and reproduce the problem by dropping and recreating the index.
So this isn't a fragmentation issue, it's not a clustered index, and the execution plan doesn't search by this index anyway.
CREATE NONCLUSTERED INDEX [idx_ntindex] ON [dbo].[WebServices] ([Source], [CallType], [SecurityKey])
GO
DECLARE #strSQL NVARCHAR(MAX) = N'';
SELECT #strSQL = #strSQL + N',[' + ISNULL(CustomerField,'') + N'] NVARCHAR(MAX)' + CHAR(10)
FROM dbo.WebServices
WHERE CallType = 'customer'
AND IsPrimaryTable = 1
AND Source = 'clientName'
ORDER BY TagOrder;
SET #strSQL = STUFF(#strSQL, 1, 1, 'CREATE TABLE ' + 'TableName' + CHAR(10) + '(') + N')';
PRINT #strSQL;
GO
DROP INDEX idx_ntindex ON dbo.WebServices
DECLARE #strSQL NVARCHAR(MAX) = N'';
SELECT #strSQL = #strSQL + N',[' + ISNULL(CustomerField,'') + N'] NVARCHAR(MAX)' + CHAR(10)
FROM dbo.WebServices
WHERE CallType = 'customer'
AND IsPrimaryTable = 1
AND Source = 'clientName'
ORDER BY TagOrder;
SET #strSQL = STUFF(#strSQL, 1, 1, 'CREATE TABLE ' + 'TableName' + CHAR(10) + '(') + N')';
PRINT #strSQL;
Gives me the results
CREATE TABLE TableName
([CUSACNR2] NVARCHAR(MAX)
)
CREATE TABLE TableName
([TAG] NVARCHAR(MAX)
,[CUSACNR] NVARCHAR(MAX)
,[DDAYS] NVARCHAR(MAX)
.
.
.
,[SHIPPING_POSTAL_CODE] NVARCHAR(MAX)
,[CUSACNR2] NVARCHAR(MAX)
)
If anyone knows WHY this is the case then please let me know. I found it but it doesn't make any sense to me.
Thanks for the help all. +1's for all the support, each suggestion got me a little closer.
Using SQL Server 2012, I've found that trying to build up a string based on an nvarchar(max) column in a table doesn't seem to work correctly. It seems to overwrite, instead of append. Arbitrary Example:
DECLARE #sql nvarchar(max);
SELECT #sql = N'';
SELECT #sql += [definition] + N'
GO
'
FROM sys.sql_modules
WHERE OBJECT_NAME(object_id) LIKE 'dt%'
ORDER BY OBJECT_NAME(object_id);
PRINT #sql;
This SHOULD print out all the SQL module definitions for all the various dt_ tables in the database, separated by GO, as a script that could then be run. However... this prints out only the LAST module definition, not the sum of all of them. It's behaving as if the "+=" were just an "=".
If you change it just slightly... cast [definition] to an nvarchar(4000) for example, it suddenly works as expected. Also, if you choose any other column that is NOT an nvarchar(max) or varchar(max) type, it works as expected. Example:
DECLARE #sql nvarchar(max);
SELECT #sql = N'';
SELECT #sql += CAST([definition] AS nvarchar(4000)) + '
GO
'
FROM sys.sql_modules
WHERE OBJECT_NAME(object_id) LIKE 'dt%'
ORDER BY OBJECT_NAME(object_id);
PRINT #sql;
Is this a known bug? Or is this working as expected? Am I doing something wrong? Is there any way for me to make this work correctly? I've tried a dozen different things, including ensuring every expression in the concatenation is the same nvarchar(max) type, including the string literal.
NOTE: The example is just an example that shows the problem, and not exactly what I'm trying to do in real life. If your database doesn't have the "dt*" tables defined, you can change the WHERE clause to specify any group of tables or stored procedures in any database you want, you'll get the same result... only the last one shows up in the #sql string, as if you just did "=" instead of "+=". Also, explicitly stating "#sql = #sql + " behaves the same way... works correctly with every string type EXCEPT nvarchar(max) or varchar(max).
I've verified that none of the [definition] values is NULL as well, so there are no NULL shenanigans going on.
The += operator only applies to numeric data types in SQL Server. Microsoft documentation here
For string concatenation, you need to write the assignment and concatenation separately.
DECLARE #sql nvarchar(max);
SELECT #sql = N'';
SELECT #sql = #sql + [definition] + N'
GO
'
FROM sys.sql_modules
WHERE OBJECT_NAME(object_id) LIKE 'dt%'
ORDER BY OBJECT_NAME(object_id);
PRINT #sql;
Also, if you are running this query in Management Studio, keep in mind that there is a limit to the size of the data that it will return (including in a print statement). So if the definitions of your modules exceed this limit, they will be truncated in the output.
To add on to #HABO's comment, the behavior of aggregate string concatenation is undefined and results are plan dependent. Use FOR XML to provide deterministic results and honor the ORDER BY clause:
DECLARE #sql nvarchar(max);
SET #sql =
(SELECT [definition] + N'
GO
'
FROM sys.sql_modules
WHERE OBJECT_NAME(object_id) LIKE 'dt%'
ORDER BY OBJECT_NAME(object_id)
FOR XML PATH(''), TYPE).value('(./text())[1]', 'nvarchar(MAX)');
PRINT #sql; --SSMS will truncate long strings
It works as expected:
DECLARE #sql nvarchar(max);
SELECT #sql =COALESCE(#sql + N' GO ','')+[definition]
FROM sys.sql_modules
Print #sql;
The following code demonstrates taking a column value from a set of ordered rows and, using for xml, creating an aggregated string with separators, including line breaks, between values.
-- Carriage return and linefeed characters.
declare #CRLF as NVarChar(16) = NChar( 13 ) + NChar( 10 );
-- Carriage return and linefeed characters XML encoded.
declare #EncodedCRLF as NVarChar(16) = N'
';
-- Separator to insert between source rows in the accumulated string.
declare #Separator as NVarChar(64) = ';' + #EncodedCRLF + 'go;' + #EncodedCRLF + 'do something with ';
-- Characters in #Separator to remove from the first element in #Result .
declare #SkipCount as Int = 4 + 2 * Len( #EncodedCRLF );
-- The result.
declare #Result as NVarChar( max ) = '';
-- Do it.
select #Result =
Replace(
Stuff(
-- Specify the sorting order of the XML elements in the following select statement.
( select #Separator + name from sys.tables order by name desc for XML path(''), type).value('.[1]', 'VarChar(max)' ),
1, #SkipCount, '' ),
#EncodedCRLF, #CRLF );
-- Output the result.
select #Result as [Result];
-- Output the result with line breaks shown.
print #Result; -- Switch to Messages tab in SSMS to see result.
I´m pretty sure the PRINT command is the reason for the troubles.
When I try:
DECLARE #SQL NVARCHAR(MAX) = N'';
SELECT #sql += SUBSTRING( [definition], 1, 100 ) + CHAR(13) + CHAR(10) + 'GO' + CHAR(13) + CHAR(10)
FROM SYS.SQL_MODULES
PRINT #sql;
I get all my procedures (the first 100 chars ...). So it seems to be a limitation to the PRINT command.
Second try, another database, without "substring":
DECLARE #SQL NVARCHAR(MAX) = N'';
SELECT #sql += [definition] + CHAR(13) + CHAR(10) + 'GO' + CHAR(13) + CHAR(10)
FROM SYS.SQL_MODULES
PRINT #sql;
Now I got 1,5 procedures: The 1st one complete, the second stopped in the middle of on line in the proc. There is a limit in PRINT
Third try, to proof my assumption:
DECLARE #SQL NVARCHAR(MAX) = N'';
SELECT #sql += [definition] + 'GO' FROM SYS.SQL_MODULES
PRINT #sql;
Now I got one line of code more! So for me, this is the proof, that PRINT has somehow a limit.
A last attempt:
DECLARE #SQL NVARCHAR(MAX) = N'';
SELECT #sql += [definition] + N'GO' FROM SYS.SQL_MODULES
PRINT SUBSTRING(#SQL,1,4000)
PRINT SUBSTRING(#SQL,4001,4000)
The result: Now I got more then 2 procedures, although there now is a line break after 4K, this may occur on ... bad places.
I have a database that contains a lot of NCHAR(n) columns. Some of these columns are simple properties, but some are also primary-keys or foreign keys.
There are many tables and columns and a huge amount of test data.
What is the best way to convert every NCHAR(n) column to a NVARCHAR(n) column of the same length and trim its content while doing so?
If you can think of anything better than changing the column in the designer, remembering the column name and trimming it in a script window, please post it as an answer.
Something like this would do the trick I think (unless any are primary keys and then you will get an error - I would suggest that you manually do the primary keys in the designer in your case because to alter them you have to drop the constraints etc and that gets a bit tricky)...
It gets all the table names, column names and sizes for all the columns that are nchar and for each performs an alter statement to change the column type and then an update that trims the data.
There may be a more performant way to do it but if you are only doing it once perhaps this will be ok (oh, change the databaseName bit to the name of your database)...
use databaseName
declare #tn nvarchar(128)
declare #cn nvarchar(128)
declare #ln int
declare #sql as nvarchar(1000)
declare c cursor for
select table_name,column_name,character_maximum_length
from information_schema.columns
where data_type ='nchar' and
TABLE_NAME not in (select TABLE_NAME from INFORMATION_SCHEMA.VIEWS)
open c
fetch next from c into #tn, #cn, #ln
while ##FETCH_STATUS = 0
begin
set #sql = 'alter table ' + #tn + ' alter column '
+ #cn + ' nvarchar(' + convert(nvarchar(50), #ln) + ')'
exec sp_executesql #sql
set #sql = 'update ' + #tn + ' set ' + #cn + ' = LTRIM(RTRIM(' + #cn + '))'
exec sp_executesql #sql
fetch next from c into #tn, #cn, #ln
end
close c
deallocate c