How do I validate a variable against multple database tables - sql-server

Does anyone know how to check a a variable against all database table with columns storing the same type of information? I have a poorly designed database that stores ssn in over 60 tables within one database. some of the variations of columns in the various tables include:
app_ssn
ca_ssn
cand_ssn
crl_ssn
cu_ssn
emtaddr_ssn
re_ssn
sfcart_ssn
sfordr_ssn
socsecno
ssn
Ssn
SSN
I want to create a stored procedure that will accept a value and check it against every table that has 'ssn' in the name.Does anyone have idea as to how to do this?

-- I assume that table/column names don't need to be surrounded by square braces. You may want to save matches in a table - I just select them. I also assume ssn is a char.
alter proc proc1
#search1 varchar(500)
as
begin
set nocount on
declare #strsql varchar(500)
declare #curtable sysname
declare #prevtable sysname
declare #column sysname
select top 1 #curtable= table_schema+'.'+table_name, #column=column_name
from INFORMATION_SCHEMA.COLUMNS
where CHARINDEX('ssn',column_name) > 0
order by table_schema+'.'+table_name +column_name
-- make sure that at least one column has ssn in the column name
if #curtable is not null
begin
while (1=1)
begin
set #strsql = 'select * from ' +#curtable +' where '+''''+#search1+''''+ ' = '+#column
print #strsql
-- any matches for passed in ssn will match here...
exec (#strsql)
set #prevtable = #curtable+#column
select top 1 #curtable= table_schema+'.'+table_name, #column=column_name
from INFORMATION_SCHEMA.COLUMNS
where CHARINDEX('ssn',column_name) > 0
and table_schema+'.'+table_name +column_name> #prevtable
order by table_schema+'.'+table_name +column_name
-- when we run out of columns that contain ssn we are done...
if ##ROWCOUNT = 0
break
end
end
end

What you will need to do is some research. But here is where you can start;
SELECT tbl.NAME AS TableName
,cl.NAME AS ColumnName
,IDENTITY(INT, 1, 1) AS ID
INTO #ColumnsToLoop
FROM sys.tables tbl
JOIN sys.columns cl ON cl.object_id = tbl.object_id
This will give you the table / column relation then you can simply build a dynamic SQL string based on each row in the query above (basically loop it) and use EXEC or sp_execsql. So basically;
DECLARE #Loop int = (select min(ID) From #ColumnsToLoop),#MX int = (Select MAX(ID) From #ColumnsToLoop)
WHILE(#Loop<=#MX)
BEGIN
DECLARE #SQL nvarchar(MAX) = 'SQL String'
//Construct the dynamic SQL String
EXEC(#SQL);
SET #Loop += 1
END

Perhaps I went a little too crazy with this one, but let me know. I thought it would best the primary key of the search results with the table name so you could join it to your tables. I also managed to do it without a single cursor or loop.
DECLARE #SSN VARCHAR(25) = '%99%',
#SQL VARCHAR(MAX);
WITH CTE_PrimaryKeys
AS
(
SELECT TABLE_CATALOG,
TABLE_SCHEMA,
TABLE_NAME,
column_name
FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE D
WHERE OBJECTPROPERTY(OBJECT_ID(constraint_name), 'IsPrimaryKey') = 1
),
CTE_Columns
AS
(
SELECT A.*,
CONCAT(A.TABLE_CATALOG,'.',A.TABLE_SCHEMA,'.',A.TABLE_NAME) AS FullTableName,
CASE WHEN B.COLUMN_NAME IS NOT NULL THEN 1 ELSE 0 END AS IsPrimaryKey
FROM INFORMATION_SCHEMA.COLUMNS A
LEFT JOIN CTE_PrimaryKeys B
ON A.TABLE_CATALOG = B.TABLE_CATALOG
AND A.TABLE_SCHEMA = B.TABLE_SCHEMA
AND A.TABLE_NAME = B.TABLE_NAME
AND A.COLUMN_NAME = B.COLUMN_NAME
),
CTE_Select
AS
(
SELECT
'SELECT ' +
--This returns the pk_col casted as Varchar and the table name in another columns
STUFF((SELECT ',CAST(' + COLUMN_NAME + ' AS VARCHAR(MAX)) AS pk_col,''' + B.TABLE_NAME + ''' AS Table_Name'
FROM CTE_Columns B
WHERE A.Table_Name = B.TABLE_NAME
AND B.IsPrimaryKey = 1
FOR XML PATH ('')),1,1,'')
+ ' FROM ' + fullTableName +
--This is where I list the columns where LIKE desired SSN
' WHERE ' +
STUFF((SELECT COLUMN_NAME + ' LIKE ''' + #SSN + ''' OR '
FROM CTE_Columns B
WHERE A.Table_Name = B.TABLE_NAME
--This is where I filter so I only get desired columns
AND (
--Uncomment the Collate if your database is case sensitive
COLUMN_NAME /*COLLATE SQL_Latin1_General_CP1_CI_AS*/ LIKE '%ssn%'
--list your column Names that don't have ssn in them
--OR COLUMN_NAME IN ('col1','col2')
)
FOR XML PATH ('')),1,0,'') AS Selects
FROM CTE_Columns A
GROUP BY A.FullTableName,A.TABLE_NAME
)
--Unioning them all together and getting rid of last trailing "OR "
SELECT #SQL = COALESCE(#sql,'') + SUBSTRING(selects,1,LEN(selects) - 3) + ' UNION ALL ' + CHAR(13) --new line for easier debugging
FROM CTE_Select
WHERE selects IS NOT NULL
--Look at your code
SELECT SUBSTRING(#sql,1,LEN(#sql) - 11)

Related

Update with dynamic tables

I have to write update using dynamic sql becaus i know only name of column that I want to update and names of columns which I will use to join tables in my update. But I don't know the numbers of tables and names. Names of tables I will get in parameter of my procedure in this way
declare #Tables = N'Customer,Employee,Owner'
So I want to have update like this:
update t
set [Status] = 100
from
TemporaryTable t
left join Customer t1 on t1.RecordId = t.RecordId
left join Employee t2 on t2.RecordId = t.RecordId
left join Owner t3 on t3.RecordId =t.RecordId
where
t1.RecordId is null
and t2.RecordId is NULL
and t3.RecordId is null
I know that each table will have column RecordId and want to left join this tables to my TemporaryTable on this column but I don't know the names and numbers of tables. For example I will have one, two, or ten tables with different names. I know that this tables names will be save in parameter #Tables in that way:
#Tables = N'Customer,Employee,Owner'
There is possilble to write this update in dynamic way?
This is an answer, which helps ... to write update using dynamic sql ... and only shows how to generate a dynamic statement. It's based on string splitting. From SQL Server 2016+ you may use STRING_SPLIT() (because here the order of the substrings is not important). For previous versions you need to find a string splitting function.
T-SQL:
DECLARE #Tables nvarchar(max) = N'Customer,Employee,Owner'
DECLARE #join nvarchar(max) = N''
DECLARE #where nvarchar(max) = N''
DECLARE #stm nvarchar(max) = N''
SELECT
#join = #join + CONCAT(
N' LEFT JOIN ',
QUOTENAME(s.[value]),
N' t',
ROW_NUMBER() OVER (ORDER BY (SELECT 1)),
N' ON t',
ROW_NUMBER() OVER (ORDER BY (SELECT 1)),
N'.RecordId = t.RecordId'
),
#where = #where + CONCAT(
N' AND t',
ROW_NUMBER() OVER (ORDER BY (SELECT 1)),
N'.RecordId is NULL'
)
FROM STRING_SPLIT(#Tables, N',') s
SET #stm = CONCAT(
N'UPDATE t SET [Status] = 100 ',
N'FROM TemporaryTable t',
#join,
N' WHERE ',
STUFF(#where, 1, 5, N'')
)
PRINT #stm
EXEC sp_executesql #stm
Notes:
One note, that I think is important - consider passing tables names using table value type for parameter, not as comma-separated text.
It seems like this will suit your needs, though I don't fully understand what you're trying to do. Here we're constructing the final SQL in two pieces (#s and #where) and then concatenating into the final SQL at the end.
declare #Tables varchar(100) = N'Customer,Employee,Owner'
declare #tablenames table (tablename nvarchar(100))
insert #tablenames (tablename)
select value
from string_split(#Tables, ',');
declare #where varchar(max) = ''
declare #s varchar(max) = '
update t
set [Status] = 100
from TemporaryTable t'
select #s += '
left join ' + tablename + ' on ' + tablename + '.RecordId = t.RecordId'
, #where += case when #where = '' then '' else ' and ' end + tablename + '.RecordId is null
'
from #tablenames
print #s + char(13) + ' where ' + #where
exec( #s + char(13) + ' where ' + #where)

Select all unique values from all columns in a table

I need to select all unique values from all columns in a table.
I have tried to implement the query below which I found in the thread How to get unique values from all columns of a table in SQL Server.
declare #Sql_Str varchar(8000)='';
select #Sql_Str=#Sql_Str+' select cast (' +name +' as varchar(500))
from <yourtable> union'
from sys.columns
where [object_id]=object_id('<yourtable>');
set #Sql_Str=SUBSTRING(#Sql_Str,1,len(#Sql_Str)-6);
exec(#Sql_Str)
I cannot get that query to work however. My table has 118 columns. I think that may be more data than the query above may handle.
Try something like this:
DECLARE #Schema VARCHAR(500)='dbo';
DECLARE #tableName VARCHAR(500)='SomeTable';
DECLARE #cmd NVARCHAR(MAX)=
(
SELECT STUFF(
(
SELECT ' UNION ALL SELECT ''' + c.TABLE_SCHEMA + ''' AS TableSchema '
+ ',''' + c.TABLE_NAME + ''' AS TableName '
+ ',''' + c.COLUMN_NAME + ''' AS ColumnName '
+ ',''' + c.DATA_TYPE + ''' AS ColumnType '
+ ',CAST(' + QUOTENAME(c.COLUMN_NAME)+' AS NVARCHAR(MAX)) AS Value '
+ ' FROM ' + QUOTENAME(c.TABLE_SCHEMA) + '.' + QUOTENAME(c.TABLE_NAME)
+ ' WHERE ' + QUOTENAME(c.COLUMN_NAME) + ' IS NOT NULL '
+ ' GROUP BY ' + QUOTENAME(c.COLUMN_NAME) + ' '
FROM INFORMATION_SCHEMA.COLUMNS AS c
WHERE TABLE_NAME=#TableName
AND TABLE_SCHEMA=#Schema
--exclude not supported types
--AND c.DATA_TYPE NOT IN('xml') --add more types
FOR XML PATH(''),TYPE
).value('.','nvarchar(max)'),1,10,'')
);
--PRINT #cmd
EXEC(#cmd);
This statement will first create a long list of UNION ALL SELECT with GROUP BY (better than DISTINCT) as dynamically created SQL and executes this with EXEC().
You can decomment PRINT to examine the statement created.
This should work in tSQL:
declare #table_name varchar(55)
set #table_name= 'IV00101' ---- <-- Change this to your table name
create table #colcount (
colname varchar(55),
dct int,
tot int
)
create table #colContent (
colname varchar(55),
col_val nvarchar(max),
col_val_count int
)
create table #sqlexecs( s varchar(max))
declare #col_name varchar(max), #sql nvarchar(max), #sql2 nvarchar(max)
declare c cursor for
select name from sys.columns where [object_id]=object_id(#table_name)
open c
fetch next from c into #col_name
while ##FETCH_STATUS = 0
begin
set #sql = 'select cn.name, count(distinct '+#col_name+') as dct_numrow, count('+#col_name+') as tot_numrow from '+#table_name+' join (select name from sys.columns where name = '''+#col_name+''' and [object_id]=object_id('''+#table_name+''')) cn on cn.name = '''+#col_name+''' group by cn.name'
set #sql2 = 'select ' +#col_name+', count('+#col_name+') as colvalcnt from '+#table_name+' group by '+#col_name
--insert into #sqlexecs values (#sql) --uncomment to view sql selects produced by #sql
--insert into #sqlexecs values (#sql2) --uncomment to view sql selects produced by #sql2
insert into #colcount execute sp_executesql #sql
------
declare #d int, #t int
set #d = (select dct from #colcount where colname = #col_name)
set #t = (select tot from #colcount where colname = #col_name)
if (#d <> #t)
begin
insert into #colContent (colname) values (#col_name)
insert into #colContent (col_val,col_val_count) execute sp_executesql #sql2
end
else
begin
insert into #colContent values (#col_name,1,1)
end
fetch next from c into #col_name
end
close c
deallocate c
--select * from #sqlexecs -- uncomment to view sql code produced by #sql and #sql2
select * from #colcount --order by dct desc
select * from #colContent
drop table #colcount
drop table #colContent
drop table #sqlexecs
The first table shows column name, distinct value count, and total value count.
The second table shows column name, distinct values, and the number of times a distinct value appears. If values in column are all distinct (column is a candidate key), colname | 1 | 1 is shown. This should work if copy/pasted, please let me know it doesn't. Dev for use in Dynamics GP.

search a string in a table without knowing the column

Note: this is NOT asking
how to select a string where the column name is known.
how to select a string in ALL tables (all google results relate to this one)
This is asking search in only ONE table.
SQL returns error info conversion failed when converting the nvarchar value S3N2V5.
I want to locate the column name where S3N2V5 exists.
No manual methods please. There are 1000000 columns.
Input S3N2V5
Output columnname1ofthistable
Assuming I understand the question, here is one way to get a list of all columns from a single table that contain the search value, using CASE:
Create and populate sample table (Please save us this step in your future questions)
CREATE TABLE T
(
COL1 char(3),
COL2 char(3),
COL3 char(3),
COL4 int
)
INSERT INTO T VALUES
('abc', 'def', 'nop', 1),
('klm', 'nop', 'qrs', 2),
('tuv', 'wzy', 'zab', 3)
Build your dynamic sql:
DECLARE #Search nvarchar(5) = 'nop'
DECLARE #SQL nvarchar(max) = 'SELECT CASE #Search'
SELECT #SQL = #SQL +' WHEN '+ COLUMN_NAME + ' THEN '''+ COLUMN_NAME +''''
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'T'
AND LOWER(DATA_TYPE) LIKE '%char%' -- only search char, varchar, nchar and nvarchar columns
SELECT #SQL = 'SELECT ColumnName FROM (' +
#SQL + ' END As ColumnName FROM T) x WHERE ColumnName IS NOT NULL'
Execute: (Note that using sp_executeSQL is SQL Injection safe, since we do not concatenate the search parameter into the query, but using it as a parameter)
EXEC sp_executeSQL #SQL, N'#Search nvarchar(5)', #Search
Results:
ColumnName
COL3
COL2
DECLARE #MyValue NVarChar(4000) = 'searchstring';
SELECT S.name SchemaName, T.name TableName
INTO #T
FROM sys.schemas S INNER JOIN
sys.tables T ON S.schema_id = T.schema_id;
WHILE (EXISTS (SELECT * FROM #T)) BEGIN
DECLARE #SQL NVarChar(4000) = 'SELECT * FROM $$TableName WHERE (0 = 1) ';
DECLARE #TableName NVarChar(1000) = (
SELECT TOP 1 SchemaName + '.' + TableName FROM #T
);
SELECT #SQL = REPLACE(#SQL, '$$TableName', #TableName);
DECLARE #Cols NVarChar(4000) = '';
SELECT
#Cols = COALESCE(#Cols + 'OR CONVERT(NVarChar(4000), ', '') + C.name + ') = CONVERT(NVarChar(4000), ''$$MyValue'') '
FROM sys.columns C
WHERE C.object_id = OBJECT_ID(#TableName);
SELECT #Cols = REPLACE(#Cols, '$$MyValue', #MyValue);
SELECT #SQL = #SQL + #Cols;
select substring(#SQL,charindex('.',#SQL)+1,charindex('(',#SQL)-charindex('.',#SQL)-8) as 'TableName'
EXECUTE(#SQL);
DELETE FROM #T
WHERE SchemaName + '.' + TableName = #TableName;
END;
DROP TABLE #T;
This will give you table Name and the entire row from the table which contains the searchstring.
Apart from anwswers mentioned in post : Older Post
1) (using column name) SELECT table_name,table_schema FROM INFORMATION_SCHEMA.COLUMNS WHERE column_name='sort_method';
I hope better you can take dump ( in.sql format ) and you can easily search the content using IDEs like N++.

SQL - How to search for your search string in all the columns of Table and then display the column

I am trying to search my whole database with a search string. For this I need to check tables, views, stored procedures + all data in all tables.
I am able to scucessfully search all the database objects , however there is a little problem when i need to search a table , i need to search the cells , if the search string exists in that particular cell , i need to show the Table Name , All the columns on that particular Row and Column name of the cell where the search string is found .
i was using code by Sorna Kumar Maduraj to get the whole rows , where i am stuck is on to display the columns for where the search string is found
For example
A Table named Searchtable has the searchstring in i searched for (lets assume it to be AdminCd) in a particular row in 2 different columns (Lets say column names are ColumnA, ColumnB)
My output would be Table Name - Row Data - ColumnNames (ColumnA, ColumnB)
My only issue lies is that suppose in a same table , a row contains the string in 2 columns and a different rows contain the searchstring in 4 columns , i need to show the columns differently as the output
the code i used , ill tie it up below
DECLARE #Tablenames VARCHAR(500)
DECLARE #SearchStr NVARCHAR(60)
DECLARE #GenerateSQLOnly Bit = 0
SET #Tablenames = 'TableName'
SET #SearchStr = '%AdminCd%'
SET NOCOUNT ON
DECLARE #MatchFound BIT -- To check wether any match is found or not
SELECT #MatchFound = 0
DECLARE #CheckTableNames Table -- To check the tables that need to be searched
(
Tablename sysname
)
DECLARE #SQLTbl TABLE -- For Table Name
(
Tablename SYSNAME
,WHEREClause VARCHAR(MAX)
,SQLStatement VARCHAR(MAX)
,Execstatus BIT
)
DECLARE #SQL VARCHAR(MAX)
DECLARE #tmpTblname sysname
DECLARE #ErrMsg VARCHAR(100)
IF LTRIM(RTRIM(#Tablenames)) IN ('' ,'%')
BEGIN
INSERT INTO #CheckTableNames
SELECT Name
FROM sys.tables
END
ELSE
BEGIN
SELECT #SQL = 'SELECT ''' + REPLACE(#Tablenames,',',''' UNION SELECT ''') + ''''
INSERT INTO #CheckTableNames
EXEC(#SQL)
END
IF NOT EXISTS(SELECT 1 FROM #CheckTableNames)
BEGIN
SELECT #ErrMsg = 'No tables are found in this database ' + DB_NAME() + ' for the specified filter'
PRINT #ErrMsg
RETURN
END
INSERT INTO #SQLTbl
( Tablename,WHEREClause)
SELECT QUOTENAME(SCh.name) + '.' + QUOTENAME(ST.NAME),
(
SELECT '[' + SC.Name + ']' + ' LIKE ''' + #SearchStr + ''' OR ' + CHAR(10)
FROM SYS.columns SC
JOIN SYS.types STy
ON STy.system_type_id = SC.system_type_id
AND STy.user_type_id =SC.user_type_id
WHERE STY.name in ('varchar','char','nvarchar','nchar','text')
AND SC.object_id = ST.object_id
ORDER BY SC.name
FOR XML PATH('')
)
FROM SYS.tables ST
JOIN #CheckTableNames chktbls
ON chktbls.Tablename = ST.name
JOIN SYS.schemas SCh
ON ST.schema_id = SCh.schema_id
WHERE ST.name <> 'SearchTMP'
GROUP BY ST.object_id, QUOTENAME(SCh.name) + '.' + QUOTENAME(ST.NAME) ;
UPDATE #SQLTbl
SET SQLStatement = 'SELECT * INTO SearchTMP FROM ' + Tablename + ' WHERE ' + substring(WHEREClause,1,len(WHEREClause)-5)
DELETE FROM #SQLTbl
WHERE WHEREClause IS NULL
WHILE EXISTS (SELECT 1 FROM #SQLTbl WHERE ISNULL(Execstatus ,0) = 0)
BEGIN
SELECT TOP 1 #tmpTblname = Tablename , #SQL = SQLStatement
FROM #SQLTbl
WHERE ISNULL(Execstatus ,0) = 0
IF #GenerateSQLOnly = 0
BEGIN
IF OBJECT_ID('SearchTMP','U') IS NOT NULL
DROP TABLE SearchTMP
EXEC (#SQL)
IF EXISTS(SELECT 1 FROM SearchTMP)
BEGIN
SELECT Tablename=#tmpTblname,* FROM SearchTMP
SELECT #MatchFound = 1
END
END
ELSE
BEGIN
PRINT REPLICATE('-',100)
PRINT #tmpTblname
PRINT REPLICATE('-',100)
PRINT replace(#SQL,'INTO SearchTMP','')
END
UPDATE #SQLTbl
SET Execstatus = 1
WHERE Tablename = #tmpTblname
END
IF #MatchFound = 0
BEGIN
SELECT #ErrMsg = 'No Matches are found in this database ' + DB_NAME() + ' for the specified filter'
PRINT #ErrMsg
RETURN
END
SET NOCOUNT OFF
go

How can I inexpensively determine if a column contains only NULL records?

I have a large table with 500 columns and 100M rows. Based on a small sample, I believe only about 50 of the columns contain any values, and the other 450 contain only NULL values. I want to list the columns that contain no data.
On my current hardware, it would take about 24 hours to query every column (select count(1) from tab where col_n is not null)
Is there a less expensive way to determine that a column is completely empty/NULL?
What about this:
SELECT
SUM(CASE WHEN column_1 IS NOT NULL THEN 1 ELSE 0) column_1_count,
SUM(CASE WHEN column_2 IS NOT NULL THEN 1 ELSE 0) column_2_count,
...
FROM table_name
?
You can easily create this query if you use INFORMATION_SCHEMA.COLUMNS table.
EDIT:
Another idea:
SELECT MAX(column_1), MAX(column_2),..... FROM table_name
If result contains value, column is populated. It should require one table scan.
Try this one -
DDL:
IF OBJECT_ID ('dbo.test2') IS NOT NULL
DROP TABLE dbo.test2
CREATE TABLE dbo.test2
(
ID BIGINT IDENTITY(1,1) PRIMARY KEY
, Name VARCHAR(10) NOT NULL
, IsCitizen BIT NULL
, Age INT NULL
)
INSERT INTO dbo.test2 (Name, IsCitizen, Age)
VALUES
('1', 1, NULL),
('2', 0, NULL),
('3', NULL, NULL)
Query 1:
DECLARE
#TableName SYSNAME
, #ObjectID INT
, #SQL NVARCHAR(MAX)
SELECT
#TableName = 'dbo.test2'
, #ObjectID = OBJECT_ID(#TableName)
SELECT #SQL = 'SELECT' + CHAR(13) + STUFF((
SELECT CHAR(13) + ', [' + c.name + '] = ' +
CASE WHEN c.is_nullable = 0
THEN '0'
ELSE 'CASE WHEN ' + totalrows +
' = SUM(CASE WHEN [' + c.name + '] IS NULL THEN 1 ELSE 0 END) THEN 1 ELSE 0 END'
END
FROM sys.columns c WITH (NOWAIT)
CROSS JOIN (
SELECT totalrows = CAST(MIN(p.[rows]) AS VARCHAR(50))
FROM sys.partitions p
WHERE p.[object_id] = #ObjectID
AND p.index_id IN (0, 1)
) r
WHERE c.[object_id] = #ObjectID
FOR XML PATH(''), TYPE).value('.', 'NVARCHAR(MAX)'), 1, 2, ' ') + CHAR(13) + 'FROM ' + #TableName
PRINT #SQL
EXEC sys.sp_executesql #SQL
Output 1:
SELECT
[ID] = 0
, [Name] = 0
, [IsCitizen] = CASE WHEN 3 = SUM(CASE WHEN [IsCitizen] IS NULL THEN 1 ELSE 0 END) THEN 1 ELSE 0 END
, [Age] = CASE WHEN 3 = SUM(CASE WHEN [Age] IS NULL THEN 1 ELSE 0 END) THEN 1 ELSE 0 END
FROM dbo.test2
Query 2:
DECLARE
#TableName SYSNAME
, #SQL NVARCHAR(MAX)
SELECT #TableName = 'dbo.test2'
SELECT #SQL = 'SELECT' + CHAR(13) + STUFF((
SELECT CHAR(13) + ', [' + c.name + '] = ' +
CASE WHEN c.is_nullable = 0
THEN '0'
ELSE 'CASE WHEN '+
'MAX(CAST([' + c.name + '] AS CHAR(1))) IS NULL THEN 1 ELSE 0 END'
END
FROM sys.columns c WITH (NOWAIT)
WHERE c.[object_id] = OBJECT_ID(#TableName)
FOR XML PATH(''), TYPE).value('.', 'NVARCHAR(MAX)'), 1, 2, ' ') + CHAR(13) + 'FROM ' + #TableName
PRINT #SQL
EXEC sys.sp_executesql #SQL
Output 2:
SELECT
[ID] = 0
, [Name] = 0
, [IsCitizen] = CASE WHEN MAX(CAST([IsCitizen] AS CHAR(1))) IS NULL THEN 1 ELSE 0 END
, [Age] = CASE WHEN MAX(CAST([Age] AS CHAR(1))) IS NULL THEN 1 ELSE 0 END
FROM dbo.test2
Results:
ID Name IsCitizen Age
----------- ----------- ----------- -----------
0 0 0 1
Could you check if colums idexing will help you reach some performance improve
CREATE UNIQUE NONCLUSTERED INDEX IndexName ON dbo.TableName(ColumnName)
WHERE ColumnName IS NOT NULL;
GO
SQL server query to get the list of columns in a table along with Data types, NOT NULL, and PRIMARY KEY constraints
Run SQL in best answer of above questions and generate a new query like below.
Select ISNULL(column1,1), ISNULL(column2,1), ISNULL(column3,1) from table
You would not need to 'count' all of the 100M records. When you simply back out of the query with a TOP 1 as soon as you hit a column with a not-null value, would save a lot of time while providing the same information.
500 Columns?!
Ok, the right answer to your question is: normalize your table.
Here's what happening for the time being:
You don't have an index on that column so SQL Server has to do a full scan of your humongous table.
SQL Server will certainly fully read every row (it means every columns even if you're only interested in one).
And since your row are most likely over 8kb... http://msdn.microsoft.com/en-us/library/ms186981%28v=sql.105%29.aspx
Seriously, normalize your table and if needed split it horizontally (put "theme grouped" columns inside separate table, to only read them when you need them).
EDIT: You can rewrite your query like this
select count(col_n) from tab
and if you want to get all columns at once (better):
SELECT
COUNT(column_1) column_1_count,
COUNT(column_2) column_2_count,
...
FROM table_name
If most records are not null maybe you can mix some of the approach suggested (for example check only nullable fields) with this:
if exists (select * from table where field is not null)
this should speed up the search because exists stops the search as soon as condition is met, in this example a single not null record is enough to decide the status of the field.
If the field has an index this should be almost instant.
Normally adding top 1 to this query is not needed because the query optimizer knows that you do not need to retrieve all the matching records.
You can use this stored procedure to the trick You need to provide the table name you wish to query note that if you'll pass to procedure the #exec parameter = 1 it will execute the select query
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[SP_SELECT_NON_NULL_COLUMNS] ( #tablename varchar (100)=null, #exec int =0)
AS BEGIN
SET NOCOUNT ON
IF #tablename IS NULL
RAISERROR('CANT EXECUTE THE PROC, TABLE NAME IS MISSING',16 ,1)
ELSE
BEGIN
IF OBJECT_ID('tempdb..#table') IS NOT NULL DROP TABLE #table
DECLARE #i VARCHAR (max)=''
DECLARE #sentence VARCHAR (max)=''
DECLARE #SELECT VARCHAR (max)
DECLARE #LocalTableName VARCHAR(50) = '['+#tablename+']'
CREATE TABLE #table (ColumnName VARCHAR (max))
SELECT #i+=
' IF EXISTS ( SELECT TOP 1 '+column_name+' FROM ' +#LocalTableName+' WHERE ' +column_name+
' '+'IS NOT NULL) INSERT INTO #table VALUES ('''+column_name+''');'
FROM INFORMATION_SCHEMA.COLUMNS WHERE table_name=#tablename
INSERT INTO #table
EXEC (#i)
SELECT #sentence = #sentence+' '+columnname+' ,' FROM #table
DROP TABLE #table
IF #exec=0
BEGIN
SELECT 'SELECT '+ LTRIM (left (#sentence,NULLIF(LEN (#sentence)-1,-1)))+
+' FROM ' +#LocalTableName
END
ELSE
BEGIN
SELECT #SELECT= 'SELECT '+ LTRIM (left (#sentence,NULLIF(LEN (#sentence)-1,-1)))+
+' FROM '+#LocalTableName
EXEC (#SELECT)
END
END
END
Use it like this:
EXEC [dbo].[SP_SELECT_NON_NULL_COLUMNS] 'YourTableName' , 1

Resources