I've created a dynamic SQL script to generate statements to replace empty strings with NULL for every column in every table in my database. I've stored the script to the variable #SQL.
When I run EXEC #SQL, it generates the following results:
(No column name)
UPDATE [TableX] SET [ColumnA] = NULL WHERE [ColumnA] =''
UPDATE [TableX] SET [ColumnB] = NULL WHERE [ColumnB] =''
UPDATE [TableX] SET [ColumnC] = NULL WHERE [ColumnC] =''
UPDATE [TableY] SET [ColumnA] = NULL WHERE [ColumnA] =''
UPDATE [TableY] SET [ColumnB] = NULL WHERE [ColumnB] =''
UPDATE [TableY] SET [ColumnB] = NULL WHERE [ColumnB] =''
And so on... (there is an inconsistent/unknown number of columns and tables, and therefore an inconsistent/unknown number of results).
My problem is that rather than simply returning these statements as results, I want to actually execute all of the individual statements. I would be very grateful if someone could advise me on the easiest way to do so.
Thanks in advance!
EXEC #SQL will not return the results in your question unless #SQL is the name of a stored procedure. It seems you are actually using EXEC (#SQL) to execute a batch of SELECT statements, each of which returns a single column, single row result with a column containing an UPDATE statement.
Below is example code you can add to the end of your existing script to concatenate the multiple result sets into a single batch of statements for execution.
DECLARE #UpdateStatementsBatch nvarchar(MAX);
DECLARE #UpdateStatementsTable TABLE(UpdateStatement nvarchar(MAX));
INSERT INTO #UpdateStatementsTable
EXEC(#SQL);
SELECT #UpdateStatementsBatch = STRING_AGG(UpdateStatement,N';') + N';'
FROM #UpdateStatementsTable;
EXEC (#UpdateStatementsBatch);
It's probably better to modify your existing code to build the batch of update statements rather than concatenate after the fact but I can't provide an example without seeing your existing code.
Here's a suggestion for something you can try to build a single update statement per table.
Obviously I have no idea what you've built to construct your existing sql but you should be able to tweak to your requirements.
Basically concatenate all columns for all tables where they are a varchar datatype. If they are an empty string, update to null, otherwise retain current value.
I don't know what version of Sql server you have so I've used for xml for compatability, replace with string_agg if supported. Add any additional filtering eg for only specific tables.
with c as (
select c.table_name, cols=Stuff(
(select concat(',',QuoteName(column_name),' = ','iif(', QuoteName(column_name), '='''', NULL, ', QuoteName(column_name), ')',Char(10))
from information_schema.columns c2
where data_type in ('varchar','nvarchar') and c2.table_name=c.table_name
for xml path('')
),1,1,'')
from information_schema.columns c
group by c.table_name
)
select concat('update ', QuoteName(c.table_name), ' set ', cols,';')
from c
where cols is not null
As this is presumably a one-off data fix you can just cut and paste the resulting sql into SSMS and run it.
Alternatively you could add another level of concatenation and exec it if you wanted to make it something you can repeat.
Related
There are ~10 different subquestions that could be answered here, but the main question is in the title. TLDR version: I have a table like the example below and I want to replace all double quote marks across the whole table. Is there a simple way to do this?
My solution using cursor seems fairly straightforward. I know there's some CURSOR hatred in the SQL Server community (bad runtime?). At what point (num rows and/or num columns) would CURSOR stink at this?
Create Reproducible Example Table
DROP TABLE IF EXISTS #example;
CREATE TABLE #example (
NumCol INT
,CharCol NVARCHAR(20)
,DateCol NVARCHAR(100)
);
INSERT INTO #example VALUES
(1, '"commas, terrible"', '"2021-01-01 20:15:57,2021:04-08 19:40:50"'),
(2, '"loadsrc,.txt"', '2020-01-01 00:00:05'),
(3, '".txt,from.csv"','1/8/2021 10:14')
Right now, my identified solutions are:
Manually update for each column UPDATE X SET CharCol = REPLACE(CharCol, '"',''). Horribly annoying to do at any more than 2 columns IMO.
Use a CURSOR to update (similar to annoyingly complicated looking solution at SQL Server- SQL Replace on all columns in all tables across an entire DB
REPLACE character using CURSOR
This gets a little convoluted with all the cursor-related script, but seems to work well otherwise.
-- declare variable to store colnames, cursor to filter through list, string for dynamic sql code
DECLARE #colname VARCHAR(10)
,#sql VARCHAR(MAX)
,#namecursor CURSOR;
-- run cursor and set colnames and update table
SET #namecursor = CURSOR FOR SELECT ColName FROM #colnames
OPEN #namecursor;
FETCH NEXT FROM #namecursor INTO #colname;
WHILE (##FETCH_STATUS <> -1) -- alt: WHILE ##FETCH_STATUS = 0
BEGIN;
SET #sql = 'UPDATE #example SET '+#colname+' = REPLACE('+#colname+', ''"'','''')'
EXEC(#sql); -- parentheses VERY important: EXEC(sql-as-string) NOT EXEC storedprocedure
FETCH NEXT FROM #namecursor INTO #colname;
END;
CLOSE #namecursor;
DEALLOCATE #namecursor;
GO
-- see results
SELECT * FROM #example
Subquestion: While I've seen it in our database elsewhere, for this particular example I'm opening a .csv file in Excel and exporting it as tab delimited. Is there a way to change the settings to export without the double quotes? If I remember correctly, BULK INSERT doesn't have a way to handle that or a way to handle importing a csv file with extra commas.
And yes, I'm going to pretend that I'm fine that there's a list of datetimes in the date column (necessitating varchar data type).
Why not just dynamically build the SQL?
Presumably it's a one-time task you'd be doing so just run the below for your table, paste into SSMS and run. But if not you could build an automated process to execute it - better of course to properly sanitize when inserting the data though!
select
'update <table> set ' +
String_Agg(QuoteName(COLUMN_NAME) + '=Replace(' + QuoteName(column_name) + ',''"'','''')',',')
from INFORMATION_SCHEMA.COLUMNS
where table_name='<table>' and TABLE_SCHEMA='<schema>' and data_type in ('varchar','nvarchar')
example DB<>Fiddle
You might try this approach, not fast, but easy to type (or generate).
SELECT NumCol = y.value('(NumCol/text())[1]','int')
,CharCol = y.value('(CharCol/text())[1]','nvarchar(100)')
,DateCol = y.value('(DateCol/text())[1]','nvarchar(100)')
FROM #example e
CROSS APPLY(SELECT e.* FOR XML PATH('')) A(x)
CROSS APPLY(SELECT CAST(REPLACE(A.x,'"','') AS XML)) B(y);
The idea in short:
The first APPLY will transform all columns to a root-less XML.
Without using ,TYPE this will be of type nvarchar(max) implicitly
The second APPLY will first replace any " in the whole text (which is one row actually) and cast this to XML.
The SELECT uses .value to fetch the values type-safe from the XML.
Update: Just add INTO dbo.SomeNotExistingTableName right before FROM to create a new table with this data. This looks better than updating the existing table (might be a #-table too). I'd see this as a staging environment...
Good luck, messy data is always a pain in the neck :-)
So the user enters the policy number in the form: 2000, 2001, 2002
I need to run a query for each of those 3 policy numbers. I am not sure how to do it.
This is the code I have right now. I was thinking about some sort of string manipulation and then use loop, but I am not sure how to do that. Can anyone please help me?
declare #sql1 varchar(1000)
declare #policy varchar(1000)
set #policy = '2000, 2001, 2002'
--THIS IS WHERE i NEED HELP???
set #policy = replace(#policy, ' ', '')
set #policy = '''' + replace(#policy, ',', ''',''') + ''''
print (#policy)
if #policy <> 'null'
set #sql1 =
(SELECT top 1
[MIL]
FROM
[DataManagement].[dbo].[lookuptable] where [policy] = #policy group by [MIL] )
exec (#sql1)
print(#sql1)
Options:
Are you sure you actually need one results set per item in #policy and not a single result set that matches any of the IDs? An IN query would let you get that result - sadly you can't do WHERE IN (#List), but it can be concatenated into a dynamic query (SQL injection concerns aside - make sure of your data source, but that applies to anything that's taking input like you seem to be.
If you declare #Policy as a user-defined table type variable, then you can pass that in as a list of IDs and simply loop through it. (More info - http://blog.sqlauthority.com/2008/08/31/sql-server-table-valued-parameters-in-sql-server-2008/, https://www.brentozar.com/archive/2014/02/using-sql-servers-table-valued-parameters/)
There's no shortage of examples online of code to split a delimited string list into a table of values. That would let you do the same as I've suggested in point 2.
The FOR XML PATH('') trick can be used to take a table of results and flatten it into a single variable. If you join against your values table and build the SELECT query from your question as the result, you can then do a single EXEC with no need for a loop at all.
Depends what you want for your environment really. I'd use option 2 to split the string, then build a combined single query using FOR XML PATH and execute that. But for some environments the table-valued parameter and loop approach would definitely be superior.
I am trying to code a stored procedure in SQL that does the following
Takes 2 inputs (BatchType and "Column Name").
Searches database and gives the batchdate and the data in the column = "Column name"
Code is as give below
ALTER PROCEDURE [dbo].[chartmilldata]
-- Add the parameters for the stored procedure here
(#BatchType nvarchar (50),
#Data nvarchar(50))
AS
BEGIN
-- Insert statements for procedure here
SELECT BatchDate,#Data FROM --Database-- WHERE BatchType = #BatchType
END
I am trying to select column from the database based on operator input. But I am not getting the output. It would be great if someone can give me a direction.
You may want to build out your SELECT statement as a string then execute it using sp_executesql.
See this page for more info:
https://msdn.microsoft.com/en-us/library/ms188001.aspx
This will allow you to set your query to substitute in your column name via your variable and then execute the statement. Be sure to sanitize your inputs though!
You'd need to use dynamic SQL, HOWEVER I would not recommend this solution, I don't think there is anything I can add as to why I wouldn't recommend it that isn't explained better in Erland Sommarskog in The Curse and Blessings of Dynamic SQL.
Nonetheless, if you had to do it in a stored procedure you could use something like:
ALTER PROCEDURE [dbo].[chartmilldata]
-- Add the parameters for the stored procedure here
(#BatchType nvarchar (50),
#Data nvarchar(50))
AS
BEGIN
-- DECLARE AND SET SQL TO EXECUTE
DECLARE #SQL NVARCHAR(MAX) = N'SELECT BatchDate = NULL, ' +
QUOTENAME(#Data) + N' = NULL;';
-- CHECK COLUMN IS VALID IN THE TABLE
IF EXISTS
( SELECT 1
FROM sys.columns
WHERE name = #Data
AND object_id = OBJECT_ID('dbo.YourTable', 'U')
)
BEGIN
SET #SQL = 'SELECT BatchDate, ' + QUOTENAME(#Data) +
' FROM dbo.YourTable WHERE BatchType = #BatchType;';
END
EXECUTE sp_executesql #SQL, N'#BatchType NVARCHAR(50)', #BatchType;
END
It would probably be advisable to change your input parameter #Data to be NVARCHAR(128) (or the alias SYSNAME) though, since this is the maximum for column names.
I have to create a stored procedure where I will pass tableName, columnName, id as parameters. The task is to select records from the passed table where columnName has passed id. If record is found update records with some fixed data. Also implement Transaction so that we can rollback in case of any error.
There are hundreds of table in database and each table has different schema that is why I have to pass columnName.
Don't know what is the best approach for this. I am trying select records into a temp table so that I can manipulate it as per requirement but its not working.
I am using this code:
ALTER PROCEDURE [dbo].[GetRecordsFromTable]
#tblName nvarchar(128),
#keyCol varchar(100),
#key int = 0
AS
BEGIN
SET NOCOUNT ON;
BEGIN TRY
--DROP TABLE #TempTable;
DECLARE #sqlQuery nvarchar(4000);
SET #sqlQuery = 'SELECT * FROM ' + #tblName + ' WHERE ' + #keyCol + ' = 2';
PRINT #sqlQuery;
INSERT INTO #TempTable
EXEC sp_executesql #sqlQuery,
N'#keyCol varchar(100), #key int', #keyCol, #key;
SELECT * FROM #TempTable;
END TRY
BEGIN CATCH
EXECUTE [dbo].[uspPrintError];
END CATCH;
END
I get an error
Invalid object name '#TempTable'
Also not sure if this is the best approach to get data and then update it.
If you absolutely must make that work then I think you'll have to use a global temp table. You'll need to see if it exists before running your dynamic sql and clean up. With a fixed table name you'll run into problems with other connections. Inside the dynamic sql you'll add select * into ##temptable from .... Actually I'm not even sure why you want the temp table in the first place. Can't the dynamic sql just return the results?
On the surface it seems like a solid idea to have one generic procedure for returning data with a couple of parameters to drive it but, without a lot of explanation, it's just not the way database are designed to work.
You should create the temp table.
IF OBJECT_ID('tempdb..##TempTable') IS NOT NULL
DROP TABLE ##TempTable
CREATE TABLE ##TempTable()
It looks like #temptables created using dynamic SQL via the EXECUTE string method have a different scope and can't be referenced by "fixed" SQLs in the same stored procedure.
However, I can reference a temp table created by a dynamic SQL statement in a subsequence dynamic SQL but it seems that a stored procedure does not return a query result to a calling client unless the SQL is fixed.
A simple 2 table scenario:
I have 2 tables. Let's call them Orders and Items. Order has a Primary key of OrderId and Items has a Primary Key of ItemId. Items.OrderId is the foreign key to identify the parent Order. An Order can have 1 to n Items.
I want to be able to provide a very flexible "query builder" type interface to the user to allow the user to select what Items he want to see. The filter criteria can be based on fields from the Items table and/or from the parent Order table. If an Item meets the filter condition including and condition on the parent Order if one exists, the Item should be return in the query as well as the parent Order.
Usually, I suppose, most people would construct a join between the Item table and the parent Order tables. I would like to perform 2 separate queries instead. One to return all of the qualifying Items and the other to return all of the distinct parent Orders. The reason is two fold and you may or may not agree.
The first reason is that I need to query all of the columns in the parent Order table and if I did a single query to join the Orders table to the Items table, I would be repoeating the Order information multiple times. Since there are typically a large number of items per Order, I'd like to avoid this because it would result in much more data being transfered to a fat client. Instead, as mentioned, I would like to return the two tables individually in a dataset and use the two tables within to populate a custom Order and child Items client objects. (I don't know enough about LINQ or Entity Framework yet. I build my objects by hand). The second reason I would like to return two tables instead of one is because I already have another procedure that returns all of the Items for a given OrderId along with the parent Order and I would like to use the same 2-table approach so that I could reuse the client code to populate my custom Order and Client objects from the 2 datatables returned.
What I was hoping to do was this:
Construct a dynamic SQL string on the Client which joins the orders table to the Items table and filters appropriate on each table as specified by the custom filter created on the Winform fat-client app. The SQL build on the client would have looked something like this:
TempSQL = "
INSERT INTO #ItemsToQuery
OrderId, ItemsId
FROM
Orders, Items
WHERE
Orders.OrderID = Items.OrderId AND
/* Some unpredictable Order filters go here */
AND
/* Some unpredictable Items filters go here */
"
Then, I would call a stored procedure,
CREATE PROCEDURE GetItemsAndOrders(#tempSql as text)
Execute (#tempSQL) --to create the #ItemsToQuery table
SELECT * FROM Items WHERE Items.ItemId IN (SELECT ItemId FROM #ItemsToQuery)
SELECT * FROM Orders WHERE Orders.OrderId IN (SELECT DISTINCT OrderId FROM #ItemsToQuery)
The problem with this approach is that #ItemsToQuery table, since it was created by dynamic SQL, is inaccessible from the following 2 static SQLs and if I change the static SQLs to dynamic, no results are passed back to the fat client.
3 around come to mind but I'm look for a better one:
1) The first SQL could be performed by executing the dynamically constructed SQL from the client. The results could then be passed as a table to a modified version of the above stored procedure. I am familiar with passing table data as XML. If I did this, the stored proc could then insert the data into a temporary table using a static SQL that, because it was created by dynamic SQL, could then be queried without issue. (I could also investigate into passing the new Table type param instead of XML.) However, I would like to avoid passing up potentially large lists to a stored procedure.
2) I could perform all the queries from the client.
The first would be something like this:
SELECT Items.* FROM Orders, Items WHERE Order.OrderId = Items.OrderId AND (dynamic filter)
SELECT Orders.* FROM Orders, Items WHERE Order.OrderId = Items.OrderId AND (dynamic filter)
This still provides me with the ability to reuse my client sided object-population code because the Orders and Items continue to be returned in two different tables.
I have a feeling to, that I might have some options using a Table data type within my stored proc, but that is also new to me and I would appreciate a little bit of spoon feeding on that one.
If you even scanned this far in what I wrote, I am surprised, but if so, I woul dappreciate any of your thoughts on how to accomplish this best.
You first need to create your table first then it will be available in the dynamic SQL.
This works:
CREATE TABLE #temp3 (id INT)
EXEC ('insert #temp3 values(1)')
SELECT *
FROM #temp3
This will not work:
EXEC (
'create table #temp2 (id int)
insert #temp2 values(1)'
)
SELECT *
FROM #temp2
In other words:
Create temp table
Execute proc
Select from temp table
Here is complete example:
CREATE PROC prTest2 #var VARCHAR(100)
AS
EXEC (#var)
GO
CREATE TABLE #temp (id INT)
EXEC prTest2 'insert #temp values(1)'
SELECT *
FROM #temp
1st Method - Enclose multiple statements in the same Dynamic SQL Call:
DECLARE #DynamicQuery NVARCHAR(MAX)
SET #DynamicQuery = 'Select * into #temp from (select * from tablename) alias
select * from #temp
drop table #temp'
EXEC sp_executesql #DynamicQuery
2nd Method - Use Global Temp Table:
(Careful, you need to take extra care of global variable.)
IF OBJECT_ID('tempdb..##temp2') IS NULL
BEGIN
EXEC (
'create table ##temp2 (id int)
insert ##temp2 values(1)'
)
SELECT *
FROM ##temp2
END
Don't forget to delete ##temp2 object manually once your done with it:
IF (OBJECT_ID('tempdb..##temp2') IS NOT NULL)
BEGIN
DROP Table ##temp2
END
Note: Don't use this method 2 if you don't know the full structure on database.
I had the same issue that #Muflix mentioned. When you don't know the columns being returned, or they are being generated dynamically, what I've done is create a global table with a unique id, then delete it when I'm done with it, this looks something like what's shown below:
DECLARE #DynamicSQL NVARCHAR(MAX)
DECLARE #DynamicTable VARCHAR(255) = 'DynamicTempTable_' + CONVERT(VARCHAR(36), NEWID())
DECLARE #DynamicColumns NVARCHAR(MAX)
--Get "#DynamicColumns", example: SET #DynamicColumns = '[Column1], [Column2]'
SET #DynamicSQL = 'SELECT ' + #DynamicColumns + ' INTO [##' + #DynamicTable + ']' +
' FROM [dbo].[TableXYZ]'
EXEC sp_executesql #DynamicSQL
SET #DynamicSQL = 'IF OBJECT_ID(''tempdb..##' + #DynamicTable + ''' , ''U'') IS NOT NULL ' +
' BEGIN DROP TABLE [##' + #DynamicTable + '] END'
EXEC sp_executesql #DynamicSQL
Certainly not the best solution, but this seems to work for me.
I would strongly suggest you have a read through http://www.sommarskog.se/arrays-in-sql-2005.html
Personally I like the approach of passing a comma delimited text list, then parsing it with text to table function and joining to it. The temp table approach can work if you create it first in the connection. But it feel a bit messier.
Result sets from dynamic SQL are returned to the client. I have done this quite a lot.
You're right about issues with sharing data through temp tables and variables and things like that between the SQL and the dynamic SQL it generates.
I think in trying to get your temp table working, you have probably got some things confused, because you can definitely get data from a SP which executes dynamic SQL:
USE SandBox
GO
CREATE PROCEDURE usp_DynTest(#table_type AS VARCHAR(255))
AS
BEGIN
DECLARE #sql AS VARCHAR(MAX) = 'SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = ''' + #table_type + ''''
EXEC (#sql)
END
GO
EXEC usp_DynTest 'BASE TABLE'
GO
EXEC usp_DynTest 'VIEW'
GO
DROP PROCEDURE usp_DynTest
GO
Also:
USE SandBox
GO
CREATE PROCEDURE usp_DynTest(#table_type AS VARCHAR(255))
AS
BEGIN
DECLARE #sql AS VARCHAR(MAX) = 'SELECT * INTO #temp FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = ''' + #table_type + '''; SELECT * FROM #temp;'
EXEC (#sql)
END
GO
EXEC usp_DynTest 'BASE TABLE'
GO
EXEC usp_DynTest 'VIEW'
GO
DROP PROCEDURE usp_DynTest
GO