I am trying to create a generic update procedure. The point of this procedure is that we want to be able to track everything that happens in a table. If a recordis updated, we need to be able to know who changed that record, what it was originally, what it is after the change and when the change occurred. We only do this on our most important tables where accountability is a must.
Right now, we do this through a combination of web server programming and SQL Server commands.
I need to take what we currently have, and make a SQL only version.
So, here are the requirements of what I need:
The original sp is called UpdateWithHistory. Right now, it takes 4 parameters all varchar (or it can be nvarchar, doesn't matter). They are the table name, the primary key field, primary key value and a comma delimited list of fields and values in the format field='value',field1='value1'...etc.
In the background, we have a mapping table that we use to map the string table names to actual tables.
In the stored procedure, I have tried various combinations of OPENROWSET, exec(), select into, xml, and other methods. None seem to work.
So basically, I have to be able to dynamically generate a simple select statement (no joins or other complicated select stuff) from the 4 supplied parameters, then store the results of that query in a table. Since it is dynamic, I don't know the number of fields being queried, or what data types they will be.
I tried select into since that will automatically create a table with the appropriate fields and data types, but it doesn't work in conjunction with the exec command. I have also tried
exec sp_executeSQL #SQL, N'#output xml output', #resultXML output
Where #resultXML is XML datatype and #SQL is the sql command. #resultXML always ends up as null, no matter what I do. I also tried the xml route because I know that "FOR XML Path" always returns one column, but I can't use that in an insert into statement....
That statement output will be used to determine the original values before the update.
I figure once I get past this hurdle the rest will be a piece of cake. Anyone got any ideas?
So here is code for something that I finally got to work, although I don't want to use global tables, so I would gladly accept a different answer...
DECLARE #curRecordString varchar(max) = 'SELECT * into ##TEMP_TABLE FROM SOMEDB.dbo.' + #tbl + ' WHERE ' + #prikey + ' = ''' + #prival + ''' '
exec(#curRecordString)
Basically, as stated before, I need to dynamically build a sql query, then store the result of running the query so that I can access it later. I would prefer to store it as XML datatype, since I will later be using XQuery to parse and compare nodes. In the code above, I am using a global temp table (not ideal, I know) to store the result of the query so that the rest of my procedure can access the data.
Like I said, I don't like this approach but hopefully someone else can come up with something better that will allow me to dynamically build a SQL query, run it, store the results so that I can access the results later in the stored procedure.
This is most definitely a hack, but...
DECLARE #s VARCHAR(MAX)
SET #s = 'SELECT (SELECT 1 as splat FOR XML PATH) a'
CREATE TABLE #save (x XML)
INSERT INTO #save
( x )
EXEC (#s)
SELECT * FROM #save s
DROP TABLE #save
Related
I have inherited a bunch of stored procedures basically as a shell and inside the quotes is this huge dynamic SQL with lots of conditions, calculations and case statements, however the table name in the FROM clause within this dynamic SQL changes every quarter.
Now before I get flamed, I like to simply say that I inherited them, how it was designed was before me. So each quarter when a call is made out to these stored procedures, it comes with the actual table name passed as a parameter and then the dynamic SQL concatenates the table name.
The problem with this approach is that, with each run over time, the prior designers simply tacked on more criteria as conditions and calculations. But the dynamic SQL string has a length limit to it. Further it becomes quite difficult to maintain and debug.
CREATE PROCEDURE .....
#dynSQL1 = 'SELECT......
FROM' + strTblName + '
WHERE.....
GROUP BY....'
...
EXEC #dynSQL1
GO
However, I like to ask you all, is there a way to turn this stored procedure with this huge dynamic SQL string into a plain vanilla stored procedure based on a parameterized table name?
My main goal is two fold, one, get away from the long string as dynamic SQL and two, easier maintenance and debugging. I would like to think in the more current version of SQL Server from SQL Server 2016/2017 and on, this issue is addressed.
Your thoughts and suggestions is greatly appreciated.
~G
So each quarter when a call is made out to these stored procedures, it comes with the actual table name passed as a parameter and then the dynamic SQL concatenates the table name.
You could change the procedure to codegen other stored procedures instead of running dynamic SQL. EG:
CREATE PROCEDURE admin.RegenerateProcedures #tableName
as
begin
declare #ddl nvarchar(max) = '
create or alter procedure dbo.SomeProc
as
begin
SELECT......
FROM dbo.' + quotename(#tableName) + '
WHERE.....
GROUP BY....
end
'
EXEC ( #ddl )
. . .
end
GO
My desired end result is to simply be able to SELECT from a Stored Procedure. I've searched the Internet and unfortunately the Internet said this can't be done and that you first need to create a Temp Table to store the data. My problem is that you must first define the columns in the Temp Table before Executing the STORED Procedure. This is just time consuming. I simply want to take the data from the stored procedure and just stick it into a Temp Table.
What is the FASTEST route to achieve this from a coding perspective? To put it simply it's time consuming to first have to lookup the returned fields from a Stored Procedure and then write them all out.
Is there some sort of tool that can just build the CREATE Table Statement based on the Stored Procedure? See screenshot for clarification.
Most of the Stored Procedures I'm dealing with have 50+ fields. I don't look forward to defining each of these fields manually.
Here is good SO Post that got me this far but not what I was hoping. This still takes too much time. What are experienced SQL Server guys doing? I've only just recently made the jump from Oracle to SQL Server and I see that Temp Tables are a big deal in SQL Server from what I can tell.
You have several options to ease your task. However, these won't be fully automatic. Be aware that these won't work if there's dynamic sql in the procedure's code. You might be able to format the result from the functions to increase the automation allowing you to copy and paste easily.
SELECT * FROM sys.dm_exec_describe_first_result_set_for_object(OBJECT_ID('report.MyStoredProcedureWithAnyColumns'), 0) ;
SELECT * FROM sys.dm_exec_describe_first_result_set(N'EXEC report.MyStoredProcedureWithAnyColumns', null, 0) ;
EXEC sp_describe_first_result_set #tsql = N'EXEC report.MyStoredProcedureWithAnyColumns';
GO
If you don't mind ##temp table and some dynamic SQL
NOTE: As Luis Cazares correctly pointed out... the ##temp runs the risk of collision due to concurrency concerns
Example
Declare #SQL varchar(max) = 'Exec [dbo].[prc-App-Lottery-Search] ''8117'''
Declare #temp varchar(500) = '##myTempTable'
Set #SQL = '
If Object_ID(''tempdb..'+#temp+''') Is Not NULL Drop Table '+#temp+';
Create Table '+#temp+' ('+stuff((Select concat(',',quotename(Name),' ',system_type_name)
From sys.dm_exec_describe_first_result_set(#SQL,null,null ) A
Order By column_ordinal
For XML Path ('')),1,1,'') +')
Insert '+#temp+' '+#SQL+'
'
Exec(#SQL)
Select * from ##myTempTable
SO,
I am trying to find a (messy?) solution to an even more messy problem. I have a SQL Server 2014 database which, in part, stores data from another software package but also stores data for me. The software creates a table with specific fields for each set of data - a Name and a Geometry field. For example, one might contain cities (dtCitiesData), another contains roads (dtRoadsData), another contains states(dtStates), etc. I also have a table (dtSpatialDataTables) which stores the names of the tables which store the data I want. That table only has 2 fields: ID and TableName.
I would like to create a SELECT statement which queries dtSpatialDataTables for all entries, then queries all tables with the name corresponding to each TableName result, and SELECTs Name and Geometry from them.
In pseudocode, effectively I want to do this:
SELECT TableName FROM dtSpatialDataTables
FOREACH TableName :
SELECT Name, Geometry FROM (TableName)
I can do this easily PHP via a first query against dtSpatialDataTables and then a loop of queries to each of the returned row TableNames but I want to know if this is possible via SQL directly.
In reality, what I want to do is create a VIEW with this query so I can directly query the VIEW rather than soak of processing time on potentially lots of queries.
Is this possible? Unfortunately, my Google-ing doesn't turn up any meaningful results.
Thanks everyone!
PS: I figure this is messy and not the way this should be done. But I have no choice in how the software puts data in my database. I simply have to use what I get. So... whether this is the "right" way or the "wrong" way, I need a solution. :)
you could do something like this using dynamic sql..
CREATE PROCEDURE dbo.usp_SpatialData_GetByID
(
#ID INT
)
AS
BEGIN
DECLARE #SQL NVARCHAR(MAX),
#Selects NVARCHAR(MAX) = 'SELECT Name, Geometry, ''<<TableName>>'' AS Source FROM <<TableName>>'
SELECT #SQL = COALESCE(#SQL + ' UNION ALL ', '') + REPLACE(#Selects, '<<TableName>>', TableName)
FROM dtSpatialDataTables
WHERE ID = #ID
EXEC(#SQL)
END
GO
I feel like you left out filtering of the Geometry tables somewhere so you might have to add a filter to the #Selects statement
Is there a way to know if the data in a SQL Server 2008 R2 table has changed since the last time you used it? I would like to know of any type of change -- whether a new record has been inserted or an existing one has been modified or deleted.
I am not interested in what the particular change might have been. I am only interested in a boolean value that indicates whether or not the table data has been changed.
Finally, I want a simple solution that does not involve writing a trigger for each CRUD operation and then have that trigger update some other log table.
I have C# program that is meant to insert a large amount of initial data into some database tables. This is a one off operation that is supposed to happen only once, or rarely ever again if ever, in the life of the application. However, during development and testing, though, we use this program a lot.
Currently, with about the 10 tables that it inserts data into, each having about 21,000 rows per table, the program takes about 45 seconds to run. This isn't really a huge problem as this is a one-off operation that is anyway going to be done internally before shipping the product to the customer.
Still, I would like to minimize this time. So, I want to not insert data into a table if there has been no change in the table data since my program last used it.
My colleague told me that I could use the CHECKSUM_AGG function in T-SQL. My question(s) are:
1) If I compute the CHECKSUM_AGG(Cast(NumericPrimaryKeyIdColumn AS int)), then the checksum only changes if a new row has been added or an existing one deleted, right? If someone has only modified values of other columns of an existing row in the table, that will have no impact on the checksum aggregate of the ID column, right? Or will it?
2) Is there another way I can solve the problem of knowing whether table data has changed since the last time my program used it?
This is very close to what I already had in mind and what #user3003007 mentioned.
One way I am thinking of is to take a CHECKSUM(*) or CHECKSUM(Columns, I, Am, Interested, In) for each such table and then do an aggregate checksum on the checksum of each row, like so:
SELECT CHECKSUM_AGG(CAST(CHECKSUM(*) as int)) FROM TableName;
This is still not a reliable method as CHECKSUM does not work on some data types. So, if I have my column of type text or ntext, the CHECKSUM will fail.
Fortunately for me, I do not have such data types in the list of columns I am interested in, so this works for me.
Have you investigated Change Data Capture?
You can use a combination of hashing and checksum_agg. The below will work as long as the the string values do not overflow the HASHBYTES function. It works by converting all of the columns to strings, concatenating those, hashing the concatenated string, turning the hash into an integer, placing all of those values into a temp table, and then running checksum_agg on the temp table. Could easily be adapted to iterate across all real tables
Edit: Combining MD5 and checksum_agg looks like it works at least for somewhat narrow tables:
declare #tablename sysname
set #tablename = 'MyTableName'
declare #sql varchar(max)
set #sql = 'select convert(int,HASHBYTES(''MD5'','''''
declare c cursor for
select column_name
from INFORMATION_SCHEMA.COLUMNS
where table_name = #tablename
open c
declare #cname sysname
fetch next from c into #cname
while ##FETCH_STATUS = 0
begin
set #sql = #sql + '+ coalesce(convert(varchar,' + #cname + '),'')'
fetch next from c into #cname
end
close c
deallocate c
set #sql = #sql + ')) as CheckSumVal
into ##myresults from ' + #tablename
print #sql
exec(#sql)
select CHECKSUM_AGG(CheckSumVal) from ##myresults
drop table ##myresults
How do you know that the change was made by you or that the change is relevant to your needs? If you're not going to do it properly (delete & re-insert or merge) then the whole thing sounds futile to me.
In any case, if you spend only an hour researching, implementing and testing your change, you'd have to run it 80 times (and sit and watch it) before you've broken even on your time. So why bother?
Add Extra column like last_updated default getdate()
Add Extra column int type .
Declare enum (enable Flag attribute option to perform bit wise
operation ).
Then you can apply checksum on this column.
No datatype problem .
An easy way to check this is to use the system DMVs to check the index usage stats, the first index on the table (id 1) is either the heap or the clustered index of the table itself and so can be used for checking when the last update occurred:
SELECT DB_NAME(database_id) AS [database_name] ,
OBJECT_NAME([object_id], [database_id]) AS [index_name] ,
[user_seeks] ,
[user_scans] ,
[user_lookups] ,
[user_updates] ,
[last_user_seek] ,
[last_user_scan] ,
[last_user_lookup] ,
[last_user_update]
FROM sys.dm_db_index_usage_stats
WHERE [index_id] = 1
From this, you can see the last time that the table was updated as well as how many updates there have been (I have left in the seeks and scans etc just in case you're interested).
It's worth taking note that this data does not persist after a reboot, but it's pretty simple to load it into a permanent table every now and then in order to make the data permanent.
simple problem, but perhaps no simple solution, at least I can't think of one of the top of my head but then I'm not the best at finding the best solutions.
I have a stored proc, this stored proc does (in a basic form) a select on a table, envision this:
SELECT * FROM myTable
okay, simple enough, except the table name it needs to search on isn't known, so we ended up with something pretty similiar to this:
-- Just to give some context to the variables I'll be using
DECLARE #metaInfoID AS INT
SET #metaInfoID = 1
DECLARE #metaInfoTable AS VARCHAR(200)
SELECT #metaInfoTable = MetaInfoTableName FROM MetaInfos WHERE MetaInfoID = #MetaInfoID
DECLARE #sql AS VARCHAR(200)
SET #sql = 'SELECT * FROM ' + #metaInfoTable
EXEC #sql
So, I, recognize this is ultimately bad, and can see immediately where I can perform a sql injection attack. So, the question is, is there a way to achieve the same results without the construction of the dynamic sql? or am I going to have to be super, super careful in my client code?
You have to use dynamic sql if you don't know the table name up front. But yes, you should validate the value before attempting to use it in an SQL statement.
e.g.
IF EXISTS(SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME=#metaInfoTable)
BEGIN
-- Execute the SELECT * FROM #metaInfoTable dynamic sql
END
This will make sure a table with that name exists. There is an overhead to doing this obviously as you're querying INFORMATION_SCHEMA. You could instead validate the #metaInfoTable contains only certain characters:
-- only run dynamic sql if table name value contains 0-9,a-z,A-Z, underscores or spaces (enclose table name in square brackets, in case it does contain spaces)
IF NOT #metaInfoTable LIKE '%^[0-9a-zA-Z_ ]%'
BEGIN
-- Execute the SELECT * FROM #metaInfoTable dynamic sql
END
Given the constraints described, I'd suggest 2 ways, with slight variations in performance an architecture.
Choose At the Client & Re-Architect
I'd suggest that you should consider a small re-architecture as much as possible to force the caller/client to decide which table to get its data from. It's a code smell to hold table names in another table.
I am taking an assumption here that #MetaInfoID is being passed from a webapp, data access block, etc. That's where the logic of which table to perform the SELECT on should be housed. I'd say that the client should know which stored procedure (GetCustomers or GetProducts) to call based on that #MetaInfoID. Create new method in your DAL like GetCustomersMetaInfo() and GetProductsMetaInfo() and GetInvoicesMetaInfo() which call into their appropriate sprocs (with no dynamic SQL needed, and no maintenance of a meta table in the DB).
Perhaps try to re-architect the system a little bit.
In SQL Server
If you absolutely have to do this lookup in the DB, and depending on the number of tables that you have, you could perform a handful of IF statements (as many as needed) like:
IF #MetaInfoID = 1
SELECT * FROM Customers
IF #MetaInfoID =2
SELECT * FROM Products
-- etc
That would probably become to be a nightmare to maintain.
Perhaps you could write a stored procedure for each MetaInfo. In this way, you gain the advantage of pre-compilation, and no SQL injection can occur here. (imagine if someone sabotaged the MetaInfoTableName column)
IF #MetaInfoID = 1
EXEC GetAllCustomers
IF #MetaInfoID = 2
EXEC GetAllProducts