We cannot create any additional column
Please keep that in mind
The whole intention of this script is to merge data into my temp table
when the data is matching don't have to do any thing.
if some data is present in #temp_cqm_class_template_xref and not in cqm_class_template_xref_temp then those data's has to be deleted from #temp_cqm_class_template_xref table
if it is the other way it has to be inserted into the #temp_cqm_class_template_xref table
IF OBJECT_ID('tempdb..#temp_cqm_class_template_xref') IS NOT NULL
DROP TABLE #temp_cqm_class_template_xref;
CREATE TABLE #temp_cqm_class_template_xref (
[template_name] [VARCHAR](30) NOT NULL
,[measure_id] [INT] NOT NULL
,[cqm_item_mstr_id] [INT] NOT NULL
,[created_by] [INT] NOT NULL
,[modified_by] [INT] NOT NULL
,[create_timestamp] [DATETIME] NOT NULL
,[modify_timestamp] [DATETIME] NOT NULL
);
MERGE INTO #temp_cqm_class_template_xref AS t
USING cqm_class_template_xref_temp AS s
ON (
t.template_name = s.template_name
AND t.measure_id = s.measure_id
AND t.cqm_item_mstr_id = s.cqm_item_mstr_id
)
WHEN NOT MATCHED
THEN
INSERT (
template_name
,measure_id
,cqm_item_mstr_id
,created_by
,modified_by
,create_timestamp
,modify_timestamp
)
VALUES (
s.template_name
,s.measure_id
,s.cqm_item_mstr_id
,s.created_by
,s.modified_by
,s.create_timestamp
,s.modify_timestamp
)
WHEN NOT MATCHED BY target
THEN
DELETE;
When I run this script, I get the following error:
Msg 10711, Level 15, State 1, Procedure
ngdev_cqm_class_template_xref_bcp_upld, Line 88 An action of type
'INSERT' is not allowed in the 'WHEN MATCHED' clause of a MERGE
statement
Merge is not a good technique to use. see:
https://www.mssqltips.com/sqlservertip/3074/use-caution-with-sql-servers-merge-statement/
Merge is hard to debug and very hard to maintain later when you have problems with the data that it is trying to merge. Don't ever use it.
Instead write an insert using a select instead of a values clause and write a delete.
You may try something like left join and figure out what columns are null
in the opposite table;
According to your logic This might help you
update t
set t.template_name=null,
t.measure_id = null,
t.cqm_item_mstr_id = null
#temp_cqm_class_template_xref t Left join cqm_class_template_xref_temp s on
t.template_name = s.template_name
AND t.measure_id = s.measure_id
AND t.cqm_item_mstr_id = s.cqm_item_mstr_id
delete from #temp_cqm_class_template_xref
where measure_id is null and cqm_item_mstr_id is null and and template_name is null
Related
I need to create a sequence in the database that cannot be using sequence or identity.
There is a table in the database called File where all the files that users send in different areas of the system are stored.
It contains the id (primary key), name, type, folder, number, hash...
CREATE TABLE dbo.[File]
(
FileId uniqueidentifier NOT NULL,
Name nvarchar(30) NOT NULL,
FileTypeId int NOT NULL,
FileFolderId int NOT NULL,
Number int NOT NULL,
Hash nvarchar(50) NOT NULL
...
) ON [PRIMARY]
And then for each feature there is a table expanding the properties of the File table, an example is ContractFile.
It has the same id of the File table and with a few more fields and the id of the Contract table, creating the relation.
CREATE TABLE dbo.ContractFile
(
FileId uniqueidentifier NOT NULL,
ContractId uniqueidentifier NOT NULL
...
) ON [PRIMARY]
So the filename should follow a pattern.
050#H4G5H4G244#001.pdf
050#H4G5H4G244#002.pdf
060#H4G5H4G244#001.pdf
The first 3 digits is a code that is in the FileType table.
The digits in the middle is the code in the Contract table.
And the last 3 is the sequence that was inserted.
Then it groups the string by the FileType and the Contract.
So I created a trigger in the ContractFile table for when inserting it get the biggest number for that FileType and for the Contract and add +1, setting the Number field of the File table.
Then the file name is updated (in the same trigger)
CREATE TRIGGER [dbo].[tgContractFileInsert]
ON [dbo].[ContractFile]
FOR INSERT
AS
BEGIN
SET NOCOUNT ON
UPDATE dbo.File
SET Number = COALESCE(
(SELECT MAX(AR.Number)
FROM dbo.ContractFile NOA
INNER JOIN dbo.File AR
ON AR.FileId = NOA.FileId
WHERE NOA.ContractId = I.ContractId AND
AR.FileTypeId = T.FileTypeId
),
0) + 1
FROM dbo.File T WITH (XLOCK)
INNER JOIN Inserted I
ON I.FileId = T.FileId
WHERE T.Number IS NULL
UPDATE dbo.File
SET Name = dbo.fnFileName(AP.Code, NOB.Code, T.Numero, T.Name)
FROM dbo.File T
INNER JOIN Inserted I
ON I.FileId = T.FileId
INNER JOIN dbo.FileType AP
ON AP.FileTypeId = T.FileTypeId
INNER JOIN dbo.Contract NOB
ON NOB.ContractId = I.ContractId
END
At first it works, but when we have a large volume being inserted, there is a deadlock.
And from what I'm seeing also when inserting more than one record will end up getting the same number, since the Inserted table will bring two records and the +1 is not checking this.
How could I solve this? What is the best way?
Avoid deadlock, will the sequence be correct even inserting more than one record at a time and have a good performance?
I'm stuck trying to figure out how to get one of the MERGE statements to work. See below code snippet:
DECLARE #PipelineRunID VARCHAR(100) = 'testestestestest'
MERGE [TGT].[AW_Production_Culture] as [Target]
USING [SRC].[AW_Production_Culture] as [Source]
ON [Target].[MD5Key] = [Source].[MD5Key]
WHEN MATCHED AND [Target].[MD5Others] != [Source].[MD5Others]
THEN UPDATE SET
[Target].[CultureID] = [Source].[CultureID]
,[Target].[ModifiedDate] = [Source].[ModifiedDate]
,[Target].[Name] = [Source].[Name]
,[Target].[MD5Others] = [Source].[MD5Others]
,[Target].[PipelineRunID] = #PipelineRunID
WHEN NOT MATCHED BY TARGET THEN
INSERT VALUES (
[Source].[AW_Production_CultureKey]
,[Source].[CultureID]
,[Source].[ModifiedDate]
,[Source].[Name]
,#PipelineRunID
,[Source].[MD5Key]
,[Source].[MD5Others]);
When I try and run this query I receive the following error:
Msg 257, Level 16, State 3, Line 16
Implicit conversion from data type varchar to varbinary is not allowed. Use the CONVERT function to run this query.
The only VARBINARY column types are MD5Key and MD5Others. As they are both linked to their corresponding columns I don't understand why my error message indicates there is a VARCHAR problem involved. Does anybody understand how and why I should use a CONVERT() function here?
Thanks!
--EDIT: Schema definitions
CREATE VIEW [SRC].[AW_Production_Culture]
WITH SCHEMABINDING
as
SELECT
CAST(CONCAT('',[CultureID]) as VARCHAR(100)) as [AW_Production_CultureKey]
,CAST(HASHBYTES('MD5',CONCAT('',[CultureID])) as VARBINARY(16)) as [MD5Key]
,CAST(HASHBYTES('MD5',CONCAT([ModifiedDate],'|',[Name])) as VARBINARY(16)) as [MD5Others]
,[CultureID],[ModifiedDate],[Name]
FROM
[SRC].[tbl_AW_Production_Culture]
CREATE TABLE [TGT].[AW_Production_Culture](
[AW_Production_CultureKey] [varchar](100) NOT NULL,
[CultureID] [nchar](6) NULL,
[ModifiedDate] [datetime] NULL,
[Name] [nvarchar](50) NULL,
[MD5Key] [varbinary](16) NOT NULL,
[MD5Others] [varbinary](16) NOT NULL,
[RecordValidFrom] [datetime2](7) GENERATED ALWAYS AS ROW START NOT NULL,
[RecordValidUntil] [datetime2](7) GENERATED ALWAYS AS ROW END NOT NULL,
[PipelineRunID] [varchar](36) NOT NULL,
PRIMARY KEY CLUSTERED
(
[MD5Key] ASC
)WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY],
PERIOD FOR SYSTEM_TIME ([RecordValidFrom], [RecordValidUntil])
) ON [PRIMARY]
WITH
(
SYSTEM_VERSIONING = ON ( HISTORY_TABLE = [TGT].[AW_Production_Culture_History] )
)
Reposting my comment as an answer for the sweet, sweet, internet points:
You're getting that error because your varbinary value is being inserted into a varchar column. As your columns have the correct types already then it means your INSERT clause has mismatched columns.
As it is, your MERGE statement is not explicitly listing the destination columns - you should always explicitly list columns in production code so that your DML queries won't break if columns are added or reordered or marked HIDDEN.
So to fix this, change your INSERT clause to explicitly list destination column names.
Also, when using MERGE you should use HOLDLOCK (Or a more suitable lock, if applicable) - otherwise you’ll run into concurrency issues. MERGE is not concurrency-safe by default!
Minor nit-picks that are largely subjective:
I personally prefer avoiding [escapedName] wherever possible and prefer using short table aliases.
e.g. use s and t instead of [Source] and [Target].
"Id" (for "identity" or "identifier") is an abbreviation, not an acronym - so it should be cased as Id and not ID.
Consider using an OUTPUT clause to help diagnose/debug issues too.
So I'd write it like so:
DECLARE #PipelineRunId VARCHAR(100) = 'testestestestest'
MERGE INTO
tgt.AW_Production_Culture WITH (HOLDLOCK) AS t
USING
src.AW_Production_Culture AS s ON t.MD5Key = s.MD5Key
WHEN MATCHED AND t.MD5Others != s.MD5Others THEN UPDATE SET
t.CultureId = s.CultureId,
t.ModifiedDate = s.ModifiedDate,
t.Name = s.Name,
t.MD5Others = s.MD5Others,
t.PipelineRunID = #PipelineRunId
WHEN NOT MATCHED BY TARGET THEN INSERT
(
AW_Production_CultureKey,
CultureId,
ModifiedDate,
[Name],
PipelineRunId,
MD5Key,
MD5Others
)
VALUES
(
s.AW_Production_CultureKey,
s.CultureId,
s.ModifiedDate,
s.[Name],
#PipelineRunId,
s.MD5Key,
s.MD5Others
)
OUTPUT
$action AS [Action],
inserted.*,
deleted.*;
I was using Microsoft SQL Server. I want to use both functions to parse data going into my table. So I use cross apply and outer apply together.
CROSS APPLY CA_Parse_CorpActnDtls_fn(MessageID) ent
outer apply CA_Parse_CorpActnOptnDtls_fn(ev.MessageID) cod
But when I did this it complain about the following erro:
Violation of PRIMARY KEY constraint 'PK_AfterParse_CA_Events'. Cannot insert duplicate key in object 'dbo.AfterParse_CA_Events'. The duplicate key value is (105818432, 37819929).
The statement has been terminated.
The whole T-sql code looks like :
insert into AfterParse_CA_Events (
EventID
,MessageID
,cdtprFunction
,CreationDate
,MsgDefIdr
,EventType
,CFI
,EventProcessingType
,MndtryVlntryEvtTp
,RecordDate
,EffectiveDate
,DueBillRdmDate
,CUSIP
,LSCI_DateOfRecord
,RoundingDesc
)
SELECT ent.EventID
,ent.MessageID
,ent.cdtprFunction
,ent.CreationDate
,ent.MsgDefIdr
,ent.EventType
,ent.CFI
,ent.EventProcessingType
,ent.MndtryVlntryEvtTp
,ent.RecordDate
,ent.EffectiveDate_Cmpny
,ent.DueBillRdmDate
,ent.CUSIP
,ROXSQL.dbo.GetNthTradeDay_fn(
case when ent.EventProcessingType = 'DISN'
then COALESCE (ent.ExDividendDate, ent.RecordDate)
ELSE COALESCE(ent.EffectiveDate_Xchg, ent.EffectiveDate_Cmpny,cod.EarliestPaymentDate_Secu,cod.PaymentDate_Secu ,cod.PaymentDate_Cash)
END,-1) AS LSCI_DateOfRecord
,cod.RoundingDesc
FROM #EventsToDo ev
CROSS APPLY CA_Parse_CorpActnDtls_fn(MessageID) ent
outer apply CA_Parse_CorpActnOptnDtls_fn(ev.MessageID) cod
you can see that I need the second function CA_Parse_CorpActnOptnDtls_fn(ev.MessageID)
Because I want to compose an LSCI_DateOfRecord data using my user defined function. so is there any way to avoid the duplicate when I using the two functions together?
or is there any ways to build a temp list for the LSCI_DateOfRecord and RoundingDesc from the second function CA_Parse_CorpActnOptnDtls_fn(ev.MessageID) separately? And then I can update the table.
Any help is greatly appreciated.
Looking at the error and your SQL code, it's not APPLY operators causing the problem per se. It's the fact that one or both of the functions is returning more than a single row for set of EvenID & MessageID, and THAT is what's causing the PK violation.
Below is a simplified demonstration, using a string splitter function (DelimitedSplit8K)
IF OBJECT_ID('tempdb..#EventsToDo ', 'U') IS NOT NULL
DROP TABLE #EventsToDo ;
GO
CREATE TABLE #EventsToDo (
EventID BIGINT NOT NULL,
MessageID BIGINT NOT NULL,
MessageText VARCHAR(1000) NOT NULL
);
GO
INSERT #EventsToDo (EventID, MessageID, MessageText) VALUES
(105818432, 37819929, 'Part 1,Part 2,Part 3,Part 4,Part 5');
GO
-----------------------------------------------------------------
-- create the AfterParse_CA_Events table with PRIMARY KEY (EvenID, MessageID)...
IF OBJECT_ID('tempdb..#AfterParse_CA_Events', 'U') IS NOT NULL
DROP TABLE #AfterParse_CA_Events;
GO
CREATE TABLE #AfterParse_CA_Events (
EvenID BIGINT NOT NULL,
MessageID BIGINT NOT NULL,
MessagePart VARCHAR(1000) NULL
PRIMARY KEY (EvenID, MessageID)
);
GO
--===============================================================
-- see what happens when we try to insert the parsed message values
-- into AfterParse_CA_Events while it has a PK of (EvenID, MessageID)...
INSERT #AfterParse_CA_Events (EvenID, MessageID, MessagePart)
SELECT
etd.EventID,
etd.MessageID,
dsk.Item
FROM
#EventsToDo etd
CROSS APPLY dbo.DelimitedSplit8K(etd.MessageText, ',') dsk;
GO
--===============================================================
-- execute the code below in a separate execution
--===============================================================
-- now, let's modify the AfterParse_CA_Events table so that we have "MessagePartID"
-- and make that part of the PK
IF OBJECT_ID('tempdb..#AfterParse_CA_Events', 'U') IS NOT NULL
DROP TABLE #AfterParse_CA_Events;
GO
CREATE TABLE #AfterParse_CA_Events (
EvenID BIGINT NOT NULL,
MessageID BIGINT NOT NULL,
MessagePartID INT NOT NULL,
MessagePart VARCHAR(1000) NOT NULL
PRIMARY KEY (EvenID, MessageID, MessagePartID)
);
GO
--===============================================================
-- Now let's try the insertion again...
INSERT #AfterParse_CA_Events (EvenID, MessageID, MessagePartID, MessagePart)
SELECT
etd.EventID,
etd.MessageID,
dsk.ItemNumber,
dsk.Item
FROM
#EventsToDo etd
CROSS APPLY dbo.DelimitedSplit8K(etd.MessageText, ',') dsk;
GO
--===============================================================
-- check the inserted values...
SELECT
*
FROM
#AfterParse_CA_Events apce;
HTH, Jason
I have a table type that I am sending as a variable in my stored procedure.
CREATE TYPE OperationKeysTableType AS TABLE
(
[KeysId] [int] NOT NULL,
[OperationId] [int] NOT NULL,
[IsChecked] [bit] NOT NULL
)
And then I want to use this table to do a search, if they match or not, insert or delete but only if it is not checked like so:
MERGE INTO dbo.tblOperationKeys AS T
USING (SELECT VT.KeysId, VT.OperationId as OperationId FROM #ValuesTable AS VT) AS S
ON T.KeysId = S.KeysId AND T.OperationId = S.OperationId
WHEN MATCHED AND S.IsChecked = 0 THEN --DELETE
DELETE
WHEN NOT MATCHED THEN--INSERT
INSERT (KeysId, OperationId) VALUES (S.KeysId, S.OperationId)
OUTPUT $action AS ChangesMade, inserted.OperationKeysId AS new_OperationKeysId, deleted.OperationKeysId AS old_OperationKeysId;
My issue is that SQL tells me IsChecked is invalid. Any idea on what I'm doing wrong?
You aren't SELECTING IsChecked.
At the line:
USING (SELECT VT.KeysId, VT.OperationId as OperationId FROM #ValuesTable AS VT) AS S
You need:
USING (SELECT VT.KeysId, VT.OperationId as OperationId, VT.IsChecked as IsChecked FROM #ValuesTable AS VT) AS S
We have a table where we store all the exceptions (message, stackTrace, etc..), the table is getting big and we would like to reduce it.
There are plenty of repeated StackTraces, Messages, etc, but enabling compression produces a modest size reduction (10%) while I think much bigger benefits could come if somehow Sql Server will intern the strings in some per-column hash-table.
I could get some of the benefits if I normalize the table and extract StackTraces to another one, but exception messages, exception types, etc.. are also repeated.
Is there a way to enable string interning for some column in Sql Server?
There is no built-in way to do this. You could easily do something like:
SELECT MessageID = IDENTITY(INT, 1, 1), Message
INTO dbo.Messages
FROM dbo.HugeTable GROUP BY Message;
ALTER TABLE dbo.HugeTable ADD MessageID INT;
UPDATE h
SET h.MessageID = m.MessageID
FROM dbo.HugeTable AS h
INNER JOIN dbo.Messages AS m
ON h.Message = m.Message;
ALTER TABLE dbo.HugeTable DROP COLUMN Message;
Now you'll need to do a few things:
Change your logging procedure to perform an upsert to the Messages table
Add proper indexes to the messages table (wasn't sure of Message data type) and PK
Add FK to MessageID column
Rebuild indexes on HugeTable to reclaim space
Do this in a test environment first!
Aaron's posting answers the questions of adding interning to a table, but afterwards you will need to modify your application code and stored-procedures to work with the new schema.
...or so you might think. You can actually create a VIEW that returns data matching the old schema, and you can also support INSERT operations on the view too, which are translated into child operations on the Messages and HugeTable tables. For readability I'll use the names InternedStrings and ExceptionLogs for the tables.
So if the old table was this:
CREATE TABLE ExceptionLogs (
LogId int IDENTITY(1,1) NOT NULL PRIMARY KEY,
Message nvarchar(1024) NOT NULL,
ExceptionType nvarchar(512) NOT NULL,
StackTrace nvarchar(4096) NOT NULL
)
And the new tables are:
CREATE TABLE InternedStrings (
StringId int IDENTITY(1,1) NOT NULL PRIMARY KEY,
Value nvarchar(max) NOT NULL
)
CREATE TABLE ExceptionLogs2 ( -- note the new name
LogId int IDENTITY(1,1) NOT NULL PRIMARY KEY,
Message int NOT NULL,
ExceptionType int NOT NULL,
StackTrace int NOT NULL
)
Add an index to InternedStrings to make the value lookups faster:
CREATE UNIQUE NONCLUSTERED INDEX IX_U_InternedStrings_Value ON InternedStrings ( Value ASC )
Then you would also have a VIEW:
CREATE VIEW ExeptionLogs AS
SELECT
LogId,
MessageStrings .Value AS Message,
ExceptionTypeStrings.Value AS ExceptionType,
StackTraceStrings .Value AS StackTrace
FROM
ExceptionLogs2
INNER JOIN InternedStrings AS MessageStrings ON
MessageStrings.StringId = ExceptionLogs2.Message
INNER JOIN InternedStrings AS ExceptionTypeStrings ON
ExceptionTypeStrings.StringId = ExceptionLogs2.ExceptionType
INNER JOIN InternedStrings AS StackTraceStrings ON
StackTraceStrings.StringId = ExceptionLogs2.StackTrace
And to handle INSERT operations from unmodified clients:
CREATE TRIGGER ExceptionLogsInsertHandler
ON ExceptionLogs INSTEAD OF INSERT AS
DECLARE #messageId int = SELECT StringId FROM InternedStrings WHERE Value = inserted.Message
IF #messageId IS NULL
BEGIN
INSERT INTO InternedStrings ( Text ) VALUES ( inserted.Message )
SET #messageId = SCOPE_IDENTITY()
END
DECLARE #exceptionTypeId int = SELECT StringId FROM InternedStrings WHERE Value = inserted.ExceptionType
IF #exceptionTypeId IS NULL
BEGIN
INSERT INTO InternedStrings ( Text ) VALUES ( inserted.ExceptionType )
SET #exceptionTypeId = SCOPE_IDENTITY()
END
DECLARE #stackTraceId int = SELECT StringId FROM InternedStrings WHERE Value = inserted.StackTrace
IF #stackTraceId IS NULL
BEGIN
INSERT INTO InternedStrings ( Text ) VALUES ( inserted.StackTrace )
SET #stackTraceId = SCOPE_IDENTITY()
END
INSERT INTO ExceptionLogs2 ( Message, ExceptionType, StackTrace )
VALUES ( #messageId, #exceptionTypeId, #stackTraceId )
Note this TRIGGER can be improved: it only supports single-row insertions, and is not entirely concurrency-safe, though because previous data won't be mutated it means that there's a slight risk of data duplication in the InternedStrings table - and because of a UNIQUE index the insert will fail. There are different possible ways to handle this, such as using a TRANSACTION and changing the queries to use holdlock and updlock.