I am using partition switching to rebuild indexes on a staging table without dropping them on the partitioned table as in microsoft's article.
I have what boils down to
BEGIN TRAN
ALTER INDEX IX_Working ON dbo.WorkingTable DISABLE
INSERT INTO dbo.WorkingTable ( Id, PartitionColumn, Values...)
SELECT Id, PartitionColumn, Values...
FROM PartitionedTable WITH (HOLDLOCK)
WHERE PartitionColumn <= #rightboundary
AND PartitionColumn > #leftboundary
INSERT INTO WorkingTable ( Id, PartitionColumn, Values...)
SELECT Id, PartitionColumn, Values...
FROM Imports
ALTER INDEX IX_Working ON WorkingTable REBUILD -- SLOW BIT
ALTER TABLE PartitionedTable SWITCH PARTITION #partition TO SwapTable
ALTER TABLE WorkingTable SWITCH TO PartitionedTable PARTITION #partition
TRUNCATE TABLE SwapTable
COMMIT
Now during this operation I need to block any updates to the partition being reindexed but still allow them on other partitions. The PartionedTable has lock escalation set to auto. I am trying to do this with the HOLDLOCK but I'm still able to do INSERT INTO PartionedTable (Id, #somevalueInTheRange, Values...) from another connection during the slow bit.
How can I block this while still allowing selects?
Can you try to use TABLOCKX along with HOLDLOCK? currently the select will put an SHARED lock which is not released because of HOLDLOCK. But SHARED lock wont prevent inserts.
Related
I'm currently looking for a method to improve the overall ability to ingest and consume data at the same time in an analytical environment (data warehouse). You may have already faced a similar situation and I'm interested about the various ways of improvement. Thanks in advance for your reading of the situation and your potential help on this matter!
The data ingestion process is currently relying on partition switching mechanism on top of Azure SQL Server. Multiple jobs per day are running. Data is loaded into staging table, then once the data is ready, there is partition swap operation happening, taking out the previous data on this partition, replaced by the new data set. The target table is configured with a clustered columnstore index. In addition, the table is configured with the lock escalation mode set to auto.
Once the data is ingested, a monitoring system is then automatically pushing this data into some Power BI datasets in import mode, by triggering refresh at partition level in PBI. The datasets are reading data in SQL Server in a very strict way, reading data partition by partition and are never reading data that is currently being ingested into the data warehouse. This system guarantees to have always the latest data up to date in the various PBI datasets with the shortest delay possible in the end-2-end.
When the alter table switch partition statement is fired, it may be locked by some other running statements that are consuming data located on the same table, but on another partition. This is due to a sch-s lock placed by the select query on the table.
Having set the lock escalation mode to auto, I was expecting the sch-s lock would be placed only at partition level, but this seems not the case.
This situation is particularly annoying as the queries running on Power BI are quite long, as a lots of records need to be moved into the dataset. As Power BI is running multiple queries in parallel, the locks can remain a very long time in total, preventing ingestion of new data to complete.
Few additional notes about the current setup:
I'm already taking advantage of the wait_at_low_priority feature, to ensure the swap partition process is not blocking another query to run in the meantime.
The partition swap process has been considered as we have snapshots of data, replacing entirely was has been ingested previously for the same partition.
Loading data in a staging table allows the computation of the columnstore index faster than inserting on the final data directly.
Here is below a test script showing the situation.
Hope this is all clear. Many thanks for your help!
Initialization
-- Clean any previous stuff.
drop table if exists dbo.test;
drop table if exists dbo.staging_test;
drop table if exists dbo.staging_out_test;
if exists(select 1 from sys.partition_schemes where name = 'ps_test') drop partition scheme ps_test;
if exists(select 1 from sys.partition_functions where name = 'pf_test') drop partition function pf_test;
go
-- Create partition function and scheme.
create partition function pf_test (int) as range right for values(1, 2, 3);
go
create partition scheme ps_test as partition pf_test all to ([primary]);
go
alter partition scheme ps_test next used [primary];
go
-- Data table.
create table dbo.test (
id int not null,
name varchar(100) not null
) on ps_test(id);
go
-- Staging table for data ingestion.
create table dbo.staging_test (
id int not null,
name varchar(100) not null
) on ps_test(id);
go
-- Staging table for taking out previous data.
create table dbo.staging_out_test (
id int not null,
name varchar(100) not null
) on ps_test(id);
go
-- Set clustered columnstore index on all tables.
create clustered columnstore index ix_test on dbo.test on ps_test(id);
create clustered columnstore index ix_staging_test on dbo.staging_test on ps_test(id);
create clustered columnstore index ix_staging_out_test on dbo.staging_out_test on ps_test(id);
go
-- Lock escalation mode is set to auto to allow locks at partition level.
alter table dbo.test set (lock_escalation = auto);
alter table dbo.staging_test set (lock_escalation = auto);
alter table dbo.staging_out_test set (lock_escalation = auto);
go
-- Insert few data...
insert into dbo.test (id, name) values(1, 'initial data partition 1'), (2, 'initial data partition 2');
insert into dbo.staging_test (id, name) values(1, 'new data partition 1'), (2, 'new data partition 2');
go
-- Display current data.
select * from dbo.test;
select * from dbo.staging_test;
select * from dbo.staging_out_test;
go
Long running query example (adjust variable #c to generate more or less records):
-- Generate a long running query hitting only one specific partition on the test table.
declare #i bigint = 1;
declare #c bigint = 100000;
with x as (
select #i n
union all
select n + 1
from x
where n < #c
)
select
d.name
from
x,
(
select name
from
dbo.test d
where
d.id = 2) d
option (MaxRecursion 0)
Partition swap example (to be run while "long running query" is running, showing the lock behavior:
select * from dbo.test;
-- Switch old data out.
alter table dbo.test switch partition $PARTITION.pf_test(1) to dbo.staging_out_test partition $PARTITION.pf_test(1) with(wait_at_low_priority (max_duration = 1 minutes, abort_after_wait = self));
-- Switch new data in.
alter table dbo.staging_test switch partition $PARTITION.pf_test(1) to dbo.test partition $PARTITION.pf_test(1) with(wait_at_low_priority (max_duration = 1 minutes, abort_after_wait = self));
go
select * from dbo.test;
I have a quick question for you... If I ran the following command:
BEGIN TRANSACTION
<commands...>
DELETE FROM <table>
COMMIT TRANSACTION
And while the above transaction is running an insert is carried out on the table. Will the delete:
remove the data added after the transaction started
only remove data that existed at the start of the transaction or that was added as part of the transaction
Hope someone can help.
You need to dive more to the Locks and Transaction Isolation Levels topic. Look at this example, which may be more common than in the previous answer. INSERT is not blocked here because DELETE just locks set of Keys for a DELETE operation.
And anyway, before DELETE operation start, if other queries in this transaction are not holding locks on this table, there is no reason for SQL Server to prevent INSERT operations from other transaction.
CREATE TABLE t (Id int PRIMARY KEY)
GO
INSERT INTO t VALUES(1)
GO
BEGIN TRAN
DELETE FROM t
-- separate window
INSERT INTO t VALUES(2)
I assume that your are running your code in one SPID and the insert will run on other SPID and the isolation level is the default one in SQL SERVER - READ COMMITTED.
Shortly, the answer is NO, as INSERT will wait for the DELETE to end. Tested like this:
1) Setup:
-- drop table dbo.Test
CREATE TABLE dbo.Test
(
Id INT NOT NULL,
Value NVARCHAR(4000)
)
GO
INSERT INTO Test (Id, Value)
SELECT ROW_NUMBER() OVER (ORDER BY (SELECT 1)), text
from sys.messages
GO
2) In query window 1
BEGIN TRANSACTION
DELETE FROM dbo.Test where ID > 100000
3) In query window 2
INSERT INTO Test (Id, Value)
SELECT 100000000 + ROW_NUMBER() OVER (ORDER BY (SELECT 1)), text
from sys.messages
sp_who2 active shows that second query (SPID) is blocked by first query, so query is waiting to get lock
3) In query window 1
COMMIT -- query 1 will finish
4) Second query will finish
So, INSERT has to wait until DELETE finishes.
Yes. Why not? If other queries in your transaction will not hold locks on a your_table, SQL Server will start locking your_table with Update locks, just after DELETE operation start. So, before this action, all other processes can successfully add new rows to the table.
DELETE in your case will delete all committed data, existed in your table before DELETE operation. Uncommitted data of this transaction will be deleted also.
I am inserting rows into single table from two instances and they are completing successfully.
When any transaction updating any table then it acquire 'Exclusive' on that Table (resource) and there must be single Exclusive lock on table while inserting data.
Granted
Requested Exclusive(X) Shared(S)
Exclusive NO NO
Shared NO Yes
Creating Sample table:
create table TestTransaction
(
Colid int Primary Key,
name varchar(10)
)
Inserting Instance 1:
Declare #counter int =1
Declare #countName varchar(10)='te'
Declare #max int=1000000
while #counter<#max
Begin
insert into TestTransaction
values
(
#counter,
#countName+Cast(#counter as varchar(7))
)
Set #counter=#counter+1
End
Inserting Instance 2:
insert into TestTransaction
values
(2000001,'yesOUTofT')
Why it is successful?
At the same time retrieval (Select) from this table is not happening because of the lock on table.
When any transaction updating any table then it acquire 'Exclusive' on that Table (resource) and there must be single Exclusive lock on table while inserting data.
That's a common myth. Locks in SQL Server are usually per-row. Various things cause them to escalate to page, partition or table level. SQL Server is designed, though, to try to lock at the smallest level first in order to allow for more concurrency.
Do not rely on any particular locking behavior in your apps if you can. Rather, make use of the isolation level setting if possible in order to obtain the required consistency guarantees you need.
I was having fun testing out the columnstore index feature of sql server 2012. Because you can't update/insert tables with such indices I read on some options: keep a separate table and use a new partition for every bulk insert or disable the index, perform updates/inserts and then rebuild the index.
For my test I chose the latter option and ended up with this stored procedure:
-- Disable the columnstore index.
ALTER INDEX [All_Columns_Columnstore_Index] ON [dbo].[Tick] DISABLE
-- Insert data into tick table from staging table.
insert into Tick
select [Date],
SymbolID,
Price
from TickTemporary
-- Delete data from staging table.
delete from TickTemporary
-- Enable (rebuild) the columnstore index.
ALTER INDEX [All_Columns_Columnstore_Index] ON [dbo].[Tick] REBUILD
If I execute these lines manually everything works fine. But if I run the procedure, I get the error that updates/inserts can't be performed on a table that has a columnstore index.
Why is this?
Update:
I followed the advice in the answer I previously accepted but I still get the same thing.
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- Disable the columnstore index.
EXEC DisableColumnStoreIndex
-- Insert data into tick table from staging table.
insert into Tick
select [Date],
SymbolID,
Price
from TickTemporary
-- Delete data from staging table.
delete from TickTemporary
-- Enable (rebuild) the columnstore index.
EXEC RebuildColumnStoreIndex
Even tried placing "begin tran" and "commit tran" around the sproc calls.
Using dynamic sql like:
declare #sql nvarchar(max)
set #sql =
'insert into Tick
select [Date],
SymbolID,
Price
from TickTemporary'
exec(#sql)
works, but really, I want to get by without dynamic sql. Isn't it possible in this case?
The check is done at compile time, not at execution time. Separate the procedure into it's own, or use dynamic SQL.
But as a general comment this is not the right approach. You should insert into a different table with identical structure, build the columnstore index on this identical table, then use partition switch to replace the old table with the new table: switch out the old table with an empty one, switch in the new table, drop the old data switched out. Similar to the procedure described in How to Update a table with a Columnstore Index. Because of the use of partition switch the users of your table experience a much shorter downtime, as the old table is still online and available during the insert and during the build columnstore stages.
Solution For This compile time execution is Option(recompile)
create PROCEDURE TEST
AS
BEGIN
ALTER INDEX [All_Columns_Columnstore_Index] ON [dbo].[Tick] DISABLE
-- Insert data into tick table from staging table.
insert into Tick
select [Date],
SymbolID,
Price
from TickTemporary **Option(recompile)**
-- Delete data from staging table.
delete from TickTemporary
-- Enable (rebuild) the columnstore index.
ALTER INDEX [All_Columns_Columnstore_Index] ON [dbo].[Tick] REBUILD
End
I am not sure if this question is an obvious one. I need to delete a load of data. Delete is expensive. I need to truncate the table but not fully so that the memory is released and watermark is changed.
Is there any feature which would allow me to truncate a table based on a condition for select rows?
Depends on how your table is organised.
1) if your (large) table is partitioned based on similar condition ( eg. you want to delete previous month's data and your table is partitioned by month), you could truncate only that partition, instead of the entire table.
2) The other option, provided you have some downtime, would be to insert the data that you want to keep into a temporary table, truncate the original table and then load the data back.
insert into <table1>
select * from <my_table>
where <condition>;
commit;
truncate table my_table;
insert into my_table
select * from <table1>;
commit;
--since the amount of data might change considerably,
--you might want to collect statistics again
exec dbms_stats.gather_table_stats
(ownname=>'SCHEMA_NAME',
tabname => 'MY_TABLE');