We have a Primary-Stanby setup on PostgreSQL 12.1. A DELETE query is run on the primary server (which takes time but finishes completely) however the same query does not finish (and runs like forever):
DELETE FROM EVENTS
WHERE Event_Type_ID != 2
AND last_update_time <= '2020-11-04'
AND Event_ID NOT IN ( SELECT DISTINCT Event_ID FROM Event_Association )
AND Event_ID NOT IN ( SELECT DISTINCT Event_ID FROM EVENTS WHERE Last_Update_Time > '2020-11-14');
The execution plan is as follows (replacing delete with select query for the same):
https://explain.depesz.com/s/GZp7
There is a INDEX created on EVENTS.Event_ID and Event_Association.Event_ID however the delete query still won't finish on the standby server.
The EVENTS table has more than 2 million rows and the Event_Association table has more than 300,000 rows.
Can someone help me resolve this issue?
Thanks
Related
I have a log table where I have a date column log_date value like 2021-03-02 07:51:41.000 in the format.
My requirement is that on click of a button I want to list out of the log entries from the last hour.
I searched SO and find out the below query which is for MySQL.
I need two separate T-SQL queries that should work on SQL Server and Oracle for my requirement since the common query is not possible.
What is the best way to fetch the last 1 hour records from the log table for SQL and Oracle (separate query)?
select count(*) as cnt
from log
where log_date >= DATE_SUB(NOW(), INTERVAL 1 HOUR);
Create a database view in each of your databases, that filter only teh last hour of the log table.
here an example for Oracle
create view log_last_hour as
select *
from log
where log_date >= sysdate - interval '1' hour;
Than you can use a single simple query that is database independent
Example
select count(*) from log_last_hour
I have an insert that is executed every 2 seconds (20 columns, 15000 rows, outside of SQL Server [sensor data]), it runs in 200 ms. I would like to keep only 10 minutes of data (sum ~4.500.000 rows) in this table then move (archive) the earliest 5 minutes to another archive table (that will store 50 days, billions of rows). The stored procedure for archiving:
begin tran
declare #UTC_min datetime2(2) = (select TOP 1 UTCLogDateTime from table1 order by UTCLogDateTime asc)
declare #UTC_copy datetime2(2) = dateadd(minute,5,#UTC_min)
INSERT INTO archive_table
SELECT *
FROM table1
where UTCLogDateTime<#UTC_copy
delete top(100000) from table1 where UTCLogDateTime<#UTC_copy
WHILE ##rowcount > 0
BEGIN
delete top(100000) from table1 where UTCLogDateTime<#UTC_copy
END
commit
I would like to ensure that the insert runs flawlessly and as fast as possible, without locking out from this archiving process. When archiving starts, the runtime for insert grows to 4-5 sec. I will also have a 2 sec live query (Power BI) to read this table.
Currently I have a clustered index on the UTCLogDateTime column for both tables.
These all processes have to run seamlessly, without locking the table from each other.
Do you have any suggestions how I can achieve this?
If you are using SQL Server 2016 and above, you may use TRUNCATE WITH partitions. This uses fewer locks compared to DELETE. Worth trying partitioning the table in TEST environment first.
To make a long story short, is it possible to replicate rows of a table in SQL server with horizontal filtering function that being evaluated continuously?
For instance, I need to replicate a table rows to subscriber which are created or updated since last two days or more. I need any rows that are being created in source table but their creation date is older than two days get replicated to subscriber and this get done continuously on any newly created/updated rows. I mean that I don't need to replicate records that are newer than two days.
I have tried transaction replication with filtering function on SQL server 2017, but filtering function just get evaluated on replication creation time and after that, any new rows didn't get propagated to subscriber.
Add a column to your table: Alter Table yourTable Add Old_Enough Bit Not Null Default 0
Create a job that runs regularly (e.g. hourly) and runs Update yourTable Set Old_Enough = 1 Where Old_Enough = 0 And DateAdd(Day, 2, yourCreationDateColumn) < GetDate()
Create an indexed view that takes Select ... From yourTable Where Old_Enough = 1
Replicate your indexed view
I have a somewhat of what feels like a complex data problem that I am trying to solve. I am more a developer than a SQL expert but our DBA whom wrote the script a few months back, recently moved on and I have been tasked with resolving this problem in the short term.
I am having some real issues with four tables of which two are receiving real-time bulk inserts, being read by users and having a daily job copying then deleting the copied over records into historic tables. The operations run 24/7 and there is no downtime for the data insertions or archiving.
A script that originally did this process started to fail. It was changing the Serializable isolation from Snapshot in ReadCommitted mode, but did a full table scan so it blocked all insert operations for 1-2 hours, which was not acceptable.
Question(s)
The pain point for the archiving is that we have to wrap up the insert and deletes together in one transaction. Is there a better way to do the below? Is it better to not have foreign keys and use a trigger or constraint? The below scripts either lock the table too much or run for too long (1-4hours).
Scenario
Four SQL tables
Order (Memory Optimized table)
OrderDetail (Memory Optimized table)
OrderHistory (File/normal table – ColumnStore index)
OrderDetailHistory (File/normal table– ColumnStore index)
Order and OrderDetail tables receive on average a bulk insert between 50-700 inserts per second is performed.
Updates are only performed during the batch insert operation.
During the day the Order table can be between 3-6 million records.
The OrderDetail table can be up to 2-3 times as large as the Order table.
OrderHistory and OrderDetailHistory can have 1-7 days data, so varies between 10-50 million records at any one point.
OrderDetail has a FK reference to Order on the Id column.
At one point of time during the day, the data is copied and inserted for each table into their respective 'history' tables, which are non-memory optimized tables.
Attempt 1
The script with the issue does this:
BEGIN TRAN
INSERT INTO OrderHistory
SELECT * FROM Order o
WHERE o.CreatedAt <= 1 day
INSERT INTO OrderDetailHistory
SELECT * FROM OrderDetail od
WHERE od.CreatedAt <= 1 day
DELETE FROM Order
WHERE CreatedAt <= 1 day
DELETE FROM OrderDetail
WHERE CreatedAt <= 1 day
COMMIT
The database is run at Snapshot Isolation level with Read Committed.
When run, we were originally getting Serializable errors, and after some reviewing realised that the delete operation would go down to a serializable isolation level to lock the nonclustered index on the CreatedAt column whilst it performed deletes because it is doing a range scan on the index, whilst still inside the transaction that we used for selecting data too.
Attempt 2
So next I modified the script, by creating two memory optimized user defined tables to select data into first outside of the transaction into these udf tables. Then in a separate transaction, insert into our history table, and then finally delete in a separate transaction. The idea is that if the inserts are a success but the delete fails then the next time it runs it will not try to re-insert data twice. The downside to this is that there will be duplication in the history data until it runs again, and it ran for 2 hours before our scheduling tool timeout, so this doesn't seem ideal.
INSERT INTO #OrderLoadingUdfTable
SELECT * FROM Order o
WHERE o.CreatedAt <= 1 day
INSERT INTO #OrderDetailLoadingUdfTable
SELECT * FROM OrderDetail od
WHERE od.CreatedAt <= 1 day
BEGIN TRAN
INSERT INTO OrderHistory
SELECT * FROM #OrderLoadingUdfTable
WHERE CreatedAt <= 1 day
AND Id NOT IN (SELECT Id FROM OrderHistory)
INSERT INTO OrderDetailHistory
SELECT * FROM #OrderDetailLoadingUdfTable
WHERE CreatedAt <= 1 day
AND Id NOT IN (SELECT Id FROM OrderDetailHistory)
COMMIT
BEGIN TRAN
DELETE FROM Order
WHERE CreatedAt <= 1 day
DELETE FROM OrderDetail
WHERE CreatedAt <= 1 day
COMMIT
I have a web application and I doubt some others have deleted some records manually. Upon enquiry nobody is admitting the mistakes. How to find out at what time those records were deleted ?? Is it possible to get the history of delete queries ?
If you have access to v$ view then you can use the following query to get it. It contains the time as FIRST_LOAD_TIME column.
select *
from v$sql v
where upper(sql_text) like '%DELETE%';
If flashback query is enabled for your database (try it with select * from table as of timestamp sysdate - 1) then it may be possible to determine the exact time the records were deleted. Use the as of timestamp clause and adjust the timestamp as necessary to narrow down to a window where the records still existed and did not exist anymore.
For example
select *
from table
as of timestamp to_date('21102016 09:00:00', 'DDMMYYYY HH24:MI:SS')
where id = XXX; -- indicates record still exists
select *
from table
as of timestamp to_date('21102016 09:00:10', 'DDMMYYYY HH24:MI:SS')
where id = XXX; -- indicates record does not exist
-- conclusion: record was deleted in this 10 second window