I found that tdengine has a parameter will create database. this definition "The KEEP parameter refers to the number of days to save a modified data file. " from tdengine's website https://www.taosdata.com/en/documentation/taos-sql#management. I think this parameter is very useful that there is no need to delete the history data. so i create a database which only keep 10 days.
CREATE DATABASE IF NOT EXISTS db_keep KEEP 10 PRECISION 'ms' ;
create table test_keep(ts timestamp,desc nchar(20));
after create db and table , I tried to insert some data into the table. the follows are my insert sqls.
insert into test_keep values(now,'now');
insert into test_keep values('2021-08-31 10:28:53.521','yesterday');
insert into test_keep values('2021-09-02 10:28:53.521','tomorrow');
insert into test_keep values('2021-08-25 10:28:53.521','6 days before');
insert into test_keep values('2021-09-20 12:28:53.521','20 days later');
insert into test_keep values('2021-08-21 10:28:53.521','10 days before');
insert into test_keep values('2021-08-11 10:28:53.521','20 days before');
While the lass three sql had execute error "DB error: Timestamp data out of range"
taos> insert into test_keep values(now,'now'); Query OK, 1 of 1 row(s)
in database (1.024000s)
taos> insert into test_keep values('2021-08-31
10:28:53.521','yesterday'); Query OK, 1 of 1 row(s) in database
(0.006000s)
taos> insert into test_keep values('2021-09-02
10:28:53.521','tomorrow'); Query OK, 1 of 1 row(s) in database
(0.004000s)
taos> insert into test_keep values('2021-08-25 10:28:53.521','6 days
before'); Query OK, 1 of 1 row(s) in database (0.004000s)
taos> insert into test_keep values('2021-09-20 12:28:53.521','20 days
later');
DB error: Timestamp data out of range (0.005000s) taos> insert into
test_keep values('2021-08-21 10:28:53.521','10 days before');
DB error: Timestamp data out of range (0.004000s) taos> insert into
test_keep values('2021-08-11 10:28:53.521','20 days before');
DB error: Timestamp data out of range (0.004000s) taos>
I thought this because of my keep is to small, so i made it larger.
ALTER DATABASE db_keep KEEP 365;
and the I tried to insert the failed sql again I found cannot insert data some days later from now .
taos> insert into test_keep values('2021-09-20 12:28:53.521','20 days
later');
DB error: Timestamp data out of range (0.005000s) taos> insert into
test_keep values('2021-08-21 10:28:53.521','10 days before'); Query
OK, 1 of 1 row(s) in database (0.004000s)
taos> insert into test_keep values('2021-08-11 10:28:53.521','20 days
before'); Query OK, 1 of 1 row(s) in database (0.004000s)
I want to ask how to use keep and the how dose it limit the data's timestamp?
For timestamp in a database, the two configuration options is most reasonable:
keep: the longest days to keep in the database refer to current timestamp(3650 days by default). When the "newest" timestamp in a persist file goes out of the range, the persist file will be deleted. Timestamp older than now - keep will be treated as Timestamp out of range.
days: the time range to store data in a file, 10 days by default. Newer timestamp would not be larger than now+ days.
So, the acceptale timestamp range in a database will be [now - keep, now + days].
For past data, the timestamp value cannot exceed (current_time - keep). Meanwhile, for future data, the timestamp value cannot exceed (current time + days).
Since my reputation is not enough for inserting images here, please refer to reference link below for details (grey means can NOT be inserted).
Reference:
https://segmentfault.com/a/1190000040617572
Related
I have a table in SQlserver, which has 1000 records 10 days ago. So, after 10 days when i checked the record count it is 2000 (not bothered about who is inserting records). when i query in sys.dm_db_index_usage_stats table to get last_user_update, there are no records available for my table even though the record has increased to 2000 from 1000. I've even checked in sys.dm_os_sys_info for sqlserver restart time and last restart was a month ago.
What could be the reason for absense of records in index_usage_stats and if record count is getting increased it should be captured in last_user_update column, am i missing something? Pls suggest.
I have an insert that is executed every 2 seconds (20 columns, 15000 rows, outside of SQL Server [sensor data]), it runs in 200 ms. I would like to keep only 10 minutes of data (sum ~4.500.000 rows) in this table then move (archive) the earliest 5 minutes to another archive table (that will store 50 days, billions of rows). The stored procedure for archiving:
begin tran
declare #UTC_min datetime2(2) = (select TOP 1 UTCLogDateTime from table1 order by UTCLogDateTime asc)
declare #UTC_copy datetime2(2) = dateadd(minute,5,#UTC_min)
INSERT INTO archive_table
SELECT *
FROM table1
where UTCLogDateTime<#UTC_copy
delete top(100000) from table1 where UTCLogDateTime<#UTC_copy
WHILE ##rowcount > 0
BEGIN
delete top(100000) from table1 where UTCLogDateTime<#UTC_copy
END
commit
I would like to ensure that the insert runs flawlessly and as fast as possible, without locking out from this archiving process. When archiving starts, the runtime for insert grows to 4-5 sec. I will also have a 2 sec live query (Power BI) to read this table.
Currently I have a clustered index on the UTCLogDateTime column for both tables.
These all processes have to run seamlessly, without locking the table from each other.
Do you have any suggestions how I can achieve this?
If you are using SQL Server 2016 and above, you may use TRUNCATE WITH partitions. This uses fewer locks compared to DELETE. Worth trying partitioning the table in TEST environment first.
Thanks for looking. I'm trying to write a SQL Server trigger that when a new record is added containing date information, will add the day of the week to the DayOfWeek column. Here's my table, with the columns in order:
Food table:
FoodName **varchar(20)**
CategoryID (FK) **int**
Price **smallmoney**
StoreID (FK) **int**
Date **datetime**
DayOfWeek **varchar(9)**
ShopperID (FK) **int**
Week **int**
Here is the trigger I've written:
-- Create a trigger to update day of the week when a record is inserted
CREATE TRIGGER DOW
ON Food
FOR INSERT
AS
BEGIN
-- Declare a variable to hold the date ID
DECLARE #dateID DATETIME
-- Get the date from the new record and store it in #dateID
SELECT #dateID = Date FROM Food
-- Insert day of the week based on the inserted date
INSERT INTO Food (DayOfWeek)
SELECT DATENAME(dw, #dateID)
END
GO
SQL Server seemed to accept the procedure, but when I ran another procedure to insert a new record, I got this error:
Msg 515, Level 16, State 2, Procedure DOW, Line 8 [Batch Start Line 21]
Cannot insert the value NULL into column 'Week', table *******; column does not allow nulls. INSERT fails.
I am not sure why this trigger is affecting the 'Week' column at all. The code should take the value entered for the Date and use the DATENAME(dw,...) function to return the day of the week, which should go into the DayOfWeek column. I've written a stored procedure that accepts a date as input and inserts the corresponding day of the week into the record, and it works just fine, but this trigger doesn't seem to want to cooperate. I'm stumped!
What your trigger does:
it fetches a Date from your table (the last one that is returned) which is not necessarily the last inserted value.
it tries to insert a new record with just the DayOfWeek of that Date specified.
it fails, because at least the Week must also be specified.
I guess that you want to update the value of the DayOfWeek for the inserted row(s) instead. To be able to do so, there must be a way to identify the row(s) that need to be updated in the Food table by knowing the values of the inserted rows. To be sure to update the correct rows, there should be a primary key that allows you to identify them. For sure you have such a primary key, and I guess that it's named FoodID, so probably you wanted to do this:
CREATE TRIGGER DOW ON Food
FOR INSERT
AS
BEGIN
SET NOCOUNT ON;
-- update the day of the week for the inserted rows
UPDATE Food
SET [DayOfWeek] = DATENAME(dw, f.[Date])
FROM Food f
INNER JOIN inserted i ON f.FoodID = i.FoodID
END
GO
There are some major problems with your trigger. In triggers, there is an inserted table (on inserts and updates) and deleted table (on deletes and updates). You should be using this table's information to know what records need updated.
This is bad because a trigger can have multiple rows
This SQL simply will not work correctly if you insert multiple rows.
DECLARE #dateID DATETIME
SELECT #dateID = Date FROM Food
This SQL is trying to insert a new row which is causing your NULL error
It is not trying to update the row you are inserting
INSERT INTO Food (DayOfWeek)
SELECT DATENAME(dw, #dateID)
It would need to be an INSTEAD OF trigger to avoid the null constraint on the column. Wolfgang's answer will still cause a null constraint error, because after triggers run AFTER the data is inserted. An INSTEAD OF trigger will run in place of the the actual insert.
CREATE TRIGGER DOW ON Food
INSTEAD OF INSERT
AS
BEGIN
SET NOCOUNT ON;
-- update the day of the week for the inserted rows
INSERT INTO Food (FoodName,CategoryID,Price,StoreID,[Date],ShopperID,[Week],[DayOfWeek])
SELECT
FoodName,CategoryID,Price,StoreID,[Date],ShopperID,[Week],DATENAME(dw, [Date]) AS [DayOfWeek]
FROM inserted
END
GO
Personally, I think storing the week and day of week is a bad idea. You already have a value that can derive that information (Date). Any time you have multiple columns that are essentially duplicate data, you will run into maintenance pain.
Table 1 - Escalation, Table2 - Data
The requirement is : for all record in data table, we need to perform escalation.
Example 1 - record 1 in data table, year is 2014 and economic year is 2018. So, we need to escalate the value(600) from 2014 to 2015, then to 2016, then to 2017.
So, final value will be 600*5*6*7
Example 2 - record 3 in data table will be escalated twice - from 2015 to 2016 and then to 2017. So, final value will be 1000*6*7
This has to be performed for all records in data table.
I don't want to use cursor as i have 3 million records to do this.
Please suggest some idea to perform using Set Operation
create table data
([year] int,
value int,
economic_year int,
modelid int,
shopid int)
create table escalation
([year] int,
shopid int,
value_es int
)
insert into data
values(
2014,600,2018,5,1),
(2014,600,2018,5,1),
(2015,1000,2018,5,1),
(2016,2000,2018,5,1),
(2017,3000,2018,5,1)
insert into escalation
values
(2014,1,4),
(2015,1,5),
(2016,1,6),
(2017,1,7)
select * from escalation
select * from data
One way you could do this is have a third table that you prepopulate with all the possible "update paths" and the respective values. For example, if your year is 2014 and economic year is 2018, then your multiplier (value_es) is 120 (4*5*6). Something like this:
You would join the data table to this new table on two columns data.year = start_year and data.economic_year = econ_year. Then, your new value would be data.value * val.
You could create a simple ETL process that initially populates this new table and updates it when the calendar rolls over to a new year. Let me know what you think or if you have any questions.
We are trying to create a stored procedure to archive data older than 6 months (180 days) from our production database in to a new archive database.
We also want to delete those archived rows from the production database.
We are thinking to include a while loop, but we want to archive only 10,000 rows a day and we need to schedule it on daily basis.
Can you please share us your experience.
Thanks
Maybe delete into would work for you? Found something useful here: https://msdn.microsoft.com/en-us/library/ms177564.aspx
USE AdventureWorks2012;
GO
DECLARE #MyTableVar TABLE
(
ProductID INT NOT NULL
);
DELETE TOP (10000) ph
OUTPUT DELETED.ProductID INTO #MyTableVar
FROM Production.ProductProductPhoto AS ph
WHERE DATEDIFF(DAY, ph.YourDay, GETDATE()) > 180
--Display the results of the table variable.
SELECT *
FROM #MyTableVar