The TS time of TD is at the millisecond level. Now if there is an update mechanism, the same TS will be updated, so can only 1000 data come up from the device at the same time? For example, 2019-04-22 13:39:14.000 to 2019-04-22 13:39:14.999?
Related
I am using flink cep sql with blink planner.
Here is my sql
select * from test_table match_recognize (
partition by agent_id,room_id,call_type
order by row_time // process time
measures
last(BF.create_time) as create_time,
last(AF.connect_time) as connect_time
one row per match after match SKIP PAST LAST ROW
pattern (BF+ AF) WITHIN INTERVAL '10' SECOND
define
BF as BF.connect_time = 0,
AF as AF.connect_time > 0
) as T ;
The test_table is a kafka table
I set table.exec.state.ttl=10000 and run my program then I keep sending message.
As I both set state ttl and cep interval to 10s, the state's size should be a fixed number after 10 seconds when I started it.
But the fact is that the state keep growing for at least 15 minutes. Besides, jvm triggered twice full gc.
Are there any configurations I haven't configured？
You cannot use checkpoint sizes to estimate state size -- they are not related in any straightforward way. Checkpoints can include unpredictable amounts of in-flight, expired, or uncompacted data -- none of which would be counted as active state.
I'm afraid there isn't any good tooling available for measuring exactly how much state you actually have. But if you are using RocksDB, then you can enable these metrics
state.backend.rocksdb.metrics.estimate-live-data-size
state.backend.rocksdb.metrics.estimate-num-keys
which will give you a reasonably accurate estimate (but you may pay a performance penalty for turning them on).
As for your concern about CEP state -- you should be fine. Anytime you have a pattern that uses WITHIN, CEP should be able to clean the state automatically.
Problem :Does it make sense to have time travel setting as 1 and retention setting as 30 days at table level (permanent table) in snowflakes .
There is no real setting called time travel. data_retention_time_in_days is actually the setting that controls the duration of your time travel. If you are asking whether an account setting of 1 day and a table-level setting of 30 days makes sense, then the answer is 'Yes'. You can set this parameter at multiple levels (account, database, schema, table) to customize your data retention period based on those object levels.
As mentioned there is no setting for Time Travel. No tasks are required to enable Time Travel. It is automatically enabled with the standard, 1-day retention period.
To increase this retention period DATA_RETENTION_TIME_IN_DAYS parameter is used which can se set upto 90 days which can be set databases, schemas, and tables.
https://docs.snowflake.com/en/user-guide/data-time-travel.html
https://www.youtube.com/watch?v=F1pevMhm7lg
On top of that there is Fail-Safe period which provides a (non-configurable) 7-day period during which historical data is recoverable by Snowflake. This period starts immediately after the Time Travel retention period ends.
https://docs.snowflake.com/en/user-guide/data-failsafe.html
I need to implement data synchronization in a distributed system taking into account concurrent writes to the data table.
The export from Main DB should read only changed rows.
Common advice is to use triggers, marking rows with timestamp of data update or consequent revision; and tell this timestamp/revision_number to the remote system.
E.g. What is the best approach to pull "Delta" data into Analytics DB from a highly transactional DB?
But the problem with concurrent writes is in the moment of time when commit takes place. Here is a problem:
[time 00:00] Transaction A starts writing a big batch of data;
marking rows with timestamp [00:00].
[time 00:02] Transaction B starts writing a small amount of data;
going to mark rows with timestamp [00:02].
[time 00:03] Transaction B finishes writing and commit happens.
[time 00:10] Export is done to the remote part of the system.
Isolation level is ReadCommitted, so it gets know only data from transaction B, timestamp 00:02.
[time 00:15] Transaction A finishes writing and commit happens.
Remote system never receves this data(!)
I like the solution Change tracking in MSSQL, but I keep in mind our intention to migrate to PostgreSQL soon. So I have to implement a general solution.
What is the correct approach to solve this problem?
In an attempt to make the system time on an ODroid as close to realtime as possible, I've tried adding a real time clock to the ODroid. The RTC has an accuracy of +/- 4ppm.
Without the realtimeclock, I would get results like this (Synced with NTP-server every 60 seconds). The blue is an Orange Pi for comparison. The x-axis is the sample, and the y-axis is the offset reported by the NTP-server in ms.
So what I tried, was the same thing (Though more samples, but same interval), but instead of just syncing with the NTP-server, I did the following:
Set the system time to the hw-clock time.
Sync with the NTP-server to update the system time, and record the offset given by the server
Update the HW-clock to the systemtime, since it has just been synced to realtime.
Then I wait 60 seconds and repeat. I didn't expect it to be perfect, but what I got shocked me a little bit.
What in the world am I looking at? The jitter becomes less and less, and follows an almost straight line, but when it reaches the perfect time (about 410 minutes in....), it the seems to continue, and let the jitter and offset grow again.
Can anyone explain this, or maybe tell me what I'm doing wrong?
This is weird!
So you are plotting the difference between your RTC time and the NTP server time. Where is the NTP server located? In the second plot you are working in a range of a couple hundred ms. NTP has accuracy limitations. From wikipedia:
https://en.wikipedia.org/wiki/Network_Time_Protocol
NTP can usually maintain time to within tens of milliseconds over the
public Internet, and can achieve better than one millisecond accuracy
in local area networks under ideal conditions. Asymmetric routes and
network congestion can cause errors of 100 ms or more
Your data is a bit weird looking though.
In SQL Server 2008 Activity Monitor, I see Wait Time on Wait Category "Latch" (not Buffer Latch) spike above 10,000ms/sec at times. Average Waiter Count is under 10, but this is by far the highest area of waits in a very busy system. Disk IO is almost zero and page life expectancy is over 80,000, so I know it's not slowed down by disk hardware and assume it's not even touching SAN cache. Does this mean SQL Server is waiting on CPU (i.e. resolving a bajillion locks) or waiting to transfer data from the local server's cache memory for processing?
Background: System is a 48-core running SQL Server 2008 Enterprise w/ 64GB of RAM. Queries are under 100ms in response time - for now - but I'm trying to understand the bottlenecks before they get to 100x that level.
Class Count Sum Time Max Time
ACCESS_METHODS_DATASET_PARENT 649629086 3683117221 45600
BUFFER 20280535 23445826 8860
NESTING_TRANSACTION_READONLY 22309954 102483312 187
NESTING_TRANSACTION_FULL 7447169 123234478 265
Some latches are IO, some are CPU, some are other resource. It really depends on which particular latch type you're seeing this. sys.dm_os_latch_stats will show which latches are hot in your deployment.
I wouldn't worry about the last three items. The two nesting_transaction ones look very healthy (low average, low max). Buffer is also OK, more or less, although the the 8s max time is a bit high.
The AM_DS_PARENT latch is related to parallel queries/parallel scans. Its average is OK, but the max of 45s is rather high. W/o going into too much detail I can tell that long wait time on this latch type indicate that your IO subsystem can encounter spikes (and the max 8s BUFFER latch waits corroborate this).