I am essentially using Consul alongside a health checking service to store health check error messages from some number of remote hosts (key: hostname => value: message). However, I want the key-value pair to be deleted if it hasn't been updated in over 24 hours, for example, if the remote host is removed entirely from health check rotation.
According to the Consul API documentation, you can set a TTL on a session when it is created. It seems you can then bind this session to a particular key with the acquire parameter when you Create/Update a key, and when the session expires, it will also delete the key. The caveat is that acquire also locks the key, which to me implies the value cannot be set by anyone else until it is later updated with the release parameter, but this removes the session->key association (ultimately removing the TTL on the key).
It isn't clear how to implement implement a simple TTL for keys in a Consul key-value store. It seems something like this could work, but I'm not sure how to keep the sessions associated with the keys:
Create Session with TTL=86400s and get the sessionID
Create Key-Value pair with acquire=sessionID
??? release=sessionID ???
Every time the key-value pair is updated, refresh the session TTL
And repeat the above for every new key-value pair inserted.
Can someone please elaborate on how to use sessions to remove expired key-value pairs from Consul?
While creating a session, you can specify the behaviour to be of delete type and TTL can be set to 24hrs (check https://www.consul.io/api/session.html#ttl). So, if the session is not renewed within the time limit specified, then the key is automatically deleted (https://www.consul.io/docs/internals/sessions.html).
Related
I've got a Citus (v10.1) sharded PostgreSQL (v13) cluster with 4 nodes. Master node address is 10.0.0.2 and the rest are up to .5 When trying to manage my sharded table, I've got this error:
ERROR: 10.0.0.5:5432 is a metadata node, but is out of sync
HINT: If the node is up, wait until metadata gets synced to it and try again.
I've been waiting. After 30 minutes or more, I've literally did drop schema ... cascade, drop extension Citus cascade; and after re-importing the data, creating a shard I've got the same error message once more and can't get past it.
Some additional info:
Another thing that might be an actual hint is I cannot distribute my function through create_distributed_function(), because it's says it in a DeadLock state, and transaction cannot be committed.
I've checked idle processes, nothing out of ordinary.
Created shard like that:
SELECT create_distributed_table('test_table', 'id');
SELECT alter_distributed_table('test_table', shard_count:=128, cascade_to_colocated:=true);
There is no topic in google search result regarding this subject.
EDIT 1:
I did bomb (20k-200k hits per second) my shard with a huge amount of requests for a function that does insert/update or delete if specific argument is set.
This is a rather strange error. It might be the case that you hit the issue in https://github.com/citusdata/citus/issues/4721
Do you have column defaults that are generated by sequences? If so, consider using bigserial types for these columns.
If it does not work, you can disable metadata syncing by SELECT stop_metadata_sync)to_node('10.0.0.5','5432') optionally followed by SELECT start_metadata_sync_to_node('10.0.0.5','5432') to stop waiting for metadata syncing and (optionally) retry metadata creation from scratch.
From learn.microsoft.com "Populating a DataSet from a DataAdapter"
Pulling all of the table to the client also locks all of the rows on the server.
I didn't find any information (in Namespace: System.Data) regarding possibility to put lock on records (or group records) in DB, that was read to a DataSet (DataTable), which can affect all users of the DB, but not only those who will work with the database through my application.
Also From learn.microsoft.com "Using UpdatedRowSource to Map Values to a DataSet"
The Update method resolves your changes back to the data source; however other clients may have modified data at the data source since the last time you filled the DataSet. To refresh your DataSet with current data, use the DataAdapter and Fill method. New rows will be added to the table, and updated information will be incorporated into existing rows. The Fill method determines whether a new row will be added or an existing row will be updated by examining the primary key values of the rows in the DataSet and the rows returned by the SelectCommand. If the Fill method encounters a primary key value for a row in the DataSet that matches a primary key value from a row in the results returned by the SelectCommand, it updates the existing row with the information from the row returned by the SelectCommand and sets the RowState of the existing row to Unchanged. If a row returned by the SelectCommand has a primary key value that does not match any of the primary key values of the rows in the DataSet, the Fill method adds a new row with a RowState of Unchanged.
If you we have modified copy records from DB (locally in-memory) in DataSet, and want to propogand these chanes to server, why we must refresh our local record? It can rejects all our changes.
I doesn't understand, in common, the strategy for organization modification a records in DB throught DataSet:
make local copy records (by "Addapter.Fill(Dataset)")
change record (or records) locally (some contunies time) and wait when user click "Update", when:
save all modification in temp table?
reread records from DB (again by "Addapter.Fill(Dataset)")?
compare records from temp table with updated Dataset?
And if any nothing is changed, quickly to update records in DB (by
"Addapter.Update(Dataset)")?
But also in that case, It's a possibly that someone would be more quickly than I (and can update "My records" between the my reread and my update?
Rereaded all articles about ADO.NET (from learn.microsoft.com), again, and found some additive information and can answer on own questions:
1) The reading records from DB to DataSet (by Addapter.Fill(Dataset) doesn’t put any lock on records in DB (the foregoing phrase from learn.microsoft.com “Pulling all of the table to …” isn’t correct).
2) “In a multiuser environment, there are two models for updating data in a database: optimistic concurrency and pessimistic concurrency. The DataSet object is designed to encourage the use of optimistic concurrency (ONLY) for long-running activities, such as remoting data and interacting with data.”
A transaction (DbTransaction Class-derived types) which consists of a single command or a group of commands and which execute as a package, can put different types of locks on rows in tables in DB (DbTransaction.IsolationLevel Property).
Not correct use of transaction can very badly influence on work DB in multi-user env (the transactions must keep as short as possible).
3) When we talk regarding a locking records in DB, we must clear understood, what the aim of locking records and who it will be corresponds with logic work of our Application.
By example, we want made system for selling tickets for cinema (to simplify, for one movie and for one session only).
Application (which will run on multi-devices) must connect to DB and
fill local DataSet, and show for users all available seats (all rows
which have in special DB field SeatStatus = “FREE”).
After “selecting seat” user must press “Make Reservation” (MakeReservation()). Method MakeReservation() must use the ”Testing for Optimistic Concurrency Violations” (see example from Microsoft it more elegant than my from first post) trying to change value of the field SeatStatus to “RESERVED”, in this time will use the “optimistic concurrency” – who will press first, that “receive seat”.
Who will “second” receive message “Sorry, place is reserved” and Update for current list of available seats (UpdateSeats()).
Also, UpdateSeats() must periodically run on all active devices (one
per sec).
User who first pressed button, on next screen must entered
credit card information and press “Pay” (PayTicket()).
Method PayTicket() must connect to Bank, check payment and change
status SeatStatus to “OCCUPIED”, and this case (IMHO) more correct will use a some transaction with “pessimistic
concurrency”.
If User isn’t pressed “Pay” in some demanded time (5 min.) will
return to previous
screen and the field SeatStatus changed to “FREE”.
P.S>
Wellcome to all who knows more correct way to realize this task more correctly.
I'm new to Flink and I'm trying to use it to have a bunch of live views of my application. At least one of the dynamic views I'd like to build would be to show entries that have not met an SLA -- or essentially expired -- and the condition for this would be a simple timestamp comparison. So I would basically want an entry to show up in my dynamic table if it has NOT been touched by an event recently. In playing around with Flink 1.6 (constrained to this due to AWS Kinesis) in a dev environment, I'm not seeing that Flink is re-evaluating a condition unless an event touches that entry.
I've got my dev environment plugged into a Kinesis stream that's sending in live access log events from a web server. This isn't my real use case but it was an easy one to begin testing with. I've written a simple table query that pulls in a request path, its last access time, and computes a boolean flag to indicate whether it hasn't been accessed in the last minute. I'm debugging this via a retract stream connected to PrintSinkFunction so all updates/deletes are printed to my console.
tEnv.registerDataStream("AccessLogs", accessLogs, "username, status, request, responseSize, referrer, userAgent, requestTime, ActionTime.rowtime");
Table paths = tEnv.sqlQuery("SELECT request AS path, MAX(requestTime) as lastTime, CASE WHEN MAX(requestTime) < CURRENT_TIMESTAMP - INTERVAL '1' MINUTE THEN 1 ELSE 0 END AS expired FROM AccessLogs GROUP BY request");
DataStream<Tuple2<Boolean, Row>> retractStream = tEnv.toRetractStream(paths, Row.class);
retractStream .addSink(new PrintSinkFunction<>());
I expect that when I access a page, an Add event is sent to this stream. Then if I wait 1 minute (do nothing), the CASE statement in my table will evaluate to 1, so I should see a Delete and then Add event with that flag set.
What I actually see is that nothing happens until I load that page again. The Delete event actually has the flag set, while the Add event that immediate follows that has it cleared again (as it should since it's no longer "expired).
// add/delete, path, lastAccess, expired
(true,/mypage,2019-05-20 20:02:48.0,0) // first page load, add event
(false,/mypage,2019-05-20 20:02:48.0,1) // second load > 2 mins later, remove event for the entry with expired flag set
(true,/mypage,2019-05-20 20:05:01.0,0) // second load, add event
Edit: The most useful tip I've come across in my searching is to create a ProcessFunction. I think this is something I could make work with my dynamic tables (in some cases I'd end up with intermediate streams to look at computed dates), but hopefully it doesn't have to come to that.
I've gotten the ProcessFunction approach to work but it required a lot more tinkering than I initially thought it would:
I had to add a field to my POJO that changes in the onTimer() method (could be a date or a version that you simply bump each time)
I had to register this field as part of the dynamic table
I had to use this field in my query in order for the query to get re-evaluated and change the boolean flag (even though I don't actually use the new field). I just added it as part of my SELECT clause.
Your approach looks promising but a comparison with a moving "now" timestamp is not supported by Flink's Table API / SQL (yet).
I would solve this in two steps.
register the dynamic table in upsert mode, i.e., a table that is upserted per key (request in your case) based on a version timestamp (requestTime in your case). The resulting dynamic table would hold the latest row for every request.
Have a query with a simple filter predicate like yours that compares the version timestamp of the rows of the dynamic (upsert) table and filters out all rows that have timestamps which are too close to now.
Unfortunately, neither of both features (upsert conversions and comparisons against the moving "now" timestamp) are available in Flink, yet. There is some ongoing work for upsert table conversions though.
I would like to know at what time exactly the Id of an object is set to the next value of a sequence declared in (oracle) database.
Here is the code :
#GeneratedValue(strategy=GenerationType.SEQUENCE, generator = "SEQ_A")
#SequenceGenerator(name="SEQ_A", sequenceName = "SEQ_A_ID")
When having a sequence, JPA sets the id exactly after you call the persist method of your EntityManager instance. There is no need to commit of flush. The id is available exactly after that call.
That's the main difference between GenerationType.SEQUENCE and .IDENTITY. The later one offers the id after contacting the db, while SEQUENCE sets it directly, because a group of ids is preloaded in the application at runtime, therefore there's no need to interact with the db for getting an id.
Explanation of JPA protocol by a different implementation:
"Unlike AUTO and IDENTITY, the SEQUENCE strategy generates an automatic value as soon as a new entity object is persisted (i.e. before commit). This may be useful when the primary key value is needed earlier. To minimize round trips to the database server, IDs are allocated in groups. The number of IDs in each allocation is specified by the allocationSize attribute. It is possible that some of the IDs in a given allocation will not be used. Therefore, this strategy does not guarantee there will be no gaps in sequence values."
http://www.objectdb.com/java/jpa/entity/generated
I'm working on a project in dead ASP (I know :( )
Anyway it is working with a kdb+ database which is major overkill but not my call. Therefore to do inserts etc we're having to write special functions so they can be handled.
Anyway we've hit a theoretical problem and I'm a bit unsure how it should be dealt with in this case.
So basically you register a company, when you submit validation will occur and the page will be processed, inserting new values to the appropriate tables. Now at this stage I want to pull ID's from the tables and use them in the session for further registration screens. The user will never add a specific ID of course so it needs to be pulled from the database.
But how can this be done? I'm particularly concerned with 2 user's simultaneously registering, how can I ensure the correct ID is passed back to the correct session?
Thank you for any help you can provide.
Instead of having the ID set at the point of insert, is it possible for you to "grab" an ID value before hand, and then use that value throughout the process?
So:
Start the registration.
System connects to the database, creates an ID (perhaps from an ID table) and Stores in ASP Session.
Company registers.
You validate and insert data into DB (including the ID session)
The things you put in the Session(...) collection is only visible to that session (i.e. the session is used only by the browser windows on one computer). The session is identified by a GUID value that is stored in a cookie on the client machine. It is "safe" to store your IDs there (other users won't be able to read them easily) .
either your id can include date and time - so it will be example - id31032012200312 - but if you still think that 2 people can register at the same type then I would use recordset locks liek the ones here - http://www.w3schools.com/ado/prop_rs_locktype.asp
To crea ids like above in asp you do - replace(date(),"/","") ' and then same with time with ":"
Thanks