I know that when Solr performs optimization, either explicitly by the optimize command, or implicitly by Lucene due to the mergeFactor, readers are not blocked. That is, the server is still available for searching
Is it also available for updates? Can other threads in my application send documents updates to solr, and possibly also send commits? Will those updates pass through into the index, or will they be blocked?
An old question though, however, some more info can help here.
optimize command in solr is a call to IndexWriter's forceMerge() method. This method does take a lock on the IndexWriter instance itself. However, the point is that adding documents does not require any lock on the IW instance, neither does it need any commitLock or fullFlushLock.
Moreover, even with forceMerge(), it is the ConcurrentMergeScheduler which picks up the merge process and does it in a different thread altogether.
Usually merge process (Not the forceMerge, which is not recommended anyway) needs to lock the IndexWriter instance only while preparing the merge info, when it needs to know what segments to take for merge, and what is the new merged segment name etc. Once it has this information, merge happens concurrently.
So, yes, you can keep adding documents even when optimize is in process - they will get buffered in RAM until the next commit/optimize or close() of IndexWriter.
Having said that, might as well add that you can not have concurrent commits to different segments - that is Lucene will do only one commit at a time. Adding documents does not flush them to any segment at all - just puts them in buffer.
The answer is "Yes". The server will be respond to search requests, but updated documents will not show up in the search results until you send a commit command. The updated documents will stack up and be committed whenever a client/thread issues a commit command to the server. If you have multiple clients/threads issuing updates and commits they will not block each other, and the updates will show as soon as the commit command completes.
Related
I am new to Elasticsearch and trying to explore some use case for my business requirement.
What happens if multiple instances try to update a document?
Is there any error handling in place or the document gets locked?
Please advise
Elasticsearch is using optimistic concurrency control to ensure that an older version of a document never overwrites a newer version.
When documents are created, updated, or deleted, the new version of the document has to be replicated to other nodes in the cluster. Elasticsearch is also asynchronous and concurrent, meaning that these replication requests are sent in parallel, and may arrive at their destination out of sequence.
For more information you can check Elasticsearch documentation about optimistic concurrency control.
Before calling execute on the StreamExecutionEnvironment and starting the stream job, is there a way to programmatically find out whether or not the job was restored from a savepoint? I need to know such information so that I can set the offset of a Kafka source depending on it while building the job graph.
It seems that the FlinkConnectorKafkaBase class which has a method initializeState has access to such information (code). However, there is no way to intercept the FunctionInitializationContext and retrieve the isRestored() value since initializeState is a final method. Also, the initializeState method gets called after the job graph is executed and so I don't think there is a feasible solution associated to it.
Another attempt I made was to find a Flink job parameter that indicates whether or not the job was started from a savepoint. However, I don't think such parameter exists.
You can get the effect you are looking for by simply doing this:
FlinkKafkaConsumer<String> myConsumer = new FlinkKafkaConsumer<>(...);
myConsumer.setStartFromEarliest();
If you use setStartFromEarliest then Flink will ignore the offsets stored in Kafka, and instead begin reading from the earliest record. Moreover, even if you use setStartFromEarliest, if Flink is resuming from a checkpoint or savepoint, it will instead use the offsets stored in that snapshot.
Note that Flink does its own Kafka offset management, and when recovering from a checkpoint ignores the offsets stored in Kafka. Flink does this as a part of providing exactly-once guarantees, which requires knowing exactly how much of the input was consumed to produce the results present in the rest of the state captured in a checkpoint or savepoint. For this reason, Flink always stores the offsets as part of every state snapshot (checkpoint or savepoint).
This is documented here and here.
As for your original question about initializeState, this is available if you implement the CheckpointedFunction interface, but it's quite rare to actually need this.
I am not completely understanding how these two features relate to one another in a (WiredTiger) MongoDB program:
1) WiredTiger Snapshots
2) Data Locking
If each read operation using the WiredTiger engine is, at read-time, provided with a database level 'snapshot' (so as to create consistency (the C in ACID), why then, do we also need locking? Let's use an example.
I perform a query at the Document level (a read operation). Okay, so I know I get the database level snapshot, so that my data is consistent EVEN IF another user is concurrently writing to that same Document, updating it.
So at this point, what is the use for having a Shared-Lock on that document, which is blocking all write (exclusive) operations on that document until the Shared-Lock is released? What could possibly go wrong in writing to that Document concurrently while I'm reading it, if I am, in fact, using a snapshot of the Document that was provided to me at read-time? Why would I care if the Document is locked during my read-operation period or not? I already have my (consistent) data from that point-in-time, no?
I'm obviously missing a key concept here... Any help?
Thanks.
You are right that the read operation will acquire a snapshot. When using the WiredTiger storage engine, MongoDB does not lock individual documents for either reads or writes. Instead WiredTiger uses Multi-Version Concurrency Control, MVCC. When performing an update of a document, that update will succeed as long as the document still has the same version as it had when acquiring the snapshot. If not, WiredTiger will return an error (WT_ROLLBACK) indicating that the update had write conflicts. In this case, the update will abort and all pending changes are undone. MongoDB will then transparently retry the operation.
For learning purposes, I want to write my own database, that is able to replicate itself. I have made some progress, but now I am facing a problem that I can not solve. Supposed I have a database (let's call this source) that I would like to replicate to another database (let's call this target).
The basic principle is easy: In the source you don't store actual tables, but instead a log of transactions. It's easy to send over the transaction log to the target, where the database then rebuilds itself. If you want to update the target, you simply request the part of the transaction log that has changed ever since. Basically this is what almost every database does.
While this works, it has one major drawback: If a table already exists for a long time, the transaction log is very long, and hence replicating the table requires lots of time…
To avoid this you can store the current state as well. This means you have an up-to-date snapshot that you can copy fast. Additionally, the target has to subscribe to the transaction log of the source. Once it contains additional entries, the target applies them to its copied table. This works well, too, and it's way better in terms of performance and transferred volume.
But now I am facing a problem: Supposed the snapshot is large, then it may happen that changes are made to it while it is being delivered. That means that the copied snapshot contains some old and some new data. Now, how do I get the target database in a consistent state? Even if I know from where to start the transaction log, I either have to apply a change that was already applied to some of the records, or I have to leave it out, but then a change is not applied at all to some other records.
Of course I could use the isolation level sequential, but then performance drops. Of course I could do what e.g. CouchDB does and remember the current table revision in every record, and keep a copy of every record for every revision. But then the required space grows enormously.
So, what shall I do?
Everything that I was able to find on the web always either relies on the idea of replaying the entire transaction log, or by using a process as in CouchDB which takes up huge amounts of space.
Any ideas?
Your snapshot needs to be consistent and you need to know at what time (in regards to the tx log) it is consistent. You then apply any transactions that have been committed since this point.
Obtaining a consistent snapshot can be done with exclusive locking, which may delay other transactions from committing, or using row versions (MVCC).
Good luck with your project.
In the doc [1], it was said that
if using a write consistency level of QUORUM with a replication factor
of 3, Cassandra will send the write to 2 replicas. If the write fails on
one of the replicas but succeeds on the other, Cassandra will report a
write failure to the client.
So assume only 2 replicas receive the update, the write failed. But due to eventually consistency, all the nodes will receive the update finally.
So, should I retry? Or just leave it as it?
Any strategy?
[1] http://www.datastax.com/docs/1.0/dml/about_writes
Those docs aren't quite correct. Regardless of the consistency level (CL), writes are sent to all available replicas. If replicas aren't available, Cassandra won't send a request to the down nodes. If there aren't enough available from the outset to satisfy the CL, an UnavailableException is thrown and no write is attempted to any node.
However, the write can still succeed on some nodes and an error be returned to the client. In the example from [1], if one replica is down before the write was attempted, what is written is true.
So assume only 2 replicas receive the update, the write failed. But
due to eventually consistency, all the nodes will receive the update
finally.
Be careful though: a failed write doesn't tell you how many nodes the write was made to. It could be none so the write may not propagate eventually.
So, should I retry? Or just leave it as it?
In general you should retry, because it may not be written at all. You should only regard your write as written when you got a successful return from the write.
If you're using counters though you should be careful with retries. Because you don't know if the write was made or not, you could get duplicate counts. For counters, you probably don't want to retry (since more often than not the write will have been made to at least one node, at least for higher consistency levels).
Retry will not change much. The problem is that you actually cannot know whether data was persisted at all, because Cassandra throws always the same exception.
You have few options:
enable hints and retry request with cl=any - successful response would mean that at least hint was created. So you know that data is there but not yet accessible.
disable hints and retry with one - successful response would mean that at least node could receive data. In case of error execute delete.
use astyanax and their retry strategy
update to Cassandra 1.2 and use write-ahead log