Hazelcast vs. Ignite benchmark - benchmarking

I am using data grids as my primary "database". I noticed a drastic difference between Hazelcast and Ignite query performance. I optimized my data grid usage by the proper custom serialization and indexes, but the difference is still noticeable IMO.
Since no one asked it here, I am going to answer my own question for all future references. This is not an abstract (learning) exercise, but a real-world benchmark, that models my data grid usage in large SaaS systems - primarily to display sorted and filtered paginated lists. I primarily wanted to know how much overhead my universal JDBC-ish data grid access layer adds compared to raw no-frameworks Hazelcast and Ignite usage. But since I am comparing apples to apples, here comes the benchmark.

I have reviewed the provided code on GitHub and have many comments:
Indexing and Joins
Probably the most important point is that Apache Ignite indexing is a lot more sophisticated than Hazelcast. Unlike Hazelcast, Ignite supports ANSI 99 SQL, so you can write your queries at will.
Most importantly, unlike Hazelcast, Ignite supports group-indexes and SQL JOINs across different caches or data types. Imagine that you have Person and Organization tables, and you need to select all Persons working for the same Organization. This would be impossible to do in 1 step in Hazelcast (correct me if I am wrong), but in Ignite it is a simple SQL JOIN query.
Given the above, Ignite indexes will take a bit longer to create, especially in your test, where you have 7 of them.
Fixes in TestEntity Class
In your code, the entity you store in cache, TestEntity, recalculates the value for idSort, createdAtSort, and modifiedAtSort every time the getter is called. Ignite calls these getters several times while the entity is being stored in the index tree. A simple fix to the TestEntity class provides 4x performance improvement: https://gist.github.com/dsetrakyan/6bfe089d53f888448503
Heap Measurement is Not Accurate
The way you measure heap is incorrect. You should at least call System.gc() before taking the heap measurement, and even that would not be accurate. For example, in the results below, I get negative heap size using your method.
Every benchmark requires a warm-up. For example, when I apply the TestEntity fix, as suggested above, and do the cache population and queries 2 times, I get better results.
MySQL Comparison
I don't think comparing a single-node Data Grid test to MySQL is fair, neither for Ignite, nor for Hazelcast. Databases have their own caching and whenever working with such small memory sizes, you are usually testing database in-memory cache vs. Data Grid in-memory cache.
The performance benefit usually comes in whenever doing a distributed test over a partitioned cache. This way a Data Grid will execute the query on each cluster node in parallel and the results should come back a lot faster.
Here are the results I got for Apache Ignite. They look a lot better after I made the aforementioned fixes.
Note that the 2nd time we execute the cache population and cache queries, we get better results because the HotSpot JVM is warmed up.
It is worth mentioning that Ignite does not cache query results. Every time you run the query, you are executing it from scratch.
[00:45:15] Ignite node started OK (id=0960e091, grid=Benchmark)
[00:45:15] Topology snapshot [ver=1, servers=1, clients=0, CPUs=4, heap=8.0GB]
Starting - used heap: 225847216 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 1001824120 bytes
Cache: 100000 entries, heap size: 775976904 bytes, inserts took 14819 ms
Starting - used heap: 1139467848 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 978473664 bytes
Cache: 100000 entries, heap size: **-160994184** bytes, inserts took 11082 ms
Query 1 count: 100, time: 110 ms, heap size: 1037116472 bytes
Query 2 count: 100, time: 285 ms, heap size: 1037116472 bytes
Query 3 count: 100, time: 19 ms, heap size: 1037116472 bytes
Query 4 count: 100, time: 123 ms, heap size: 1037116472 bytes
Query 1 count: 100, time: 10 ms, heap size: 1037116472 bytes
Query 2 count: 100, time: 116 ms, heap size: 1056692952 bytes
Query 3 count: 100, time: 6 ms, heap size: 1056692952 bytes
Query 4 count: 100, time: 119 ms, heap size: 1056692952 bytes
[00:45:52] Ignite node stopped OK [uptime=00:00:36:515]
I will create another GitHub repo with the corrected code and post it here when I am more awake (coffee is not helping anymore).

Here is the benchmark source code: https://github.com/a-rog/px100data/tree/master/examples/HazelcastVsIgnite
It is part of the JDBC-ish NoSQL framework I mentioned earlier: Px100 Data
Building and running it:
cd <project-dir>
mvn clean package
cd target
java -cp "grid-benchmark.jar:lib/*" -Xms512m -Xmx3000m -Xss4m com.px100systems.platform.benchmark.HazelcastTest 100000
java -cp "grid-benchmark.jar:lib/*" -Xms512m -Xmx3000m -Xss4m com.px100systems.platform.benchmark.IgniteTest 100000
As you can see, I set the memory limits high to avoid garbage collection. You can also run my own framework test (see Px100DataTest.java) and compare to the two above, but let's concentrate on pure performance. Neither test uses Spring or anything else except for Hazelcast 3.5.1 and Ignite 1.3.3 - the latest at the moment.
The benchmark transactionally inserts the specified number of appr. 1K-size records (100000 of them - you can increase it, but beware of memory) in batches (transactions) of 1000. Then it executes two queries with ascending and descending sort: four total. All query fields and ORDER BY are indexed.
I am not going to post the entire class (download it from GitHub). The Hazelcast query looks like this:
PagingPredicate predicate = new PagingPredicate(
new Predicates.AndPredicate(new Predicates.LikePredicate("textField", "%Jane%"),
new Predicates.GreaterLessPredicate("id", first.getId(), false, false)),
(o1, o2) -> ((TestEntity)o1.getValue()).getId().compareTo(((TestEntity)o2.getValue()).getId()),
The matching Ignite query:
SqlQuery<Object, TestEntity> query = new SqlQuery<>(TestEntity.class,
"FROM TestEntity WHERE textField LIKE '%Jane%' AND id > '" + first.getId() + "' ORDER BY id LIMIT 100");
Here are the results executed on my 2012 8-core MBP with 8G of memory:
Starting - used heap: 49791048 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 580885264 bytes
Map: 100000 entries, used heap: 531094216 bytes, inserts took 5458 ms
Query 1 count: 100, time: 344 ms, heap size: 298844824 bytes
Query 2 count: 100, time: 115 ms, heap size: 454902648 bytes
Query 3 count: 100, time: 165 ms, heap size: 657153784 bytes
Query 4 count: 100, time: 106 ms, heap size: 811155544 bytes
Starting - used heap: 100261632 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 1241999968 bytes
Cache: 100000 entries, heap size: 1141738336 bytes, inserts took 14387 ms
Query 1 count: 100, time: 222 ms, heap size: 917907456 bytes
Query 2 count: 100, time: 128 ms, heap size: 926325264 bytes
Query 3 count: 100, time: 7 ms, heap size: 926325264 bytes
Query 4 count: 100, time: 103 ms, heap size: 934743064 bytes
One obvious difference is the insert performance - noticeable in real life. However very rarely one inserts a 1000 records. Typically it is one insert or update (saving entered user's data, etc.), so it doesn't bother me. However the query performance does. Most data-centric business software is read-heavy.
Note the memory consumption. Ignite is much more RAM-hungry than Hazelcast. Which can explain better query performance. Well, if I decided to use an in-memory grid, should I worry about the memory?
You can clearly tell when data grids hit indexes and when they don't, how they cache compiled queries (the 7ms one), etc. I don't want to speculate and will let you play with it, as well, as Hazelcast and Ignite developers provide some insight.
As far, as general performance, it is comparable, if not below MySQL. IMO in-memory technology should do better. I am sure both companies will take notes.
The results above are pretty close. However when used within Px100 Data and the higher level Px100 (which heavily relies on indexed "sort fields" for pagination) Ignite pulls ahead and is better suited for my framework. I care primarily about the query performance.


I don't understand how some parameters of Query Store in MS SQL Server works

According to official documentation: In sys.database_query_store_options we have options which can adjust Query Store workflow and performance.
From documentation:
"flush_interval_seconds - The period for regular flushing of Query Store data to disk in seconds. Default value is 900 (15 min)"
"interval_length_minutes - The statistics aggregation interval in minutes. Arbitrary values are not allowed. Use one of the following: 1, 5, 10, 15, 30, 60, and 1440 minutes. The default value is 60 minutes."
And now i have a problem:
If Query Store flush data to disk every 15min, why do i see query in QS tables in seconds after execution?
As i understand QS tables are 'permanent' and they are stored in data base (on disk), so how does this parameter (flush_interval_seconds) work?
The same thing about interval_length_minute - when i saved QS output after 1 minute from last query execution and after 61 minutes i realised that they are more less the same, so what about this aggregation?
flush_interval_seconds - The period for regular flushing of Query Store data to disk in seconds. That means flushing from memory to disk so that the information wouldn't be lost after server restart. Before the flushing you just read info from memory.
interval_length_minute - this is aggregation interval for query runtime statistics. The lower it is the finer granularity of the runtime statistics becomes.
None of the options sets a period after which the info will be available.

Cannot repair specific tables on specific nodes in Cassandra

I'm running 5 nodes in one DC of Cassandra 3.10.
As I'm trying to maintain those nodes I'm running on daily basis on every node
nodetool repair -pr
and weekly
nodetool repair -full
This is only table I have difficulties:
Table: user_tmp
SSTable count: 4
Space used (live): 366.71 MiB
Space used (total): 366.71 MiB
Space used by snapshots (total): 216.87 MiB
Off heap memory used (total): 5.28 MiB
SSTable Compression Ratio: 0.4690289976332873
Number of keys (estimate): 1968368
Memtable cell count: 2353
Memtable data size: 84.98 KiB
Memtable off heap memory used: 0 bytes
Memtable switch count: 1108
Local read count: 62938927
Local read latency: 0.324 ms
Local write count: 62938945
Local write latency: 0.018 ms
Pending flushes: 0
Percent repaired: 76.94
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 4.51 MiB
Bloom filter off heap memory used: 4.51 MiB
Index summary off heap memory used: 717.62 KiB
Compression metadata off heap memory used: 76.96 KiB
Compacted partition minimum bytes: 51
Compacted partition maximum bytes: 654949
Compacted partition mean bytes: 194
Average live cells per slice (last five minutes): 2.503074492537404
Maximum live cells per slice (last five minutes): 179
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
Dropped Mutations: 19 bytes
Percent repaired is never above 80% on this table on this and one more node but on others is above 85%. RF is 3, and strategy is SizeTieredCompactionStrategy
gc_grace_period is on 10days and as I somewhere in that period I'm getting writetimeout on exactly this table but after consumer which got this timeout is immediately replaced with another one everything keep going like nothing happened. Its like one time writetimeout.
My question is: Are you maybe have suggestion for better repair strategy because I'm kind of a noob and every suggest is a big win for me + any other for this table?
Maybe repair -inc instead of repair -pr
The nodetool repair command in Casandra 3.10 defaults to running incremental repair. There have been some major issues with incremental repair and it's currently not recommended by the community to run incremental repair. Please see this article for some great insight into repair and the issues with incremental repair: http://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html
I would recommend, as does many others, to run:
nodetool repair -full -pr
Please be aware that you need to run repair on every node in your cluster. This means that if you run repair on one node per day you can have a max of 7 nodes (since with default gc_grace you should aim to finish repair within 7 days). And you also have to rely on that nothing goes wrong when doing repair since you would have to restart any failing jobs.
This is why tools like Reaper exist. It solves these issues with ease, it automates repair and makes life simpler. Reaper runs scheduled repairs and provides a web interface to make administration easier. I would highly recommend using reaper for routine maintance and nodetool repair for unplanned activities.
Edit: Link http://cassandra-reaper.io/

Cassandra sudden errors Mutation of x bytes is too large

After some weeks of stress tests with a Cassandra 3.10 cluster, suddenly errors "Mutation of x bytes is too large" have appeared, with high CPU load (e.g. load average: 33.26, 32.47, 32.15) on all the 6 nodes.
There has been not change in the size of the executed queries, so we have tried to increase the commit_log_segment_size_in_mb . It doesn't help much. The result of nodetool info is as bellows.
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 126.04 GiB
Generation No : 1498838154
Uptime (seconds) : 68795
Heap Memory (MB) : 3258.00 / 8004.00
Off Heap Memory (MB) : 597.23
Some blob are inserted, with predictable max size (and unchanged by the degradation time) or ~18kb (avg ~10kb).
Data are mainly updated using single-partition atomic/logged batches (with configurable and so preditable max batch size), as it was providing the best throughput according the stress tests.
Note that even if reverting now to individual async update doesn't fix this degradation.
What could have caused this sudden degradation of the performances ?
The amount of data had increased (~ * 1.1), but I would not expect it to cause such disproportionate issue (there is no lack of available disk storage related to that). I would expect Cassandra to scale well in such situation.
Thanks for ideas

Almost empty plan cache

I am experiencing a strange situation - my plan cache is almost empty. I use the following query to see what's inside:
SELECT dec.plan_handle,qs.sql_handle, dec.usecounts, dec.refcounts, dec.objtype
, dec.cacheobjtype, des.dbid, des.text,deq.query_plan
FROM sys.dm_exec_cached_plans AS dec
join sys.dm_exec_query_stats AS qs on dec.plan_handle=qs.plan_handle
CROSS APPLY sys.dm_exec_sql_text(dec.plan_handle) AS des
CROSS APPLY sys.dm_exec_query_plan(dec.plan_handle) AS deq
WHERE cacheobjtype = N'Compiled Plan'
AND objtype IN (N'Adhoc', N'Prepared')
One moment it shows me 82 rows, the next one 50, then 40 then 55 and so on while an hour before I couldn't reach the end of the plan cache issuing the same command. The point is that SQL Server keeps the plan cache very-very small.
The main reason of my investigation is high CPU compared to our baselines without any high loads, under normal during-the day workload - constantly 65-80%
Perfmon counters show low values for Plan Cache Hit Ratio - around 30-50%, high compilations - 400 out of 2000 batch requests per second and high CPU - 73 avg. What could cause this behaviour?
The main purpose of the question is to learn the possible reasons for an empty plan cache.
Memory is OK - min: 0 max: 245000.
I also didn't notice any signs of memory pressure - PLE, lazy writes, free list stalls disk activity were just ok, logs did not tell me a thing.
I came here for possible causes of this so I could proceed with investigation.
EDIT: I have also considered this thread:
SQL Server 2008 plan cache is almost always empty
But none of the recommendations/possible reasons are relevant.
The main purpose of the question is to learn the possible reasons for an empty plan cache.
If it is to learn,the answer from Martin Smith,in the thread you referred will help you
If you want to know in particular,why plan is getting emptied,i recommend using extended events and try below extended event

Lucene query performance

I have a lucene index which has close to 480M documents. The size of the index is 36G. And I ran around 10000 queries against the index. Each query is a boolean AND query with 3 term queries inside. That is the query has 3 operands which MUST occur. Executing such 3 word queries gives the following latency percentiles.
50th = 16 ms
75th = 52 ms
90th = 121 ms
95th = 262 ms
99th = 76010 ms
99.9th = 76037 ms
Is the latency expected to degrade when the number of docs is as high as 480M? All the segments in the index are merged into one segment. Even when the segments are not merged, the latencies are not very different. Each document has 5-6 stored fields. But as mentioned above, the above latencies are for boolean queries that don't access any stored fields, but just do a posting list lookup on 3 tokens.
Any ideas on what could be wrong here?