I have a cockroachdb instance running in production and would like to know the settings for the --max-sql-memory and --cache specified when the database was started. I am trying to enhance performance by following this production checklist but I am not able infer the setting either on dashboard or sql console.
Where can I check the values of max-sql-memory and cache value ?
Note: I am able to access the cockroachdb admin console and sql tables.
You can find this information in the logs, shortly after node startup:
I190626 10:22:47.714002 1 cli/start.go:1082 CockroachDB CCL v19.1.2 (x86_64-unknown-linux-gnu, built 2019/06/07 17:32:15, go1.11.6)
I190626 10:22:47.815277 1 server/status/recorder.go:610 available memory from cgroups (8.0 EiB) exceeds system memory 31 GiB, using system memory
I190626 10:22:47.815311 1 server/config.go:386 system total memory: 31 GiB
I190626 10:22:47.815411 1 server/config.go:388 server configuration:
max offset 500000000
cache size 7.8 GiB <====
SQL memory pool size 7.8 GiB <====
scan interval 10m0s
scan min idle time 10ms
scan max idle time 1s
event log enabled true
If the logs have been rotated, the value depends on the flags.
The defaults for v19.1 are 128MB, with recommended settings being 0.25 (a quarter of system memory).
The settings are not currently logged periodically or exported through metrics.
After some weeks of stress tests with a Cassandra 3.10 cluster, suddenly errors "Mutation of x bytes is too large" have appeared, with high CPU load (e.g. load average: 33.26, 32.47, 32.15) on all the 6 nodes.
There has been not change in the size of the executed queries, so we have tried to increase the commit_log_segment_size_in_mb . It doesn't help much. The result of nodetool info is as bellows.
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 126.04 GiB
Generation No : 1498838154
Uptime (seconds) : 68795
Heap Memory (MB) : 3258.00 / 8004.00
Off Heap Memory (MB) : 597.23
Some blob are inserted, with predictable max size (and unchanged by the degradation time) or ~18kb (avg ~10kb).
Data are mainly updated using single-partition atomic/logged batches (with configurable and so preditable max batch size), as it was providing the best throughput according the stress tests.
Note that even if reverting now to individual async update doesn't fix this degradation.
What could have caused this sudden degradation of the performances ?
The amount of data had increased (~ * 1.1), but I would not expect it to cause such disproportionate issue (there is no lack of available disk storage related to that). I would expect Cassandra to scale well in such situation.
Thanks for ideas
I am experiencing a strange situation - my plan cache is almost empty. I use the following query to see what's inside:
SELECT dec.plan_handle,qs.sql_handle, dec.usecounts, dec.refcounts, dec.objtype
, dec.cacheobjtype, des.dbid, des.text,deq.query_plan
FROM sys.dm_exec_cached_plans AS dec
join sys.dm_exec_query_stats AS qs on dec.plan_handle=qs.plan_handle
CROSS APPLY sys.dm_exec_sql_text(dec.plan_handle) AS des
CROSS APPLY sys.dm_exec_query_plan(dec.plan_handle) AS deq
WHERE cacheobjtype = N'Compiled Plan'
AND objtype IN (N'Adhoc', N'Prepared')
One moment it shows me 82 rows, the next one 50, then 40 then 55 and so on while an hour before I couldn't reach the end of the plan cache issuing the same command. The point is that SQL Server keeps the plan cache very-very small.
The main reason of my investigation is high CPU compared to our baselines without any high loads, under normal during-the day workload - constantly 65-80%
Perfmon counters show low values for Plan Cache Hit Ratio - around 30-50%, high compilations - 400 out of 2000 batch requests per second and high CPU - 73 avg. What could cause this behaviour?
The main purpose of the question is to learn the possible reasons for an empty plan cache.
Memory is OK - min: 0 max: 245000.
I also didn't notice any signs of memory pressure - PLE, lazy writes, free list stalls disk activity were just ok, logs did not tell me a thing.
I came here for possible causes of this so I could proceed with investigation.
EDIT: I have also considered this thread:
SQL Server 2008 plan cache is almost always empty
But none of the recommendations/possible reasons are relevant.
The main purpose of the question is to learn the possible reasons for an empty plan cache.
If it is to learn,the answer from Martin Smith,in the thread you referred will help you
If you want to know in particular,why plan is getting emptied,i recommend using extended events and try below extended event
I am using data grids as my primary "database". I noticed a drastic difference between Hazelcast and Ignite query performance. I optimized my data grid usage by the proper custom serialization and indexes, but the difference is still noticeable IMO.
Since no one asked it here, I am going to answer my own question for all future references. This is not an abstract (learning) exercise, but a real-world benchmark, that models my data grid usage in large SaaS systems - primarily to display sorted and filtered paginated lists. I primarily wanted to know how much overhead my universal JDBC-ish data grid access layer adds compared to raw no-frameworks Hazelcast and Ignite usage. But since I am comparing apples to apples, here comes the benchmark.
I have reviewed the provided code on GitHub and have many comments:
Indexing and Joins
Probably the most important point is that Apache Ignite indexing is a lot more sophisticated than Hazelcast. Unlike Hazelcast, Ignite supports ANSI 99 SQL, so you can write your queries at will.
Most importantly, unlike Hazelcast, Ignite supports group-indexes and SQL JOINs across different caches or data types. Imagine that you have Person and Organization tables, and you need to select all Persons working for the same Organization. This would be impossible to do in 1 step in Hazelcast (correct me if I am wrong), but in Ignite it is a simple SQL JOIN query.
Given the above, Ignite indexes will take a bit longer to create, especially in your test, where you have 7 of them.
Fixes in TestEntity Class
In your code, the entity you store in cache, TestEntity, recalculates the value for idSort, createdAtSort, and modifiedAtSort every time the getter is called. Ignite calls these getters several times while the entity is being stored in the index tree. A simple fix to the TestEntity class provides 4x performance improvement: https://gist.github.com/dsetrakyan/6bfe089d53f888448503
Heap Measurement is Not Accurate
The way you measure heap is incorrect. You should at least call System.gc() before taking the heap measurement, and even that would not be accurate. For example, in the results below, I get negative heap size using your method.
Warmup
Every benchmark requires a warm-up. For example, when I apply the TestEntity fix, as suggested above, and do the cache population and queries 2 times, I get better results.
MySQL Comparison
I don't think comparing a single-node Data Grid test to MySQL is fair, neither for Ignite, nor for Hazelcast. Databases have their own caching and whenever working with such small memory sizes, you are usually testing database in-memory cache vs. Data Grid in-memory cache.
The performance benefit usually comes in whenever doing a distributed test over a partitioned cache. This way a Data Grid will execute the query on each cluster node in parallel and the results should come back a lot faster.
Results
Here are the results I got for Apache Ignite. They look a lot better after I made the aforementioned fixes.
Note that the 2nd time we execute the cache population and cache queries, we get better results because the HotSpot JVM is warmed up.
It is worth mentioning that Ignite does not cache query results. Every time you run the query, you are executing it from scratch.
[00:45:15] Ignite node started OK (id=0960e091, grid=Benchmark)
[00:45:15] Topology snapshot [ver=1, servers=1, clients=0, CPUs=4, heap=8.0GB]
Starting - used heap: 225847216 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 1001824120 bytes
Cache: 100000 entries, heap size: 775976904 bytes, inserts took 14819 ms
------------------------------------
Starting - used heap: 1139467848 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 978473664 bytes
Cache: 100000 entries, heap size: **-160994184** bytes, inserts took 11082 ms
------------------------------------
Query 1 count: 100, time: 110 ms, heap size: 1037116472 bytes
Query 2 count: 100, time: 285 ms, heap size: 1037116472 bytes
Query 3 count: 100, time: 19 ms, heap size: 1037116472 bytes
Query 4 count: 100, time: 123 ms, heap size: 1037116472 bytes
------------------------------------
Query 1 count: 100, time: 10 ms, heap size: 1037116472 bytes
Query 2 count: 100, time: 116 ms, heap size: 1056692952 bytes
Query 3 count: 100, time: 6 ms, heap size: 1056692952 bytes
Query 4 count: 100, time: 119 ms, heap size: 1056692952 bytes
------------------------------------
[00:45:52] Ignite node stopped OK [uptime=00:00:36:515]
I will create another GitHub repo with the corrected code and post it here when I am more awake (coffee is not helping anymore).
Here is the benchmark source code: https://github.com/a-rog/px100data/tree/master/examples/HazelcastVsIgnite
It is part of the JDBC-ish NoSQL framework I mentioned earlier: Px100 Data
Building and running it:
cd <project-dir>
mvn clean package
cd target
java -cp "grid-benchmark.jar:lib/*" -Xms512m -Xmx3000m -Xss4m com.px100systems.platform.benchmark.HazelcastTest 100000
java -cp "grid-benchmark.jar:lib/*" -Xms512m -Xmx3000m -Xss4m com.px100systems.platform.benchmark.IgniteTest 100000
As you can see, I set the memory limits high to avoid garbage collection. You can also run my own framework test (see Px100DataTest.java) and compare to the two above, but let's concentrate on pure performance. Neither test uses Spring or anything else except for Hazelcast 3.5.1 and Ignite 1.3.3 - the latest at the moment.
The benchmark transactionally inserts the specified number of appr. 1K-size records (100000 of them - you can increase it, but beware of memory) in batches (transactions) of 1000. Then it executes two queries with ascending and descending sort: four total. All query fields and ORDER BY are indexed.
I am not going to post the entire class (download it from GitHub). The Hazelcast query looks like this:
PagingPredicate predicate = new PagingPredicate(
new Predicates.AndPredicate(new Predicates.LikePredicate("textField", "%Jane%"),
new Predicates.GreaterLessPredicate("id", first.getId(), false, false)),
(o1, o2) -> ((TestEntity)o1.getValue()).getId().compareTo(((TestEntity)o2.getValue()).getId()),
100);
The matching Ignite query:
SqlQuery<Object, TestEntity> query = new SqlQuery<>(TestEntity.class,
"FROM TestEntity WHERE textField LIKE '%Jane%' AND id > '" + first.getId() + "' ORDER BY id LIMIT 100");
query.setPageSize(100);
Here are the results executed on my 2012 8-core MBP with 8G of memory:
Hazelcast
Starting - used heap: 49791048 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 580885264 bytes
Map: 100000 entries, used heap: 531094216 bytes, inserts took 5458 ms
Query 1 count: 100, time: 344 ms, heap size: 298844824 bytes
Query 2 count: 100, time: 115 ms, heap size: 454902648 bytes
Query 3 count: 100, time: 165 ms, heap size: 657153784 bytes
Query 4 count: 100, time: 106 ms, heap size: 811155544 bytes
Ignite
Starting - used heap: 100261632 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 1241999968 bytes
Cache: 100000 entries, heap size: 1141738336 bytes, inserts took 14387 ms
Query 1 count: 100, time: 222 ms, heap size: 917907456 bytes
Query 2 count: 100, time: 128 ms, heap size: 926325264 bytes
Query 3 count: 100, time: 7 ms, heap size: 926325264 bytes
Query 4 count: 100, time: 103 ms, heap size: 934743064 bytes
One obvious difference is the insert performance - noticeable in real life. However very rarely one inserts a 1000 records. Typically it is one insert or update (saving entered user's data, etc.), so it doesn't bother me. However the query performance does. Most data-centric business software is read-heavy.
Note the memory consumption. Ignite is much more RAM-hungry than Hazelcast. Which can explain better query performance. Well, if I decided to use an in-memory grid, should I worry about the memory?
You can clearly tell when data grids hit indexes and when they don't, how they cache compiled queries (the 7ms one), etc. I don't want to speculate and will let you play with it, as well, as Hazelcast and Ignite developers provide some insight.
As far, as general performance, it is comparable, if not below MySQL. IMO in-memory technology should do better. I am sure both companies will take notes.
The results above are pretty close. However when used within Px100 Data and the higher level Px100 (which heavily relies on indexed "sort fields" for pagination) Ignite pulls ahead and is better suited for my framework. I care primarily about the query performance.
i am running a delete statement:
DELETE FROM TransactionEntries
WHERE SessionGUID = #SessionGUID
The actual execution plan of the delete is:
Execution Tree
--------------
Clustered Index Delete(
OBJECT:([GrobManagementSystemLive].[dbo].[TransactionEntries].IX_TransactionEntries_SessionGUIDTransactionGUID]),
WHERE:([TransactionEntries].[SessionGUID]=[#SessionGUID])
)
The table is clustered by SessionGUID, so the 240 rows are physically together.
The table has no triggers on it.
The operation takes:
Duration: 11821 ms
CPU: 297
Reads: 14340
Writes: 1707
The table contains 11 indexes:
1 clustered index (SessionGUID)
1 unique (primary key) index
9 other non-unique, non-clustered indexes
How can i figure out why this delete operation is performing 14,340 reads, and takes 11 seconds?
the Avg. Disk Read Queue Length reaches 0.8
the Avg. Disk sec/Read never exceeds 4ms
the Avg. Disk Write Queue Length reaches 0.04
the Avg. Disk sec/Write never exceeds 4ms
What are the other reads for? The execution plan gives no indication of what it's reading.
Update:
EXECUTE sp_spaceused TransactionEntries
TransactionEntries
Rows 6,696,199
Data: 1,626,496 KB (249 bytes per row)
Indexes: 7,303,848 KB (1117 bytes per row)
Unused: 91,648 KB
============
Reserved: 9,021,992 KB (1380 bytes per row)
With 1,380 bytes per row, and 240 rows, that's 340 kB to be deleted.
Counter intuitive that it can be so difficult for 340 kB.
Update Two: Fragmentation
Name Scan Density Logical Fragmentation
============================= ============ =====================
IX_TransactionEntries_Tran... 12.834 48.392
IX_TransactionEntries_Curr... 15.419 41.239
IX_TransactionEntries_Tran... 12.875 48.372
TransactionEntries17 98.081 0.0049325
TransactionEntries5 12.960 48.180
PK_TransactionEntries 12.869 48.376
TransactionEntries18 12.886 48.480
IX_TranasctionEntries_CDR... 12.799 49.157
IX_TransactionEntries_CDR... 12.969 48.103
IX_TransactionEntries_Tra... 13.181 47.127
i defragmented TransactionEntries17
DBCC INDEXDEFRAG (0, 'TransactionEntries', 'TransactionEntries17')
since INDEXDEFRAG is an "online operation" (i.e. it only holds IS Intent Shared locks). i was going to then manually defragment the others until the business operations called, saying that the system is dead - and they switched to doing everything on paper.
What say you; 50% fragmentation, and only 12% scan density, cause horrible index scan performance?
As #JoeStefanelli points out in comments, it's the extra non-clustered indexes.
You are deleting 240 rows from the table.
This equates to 2640 index rows, 240 of which include all fields in the table.
Depending on how wide they are and how many included fields you have, this could equate to all the extra read activity you are seeing.
The non-clustered index rows will definitely NOT be grouped together on disk, which will increase delays.
I think the indexing might be the likeliest culprit but I wanted to throw out another possibility. You mentioned no triggers, but are there any tables that have a foreign key relationship to this table? They would have to be checked to make sure no records are in them and if you have cascade delete turned on, those records would have to be deleted as well.
Having banged my head on many-a-SQL performance issue, my standard operating procedure for something like this is to:
Back up the data
Delete one of the indexes on the table in question
Measure the operation
Restore DB
Repeat w/#2 until #3 shows a drastic change. That's likely your culprit.