MariaDB storage increase - database

Question

linux and db's nameswe faced the problem with mariadb storage.
we have centos 8 with monitoring system and mariadb as database system. while creating the linux machine we gave it 120 gb, but after adding the nodes to monitoring system the space was fulled. so we increase the linux root space by 50 gb.
but the problem is still exist:
mariadb service didn't work, so we have to restart it
we need to restart nginx service for web access.
after this 2 manipulations monitoring system works for a 5-6 hours, then we again have to restart DB and web service.
We think that database didn't use all 170GB and 'see' only the initial 120GB. As test we delete approximately 15 devices (+-15GB) from monitoring system and test it for a 5 days and there was not any DB or WEB issues.
MariaDB - 10.3.28 version
The used engine is InnoDB
We checked for innodb_page_size
Innodb_page_size = 16384
Could someone help us
innodb status1
innodb status2

Answer 1

Analysis of GLOBAL STATUS and VARIABLES:
Observations:
Version: 10.3.28-MariaDB
3.6 GB of RAM
Uptime = 6d 22:26:28
429 Queries/sec : 145 Questions/sec
The More Important Issues:
Do you have 3.6GB of RAM? Are you using InnoDB? If yes to both of those, then make these two changes; they may help performance a lot:
key_buffer_size = 40M
innodb_buffer_pool_size = 2G
I'm getting conflicting advice on table_open_cache; let's leave it alone for now.
If you have SSD, I recommend these two:
innodb_io_capicity = 1000
innodb_flush_neighbors = 0
innodb_log_file_size = 200M -- Caution: It may be complex to change this in the version you are running. If so, leave it alone.
You seem to DELETE far more rows than you INSERT; what is going on?
Unless you do a lot of ALTERs, make this change:
myisam_sort_buffer_size = 50M
Details and other observations:
( (key_buffer_size - 1.2 * Key_blocks_used * 1024) ) = ((128M - 1.2 * 0 * 1024)) / 3865470566.4 = 3.5% -- Percent of RAM wasted in key_buffer.
-- Decrease key_buffer_size (now 134217728).
( Key_blocks_used * 1024 / key_buffer_size ) = 0 * 1024 / 128M = 0 -- Percent of key_buffer used. High-water-mark.
-- Lower key_buffer_size (now 134217728) to avoid unnecessary memory usage.
( (key_buffer_size / 0.20 + innodb_buffer_pool_size / 0.70) ) = ((128M / 0.20 + 128M / 0.70)) / 3865470566.4 = 22.3% -- Most of available ram should be made available for caching.
-- http://mysql.rjweb.org/doc.php/memory
( Key_reads + Key_writes + Innodb_pages_read + Innodb_pages_written + Innodb_dblwr_writes + Innodb_buffer_pool_pages_flushed ) = (0 + 0 + 107837817 + 14228075 + 669027 + 14217155) / 599188 = 228 /sec -- IOPs?
-- If the hardware can handle it, set innodb_io_capacity (now 200) to about this value.
( ( Key_reads + Key_writes + Innodb_pages_read + Innodb_pages_written + Innodb_dblwr_writes + Innodb_buffer_pool_pages_flushed ) / innodb_io_capacity / Uptime ) = ( 0 + 0 + 107837817 + 14228075 + 669027 + 14217155 ) / 200 / 599188 = 114.3% -- This may be a metric indicating what innodb_io_capacity is set reasonably.
-- Increase innodb_io_capacity (now 200) if the hardware can handle it.
( Table_open_cache_misses ) = 12,156,771 / 599188 = 20 /sec
-- May need to increase table_open_cache (now 2000)
( Table_open_cache_misses / (Table_open_cache_hits + Table_open_cache_misses) ) = 12,156,771 / (184539214 + 12156771) = 6.2% -- Effectiveness of table_open_cache.
-- Increase table_open_cache (now 2000) and check table_open_cache_instances (now 8).
( innodb_buffer_pool_size ) = 128M -- InnoDB Data + Index cache
-- 128M (an old default) is woefully small.
( innodb_buffer_pool_size ) = 128 / 3865470566.4 = 3.5% -- % of RAM used for InnoDB buffer_pool
-- Set to about 70% of available RAM. (To low is less efficient; too high risks swapping.)
( (key_buffer_size / 0.20 + innodb_buffer_pool_size / 0.70) ) = ((128M / 0.20 + 128M / 0.70)) / 3865470566.4 = 22.3% -- (metric for judging RAM usage)
( innodb_lru_scan_depth ) = 1,024
-- "InnoDB: page_cleaner: 1000ms intended loop took ..." may be fixed by lowering lru_scan_depth
( innodb_io_capacity ) = 200 -- When flushing, use this many IOPs.
-- Reads could be slugghish or spiky.
( innodb_io_capacity_max / innodb_io_capacity ) = 2,000 / 200 = 10 -- Capacity: max/plain
-- Recommend 2. Max should be about equal to the IOPs your I/O subsystem can handle. (If the drive type is unknown 2000/200 may be a reasonable pair.)
( Innodb_log_writes ) = 33,145,091 / 599188 = 55 /sec
( Innodb_os_log_written / (Uptime / 3600) / innodb_log_files_in_group / innodb_log_file_size ) = 67,002,682,368 / (599188 / 3600) / 2 / 48M = 4 -- Ratio
-- (see minutes)
( Uptime / 60 * innodb_log_file_size / Innodb_os_log_written ) = 599,188 / 60 * 48M / 67002682368 = 7.5 -- Minutes between InnoDB log rotations Beginning with 5.6.8, this can be changed dynamically; be sure to also change my.cnf.
-- (The recommendation of 60 minutes between rotations is somewhat arbitrary.) Adjust innodb_log_file_size (now 50331648). (Cannot change in AWS.)
( innodb_flush_method ) = innodb_flush_method = fsync -- How InnoDB should ask the OS to write blocks. Suggest O_DIRECT or O_ALL_DIRECT (Percona) to avoid double buffering. (At least for Unix.) See chrischandler for caveat about O_ALL_DIRECT
( default_tmp_storage_engine ) = default_tmp_storage_engine =
( innodb_flush_neighbors ) = 1 -- A minor optimization when writing blocks to disk.
-- Use 0 for SSD drives; 1 for HDD.
( ( Innodb_pages_read + Innodb_pages_written ) / Uptime / innodb_io_capacity ) = ( 107837817 + 14228075 ) / 599188 / 200 = 101.9% -- If > 100%, need more io_capacity.
-- Increase innodb_io_capacity (now 200) if the drives can handle it.
( innodb_io_capacity ) = 200 -- I/O ops per second capable on disk . 100 for slow drives; 200 for spinning drives; 1000-2000 for SSDs; multiply by RAID factor.
( sync_binlog ) = 0 -- Use 1 for added security, at some cost of I/O =1 may lead to lots of "query end"; =0 may lead to "binlog at impossible position" and lose transactions in a crash, but is faster. 0 is OK for Galera.
( innodb_adaptive_hash_index ) = innodb_adaptive_hash_index = ON -- Usually should be ON.
-- There are cases where OFF is better. See also innodb_adaptive_hash_index_parts (now 8) (after 5.7.9) and innodb_adaptive_hash_index_partitions (MariaDB and Percona). ON has been implicated in rare crashes (bug 73890). 10.5.0 decided to default OFF.
( innodb_print_all_deadlocks ) = innodb_print_all_deadlocks = OFF -- Whether to log all Deadlocks.
-- If you are plagued with Deadlocks, turn this on. Caution: If you have lots of deadlocks, this may write a lot to disk.
( myisam_sort_buffer_size ) = 134,216,704 / 3865470566.4 = 3.5% -- Used for ALTER, CREATE INDEX, OPTIMIZE, LOAD DATA; set when you need it. Also for MyISAM's REPAIR TABLE.
-- Decrease myisam_sort_buffer_size (now 134216704) to keep from blowing out RAM.
( innodb_ft_result_cache_limit ) = 2,000,000,000 / 3865470566.4 = 51.7% -- Byte limit on FULLTEXT resultset. (Possibly not preallocated, but grows?)
-- Lower the setting.
( innodb_autoextend_increment * 1048576 ) = (64 * 1048576) / 3865470566.4 = 1.7% -- How much to increase ibdata1 by (when needed).
-- Decrease setting to avoid premature swapping.
( character_set_server ) = character_set_server = latin1
-- Charset problems may be helped by setting character_set_server (now latin1) to utf8mb4. That is the future default.
( local_infile ) = local_infile = ON
-- local_infile (now ON) = ON is a potential security issue
( Created_tmp_disk_tables / Created_tmp_tables ) = 542,381 / 1084764 = 50.0% -- Percent of temp tables that spilled to disk
-- Maybe increase tmp_table_size (now 16777216) and max_heap_table_size (now 16777216); improve indexes; avoid blobs, etc.
( Com_delete / Com_insert ) = 2,294,352 / 1521534 = 150.8% -- Deletes / Inserts (as a pct). (Ignores LOAD, REPLACE, etc.)
( Com_insert + Com_delete + Com_delete_multi + Com_replace + Com_update + Com_update_multi ) = (1521534 + 2294352 + 21366 + 0 + 45590666 + 0) / 599188 = 82 /sec -- writes/sec
-- 50 writes/sec + log flushes will probably max out I/O write capacity of normal drives
( Com__biggest ) = Com__biggest = Com_stmt_execute -- Which of the "Com_" metrics is biggest.
-- Normally it is Com_select (now 34545111). If something else, then it may be a sloppy platform, or may be something else.
( binlog_format ) = binlog_format = MIXED -- STATEMENT/ROW/MIXED.
-- ROW is preferred by 5.7 (10.3)
( slow_query_log ) = slow_query_log = OFF -- Whether to log slow queries. (5.1.12)
( long_query_time ) = 10 -- Cutoff (Seconds) for defining a "slow" query.
-- Suggest 2
( back_log ) = 80 -- (Autosized as of 5.6.6; based on max_connections)
-- Raising to min(150, max_connections (now 151)) may help when doing lots of connections.
( thread_cache_size / Max_used_connections ) = 151 / 51 = 296.1%
-- There is no advantage in having the thread cache bigger than your likely number of connections. Wasting space is the disadvantage.
You have the Query Cache half-off. You should set both query_cache_type = OFF and query_cache_size = 0 . There is (according to a rumor) a 'bug' in the QC code that leaves some code on unless you turn off both of those settings.
Abnormally small:
(Com_select + Qcache_hits) / (Com_insert + Com_update + Com_delete + Com_replace) = 0.699
Com_show_tables = 0
Innodb_buffer_pool_bytes_data = 193 /sec
Table_locks_immediate = 0.53 /HR
eq_range_index_dive_limit = 0
innodb_spin_wait_delay = 4
Abnormally large:
Com_delete_multi = 0.036 /sec
Com_stmt_close = 141 /sec
Com_stmt_execute = 141 /sec
Com_stmt_prepare = 141 /sec
Handler_discover = 0.94 /HR
Innodb_buffer_pool_read_ahead = 162 /sec
Innodb_buffer_pool_reads * innodb_page_size / innodb_buffer_pool_size = 114837.2%
Innodb_data_pending_fsyncs = 2
Innodb_os_log_fsyncs = 55 /sec
Opened_plugin_libraries = 0.006 /HR
Table_open_cache_active_instances = 4
Tc_log_page_size = 4,096
Abnormal strings:
aria_recover_options = BACKUP,QUICK
innodb_fast_shutdown = 1
log_slow_admin_statements = ON
myisam_stats_method = NULLS_UNEQUAL
old_alter_table = DEFAULT