sqlite3 transaction on 200 columns table takes long to complete sporadically - database

I am using sqlite3 as db manager for my application, developed on a rapsberry pi3.
My table is composed of around 200 columns (not so much), mostly boolean and numeric fields.
I add a (complete) record every minute. DB is accessed in a C program using transactions.
the transaction includes one insert and 6 updates (to maintain the code readable), avoiding to write a very long single insertion query.
The db file is on the filesystem (hence on the sd card) inside the home folder.
Every transsaction the db is opened, the pragmas
PRAGMA synchronous = NORMAL;
PRAGMA journal_mode = WAL;
are set and the query is performed.
I have good performance averagely but from the timing log I see a peak every once in a while.
Extract of the log is reported:
Apr 28 07:06:13 db write took 45.200000 ms
Apr 28 07:07:13 db commit took 0.302000 ms
Apr 28 07:07:13 db write took 75.858000 ms
Apr 28 07:08:13 db commit took 0.354000 ms
Apr 28 07:08:13 db write took 75.395000 ms
Apr 28 07:09:13 db commit took 0.268000 ms
Apr 28 07:09:13 db write took 40.620000 ms
Apr 28 07:10:13 db commit took 0.437000 ms
Apr 28 07:10:13 db write took 81.910000 ms
Apr 28 07:11:13 db commit took 0.205000 ms
Apr 28 07:11:13 db write took 43.315000 ms
Apr 28 07:12:13 db commit took 0.301000 ms
Apr 28 07:12:13 db write took 75.456000 ms
Apr 28 07:13:15 db commit took 1872.488000 ms <-----
Apr 28 07:13:15 db write took 1951.572000 ms <-----
Apr 28 07:14:13 db commit took 7.934000 ms
Apr 28 07:14:13 db write took 62.853000 ms
Apr 28 07:15:13 db commit took 0.274000 ms
Apr 28 07:15:13 db write took 80.568000 ms
Apr 28 07:16:13 db commit took 0.277000 ms
The arrow points to one of the time peak that are recurring (with variable periods) during the execution.
To bettere understand the situation, analyzing the benchmark I had two peaks in the last 12 hours, one is about 1 sec (not reported) and this one.
Could the time peaks happen because of filesystem activity on the sd?
Could making a different partition on the sd card have an impact on such performance?
Is there any other pragma that could protect my application from this behaviour?
Adding the pragmas has significantly improved the situation so far but I think is not acceptable yet.
Thanks for your time and patience.
Any hint is welcomed.
Regards,
mopyot

The database regularly moves the data from the write-ahead log into the actual database file; this is called checkpointing:
By default, SQLite will automatically checkpoint whenever a COMMIT occurs that causes the WAL file to be 1000 pages or more in size, or when the last database connection on a database file closes. […]
But programs that want more control can force a checkpoint using the wal_checkpoint pragma … The automatic checkpoint threshold can be changed or automatic checkpointing can be completely disabled using the wal_autocheckpoint pragma …

Related

Why does Redis save partial data sets?

Version: Redis 5.0.3
In redis.conf there is an option to set snapshot period. I set this period as every 5 sec if one value is changed to see the Redis performance when it dumps. I ran two application; one is redis-server and the other is redis-benchmark.
While I was watching a log, I found out some interesting thing like below.
7269:C 27 Feb 2019 14:48:39.463 * RDB: 4535 MB of memory used by copy-on-write
7257:M 27 Feb 2019 14:48:39.939 * Background saving terminated with success
7257:M 27 Feb 2019 14:48:45.085 * 10 changes in 5 seconds. Saving...
7257:M 27 Feb 2019 14:48:45.187 * Background saving started by pid 7270
7270:C 27 Feb 2019 14:49:00.313 * DB saved on disk
7270:C 27 Feb 2019 14:49:00.401 * RDB: 4535 MB of memory used by copy-on-write
7257:M 27 Feb 2019 14:49:00.882 * Background saving terminated with success
7257:M 27 Feb 2019 14:49:06.011 * 10 changes in 5 seconds. Saving...
7257:M 27 Feb 2019 14:49:06.114 * Background saving started by pid 7271
7271:C 27 Feb 2019 14:49:21.086 * DB saved on disk
7271:C 27 Feb 2019 14:49:21.173 * RDB: 4534 MB of memory used by copy-on-write
7257:M 27 Feb 2019 14:49:21.706 * Background saving terminated with success
7257:M 27 Feb 2019 14:49:27.048 * 10 changes in 5 seconds. Saving...
7257:M 27 Feb 2019 14:49:27.155 * Background saving started by pid 7273
7273:C 27 Feb 2019 14:49:42.295 * DB saved on disk
7273:C 27 Feb 2019 14:49:42.382 * RDB: 4529 MB of memory used by copy-on-write
7257:M 27 Feb 2019 14:49:42.846 * Background saving terminated with success
7257:M 27 Feb 2019 14:49:48.023 * 10 changes in 5 seconds. Saving...
7257:M 27 Feb 2019 14:49:48.126 * Background saving started by pid 7274
7274:C 27 Feb 2019 14:50:05.251 * DB saved on disk
7274:C 27 Feb 2019 14:50:05.367 * RDB: 15 MB of memory used by copy-on-write
7257:M 27 Feb 2019 14:50:05.583 * Background saving terminated with success
As you can see, the dumped data has almost same size with others and the last one is even little. The thing that I don't understand is why the size is the same and why the last one has small size. (While the redis is dumping, the client requests set operation and the last dump probably means the end of set operation and start of get operation.)
To find out the reason, I look up the code but still don't know why the number has shown like above.
If you see rdb.c in the redis package, you can find this kind of source code.
int rdbSave(char *filename, rdbSaveInfo *rsi) {
...
snprintf(tmpfile, 256, "temp-%d.rdb", (int) getpid());
fp = fopen(tmpfile, "w");
...
rdbSaveRio(...);
}
From my understanding, everytime redis dumps the in-memory data, it should overwrite the previous saved data and this data should be lager than before. However, based on the log, the size is not linearly increased and even it is decreased at the last dump.
Am I missing some part of Redis features?
Edit
According to the comment, I definitely misinterpreted the logs. However, I still have a question about the performance. When snapshotting happens and if there are private dirty memories, the system saves them into the disk. In this point, based on the Redis dump mechanism, although the system sees the private dirty memory caused by set operation and only records this number on the log, it saves the all data from the memory. It means that everytime dumping happens, the size of disk expands and I'm pretty sure that it would lead to worse performance. But, when I see the result of benchmark, I can see the same performance drop despite of increasing disk size. I wonder why this shows the same drop rate and what is going on internally.
graph
In the above graph, the blue line indicates the throughput and you can see that it drops when snapshot happens and you can also notice that even though the second drop phase saves the larger disk size than the first phase, the drop rate is the same. So, my question would be is performance only affected by the private memory saving?

Oracle db stops working frequently

Recently, I faced a problem with Oracle 11g. It stops working frequently, every few hours, and respectively, must be started again. This was never happened before, it just began during last month.
Here is the content of clsc.log file
2016-10-06 07:00:14.344: [ default][2031720192]utgdv:2:ocr loc file /etc/oracle/olr.loc cannot be opened. errno 2
[ CLSE][2031720192]clse_get_crs_home: Error retrieving OLR configuration [0] [Error opening olr.loc file. No such file or directory]
Can anyone help me?
Edit: There is alert log messages like here, each time instance has stopped working (In this case, it has stopped working on Wed night and started again on Thu)
Thread 1 advanced to log sequence 35012 (LGWR switch)
Current log# 2 seq# 35012 mem# 0: /home/db/app/oracle/oradata/orcl/redo02.log
Wed Oct 05 18:53:08 2016
Thread 1 advanced to log sequence 35013 (LGWR switch)
Current log# 3 seq# 35013 mem# 0: /home/db/app/oracle/oradata/orcl/redo03.log
Wed Oct 05 19:05:08 2016
Thread 1 advanced to log sequence 35014 (LGWR switch)
Current log# 1 seq# 35014 mem# 0: /home/db/app/oracle/oradata/orcl/redo01.log
Wed Oct 05 19:15:03 2016
Thread 1 advanced to log sequence 35015 (LGWR switch)
Current log# 2 seq# 35015 mem# 0: /home/db/app/oracle/oradata/orcl/redo02.log
Thu Oct 06 06:41:11 2016
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
.
.
.

Schedule a DB job based on specific date

I have DB job and now I would like to run this job only on specific dates. dates could be in any month like first run on 15 May 2015 and next run woule be 20 July 2015 .. 11 OCT 2015 etc etc.
Through scheduler we dont see such fluxibility. I guess this can be done through some function and call that in scheduler.
Could someone please help me on that.
Your task can be solved without using any additional functions, you can create a schedule with FREQ=YEARLY and with BYDATE clause you can specify an exact dates. For example:
BEGIN
dbms_scheduler.create_schedule('my_sched',
repeat_interval => 'FREQ=YEARLY;BYDATE=0515,0720,1011');
END;
This plsql block will create the schedule named my_sched which contains 15 May, 20 July and 11 OCT of every year. You can use this schedule while creating your jobs.
You can read more about oracle calendaring syntax here: http://docs.oracle.com/cd/B19306_01/appdev.102/b14258/d_sched.htm#BABEJGCH

Apache2: server-status reported value for "requests/sec" is wrong. What am I doing wrong?

I am running Apache2 on Linux (Ubuntu 9.10).
I am trying to monitor the load on my server using mod_status.
There are 2 things that puzzle me (see cut-and-paste below):
The CPU load is reported as a ridiculously small number,
whereas, "uptime" reports a number between 0.05 and 0.15 at the same time.
The "requests/sec" is also ridiculously low (0.06)
when I know there are at least 10 requests coming in per second right now.
(You can see there are close to a quarter million "accesses" - this sounds right.)
I am wondering whether this is a bug (if so, is there a fix/workaround),
or maybe a configuration error (but I can't imagine how).
Any insights would be appreciated.
-- David Jones
- - - - -
Current Time: Friday, 07-Jan-2011 13:48:09 PST
Restart Time: Thursday, 25-Nov-2010 14:50:59 PST
Parent Server Generation: 0
Server uptime: 42 days 22 hours 57 minutes 10 seconds
Total accesses: 238015 - Total Traffic: 91.5 MB
CPU Usage: u2.15 s1.54 cu0 cs0 - 9.94e-5% CPU load
.0641 requests/sec - 25 B/second - 402 B/request
11 requests currently being processed, 2 idle workers
- - - - -
After I restarted my Apache server, I realized what is going on. The "requests/sec" is calculated over the lifetime of the server. So if your Apache server has been running for 3 months, this tells you nothing at all about the current load on your server. Instead, reports the total number of requests, divided by the total number of seconds.
It would be nice if there was a way to see the current load on your server. Any ideas?
Anyway, ... answered my own question.
-- David Jones
Apache status value "Total Accesses" is total access count since server started, it's delta value of seconds just what we mean "Request per seconds".
There is the way:
1) Apache monitor script for zabbix
https://github.com/lorf/zapache/blob/master/zapache
2) Install & config zabbix agentd
UserParameter=apache.status[*],/bin/bash /path/apache_status.sh $1 $2
3) Zabbix - Create apache template - Create Monitor item
Key: apache.status[{$APACHE_STATUS_URL}, TotalAccesses]
Type: Numeric(float)
Update interval: 20
Store value: Delta (speed per second) --this is the key option
Zabbix will calculate the increment of the apache request, store delta value, that is "Request per seconds".

Sybase: what device names does a database use?

Does anyone know how I can query the database to find what devices a database uses?
There is the sysdatabases table and the sysdevices table but I don't know how to link them
Anyone know?
The best way is to run sp_helpdb against the database you're interested in:
1> sp_helpdb tempdb2
2> go
... other stuff here...
device_fragments size usage created free kbytes
------------------------------ ------------- -------------------- ------------------------- ----------------
tempdb2data 2048.0 MB data only Dec 17 2008 11:42AM 2086568
tempdb2log 2048.0 MB log only Dec 17 2008 11:42AM not applicable
tempdb2log 2048.0 MB log only Dec 17 2008 11:42AM not applicable
tempdb2data 2048.0 MB data only Dec 17 2008 11:43AM 2088960
tempdb2log 4096.0 MB log only Dec 17 2008 11:44AM not applicable
--------------------------------------------------------------
log only free kbytes = 8355836
1 Just a note re your first question. If you USE database first, you will get even more detail in the report.
2 Do you still need the second question answered, how to link sysdatabase and sysdevices, as in, are you writing queries against the catalogue ? If so, I need your ASE version, the answers are different.

Resources