I'm writing a multithreaded program to check the performance of a database server. In my program, one thread should access(read/wright) the database for a given rate(eg. 100 times per second) But my problem is, how can I access the database for the given rate. How can I send 1000 requests in one second to the database.
Related
How can I calculate the optimal total connection count from my service to the my DB endpoint? Is there a basic formula based on expected number of queries per second and CPU and IO taken by each query?
Similarly, is there a formula to calculate the optimal database instance type/size to use based on traffic patterns and query characteristics (CPU, IO consumed or latency of query)?
I will be using this to create the connection pool in my service. I'm assuming that if my service has N hosts then per host the connection pool size need to be the total optimal connection count divided by N.
Note: By instance type I mean similar to AWS EC2 instance type which provides info on vCPU and memory (RAM)
When it comes to sizing the database machine, the key measure is the number of concurrently active database sessions. Since each active session corresponds to at least one database process, you have to provide enough CPU power and I/O capacity to handle them concurrently.
An estimate for that number would be average query duration in seconds * number of queries per second. You have to have at least that many cores, and your I/O system has to be able to handle that many concurrent I/O requests.
When it comes to dimensioning a connection pool, you also have to factor in the time that the database spends idle in a transaction while waiting for the next statement from the client.
The maximal connection pool size would be number of concurrent queries the database can handle / (transaction busy ratio * average query duration in seconds * number of queries per second). The transaction busy ratio is active time for a transaction / total time for a transaction - so if all your transactions consist of only a single statement (which means that no time is spent waiting for the next statement in a transaction), that ratio would be 1.
In practice, it is difficult to estimate or measure the ideal pool size, and you have to run load tests to see how big you can make the pool without overloading your database.
A generally assessed poor technique is to create an own database session for every atomic DB activity.
You may sometimes encounter such strategies like:
processing a large amount of items in a loop, each processing step in the loop creates a DB session, executes a small set of SQL statements and terminates the session
a polling process checks a SQL result one time a second, each in a new DB session
But what costs are generated by frequently connecting and disconnecting DB session?
The internal recording of database activity (AWR/ASH) has no answer because establishing the DB connection is not a SQL activity.
The superficial practical answer depends how you define 'connection' - is a connection what the app knows as a connection, or is it the network connection to the DB, or is it the DB server process & memory used to do any processing? The theoretical overall answer is that the process of establishing some application context and starting a DB server process with some memory allocation included - and then doing the reverse when the app has finished running SQL statements - is 'expensive'. This was measured in Peter Ramm's answer.
In practice, long running applications that expect to handle a number of users would create a connection pool (e.g. in Node.js or in Python). These remain open for the life of the application. From the application's point of view, getting a connection from the pool to do some SQL is a very quick operation. The initial cost (a few seconds of startup at most) of creating the connection pool can be amortized over the process life of the application.
The number of server processes (and therefore overhead costs) on the database tier can be reduced by additional use of a 'Database Resident Connection Pool'.
These connection pools have other benefits for Oracle in terms of supporting Oracle's High Availability features, often transparently. But that's off topic.
A simple comparison of system load gives a fuzzy hint to the price of connection creation.
Example:
An idle database instance on a single host with 4 older CPU cores (Intel Xeon E312xx, 2,6 GHz)
a external (not on DB host) SQLPlus client which executes a single "SELECT SYSTIMESTMP FROM DUAL" per DB session
Delay between the SQLPlus calls is time so that 1 connection per second is created and destroyed.
6 Threads active each with 1 session creation per second
Result:
with idle database CPU load over 4 CPU nodes is in average 0.22%
with 6 threads creating and destroying sessions each second CPU load is 6.09%
io wait also occurs with 1.07% in average
so in average 5.87% of 4 CPU nodes are allocated by this 6 threads
Equivalent to 23.48% of one CPU node for 6 threads or 3,91% per thread
That means:
Connecting and disconnecting an Oracle DB session once per second costs approximately 4% of a CPU core of DB server.
This value in mind should help to consider if it's worth to change process behavior regarding session creation or not.
p.s.: This does not consider the additional cost of session creation at client side.
I am using Postgres Database as primary database with Rails application. Currently the configuration for the database is: CPU - 8 cores, RAM - 32 GB, hosted on AWS. However I need to scale this server as I am facing a lot of serious bottlenecks.
I need to be able to handle 10000 parallel requests to this server. Even if this is not achievable I at-least need to know what would be the maximum number that this database can handle. The requests includes complex SELECT and UPDATE queries.
I have changed settings such as max_connections to 10000 and in the rails_config/database.yml the connection_pool is set to 10000.
I am running a rails code which currently runs in 50 parallel threads. The code runs fine for a while until I receive this error:
PG::ConnectionBad: PQsocket() can't get socket descriptor. After this when I log into the postgres server and try to execute something, I get this:
db_production=# select id from product_tbl limit 5;
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Succeeded.
db_production=#
After this restarting the rails code also works fine but only for a while until I again receive the same error.
I looked at the CPU usage while the threads are running. Some observations:
One SELECT query uses all the CPU (shows 100% usage) for one core, the rest are sitting idle at 0%. This can be inferred because when only one process or query is running, only one CPU shows 95-100% usage. Does this mean I will only be able to run 8 queries or threads in parallel as there are only 8 cores?
RAM is kept underutilized. Even when 7 out of 8 queries or threads are running in parallel, the RAM used is only around 3 GB. Can I somehow increase this and accommodate more parallel queries?
For every running thread, this is the configuration: virtual memory usage->751M, resident memory usage->154-158M, shared memory used->149-150M, CPU%->60-100%, MEM%->0.5-0.6%. Is everything right here or I need to tweak some settings to make use of the resources more effectively?
What are the configuration settings I need to change in order to scale the database? What must be the ideal number of threads such that I at-least don't receive the connection error? What are the checks and steps to scale the database to handle large parallel connections? How can I make full use of the available resources?
Last week we updated our DB Password and ever since after every db bounce the connections are getting filled up.
We have 20+ schema and connections to only one Schema gets filled up. Nothing shows up in the sessions. There can be old apps accessing our database with old password and filling up connections.
How to identify how many processes are trying to connect to DB server and how many are failed.
Every time we bounce our db servers connections go through post 1hr no one else can make new connections.
BTW: in our company, we have LOGON and LOGOFF triggers which persist the session connect and disconnect information.
It is quite possible that what you are seeing are recursive sessions created by Oracle when it needs to parse SQL statements [usually not a performance problem, but processes parameter may need to be increased]: ...
for example 1, high values for dynamic_sampling cause more recursive SQL to be generated ;
example 2: I have seen a situation for this application of excessive hard parsing; this will drive up the process count as hard parsing will require new processes to execute parse related recursive SQL (increased the processes parameter in this case since it was a vendor app). Since your issue is related to the bounce, it could be that the app startup requires a lot of parsing.
Example 3:
“Session Leaking” Root Cause Analysis:
Problem Summary: We observed periods where many sessions being created, without a clear understanding of what part of the application is creating them and why.
RCA Approach: Since the DB doesn't persist inactive sessions, I monitored the situation by manually snapshotting v$session.
Analysis:
I noticed a pattern where multiple sessions have the same process#.
As per Oracle doc’s, these sessions are recursive sessions created by oracle under an originating process which needs to do recursive SQL to satisfy the query (at parse level). They go away when the process that created them is done and exits.
If the process is long running, then they will stay around inactive until it is done.
These recursive sessions don't count against your session limit and the inactive sessions are in an idle wait event and not consuming resources.
The recursive session are most certainly a result of recursive SQL needed by the optimizer where optimizer stats are missing (as is the case with GTT’s) and the initialization parameter setting of 4 for optimizer_dynamic_sampling .
The 50,000 sessions in an hour that we saw the other day is likely a result of a couple thousand select statements running (I’ve personally counted 20 recursive sessions per query, but this number can vary).
The ADDM report showed that the impact is not much:
Finding 4: Session Connect and Disconnect
Impact is .3 [average] active sessions, 6.27% of total activity [currently on the instance].
Average Active Sessions is a measure of database load (values approaching CPU count would be considered high). Your instance can handle up to 32 active sessions, so the impact is about 1/100th of the capacity.
I create job arrays of thousands of simulations that get executed on a network connected cluster of servers which all have local disk as well being connected to NFS disc drives.
Is there a database that can be distributed amongst the servers that operates in the following way:
When I submit my job array each individual job running on an individual server to send results to the distributed DB.
Whilst the job array is still running the user can request partial summaries from the DB - the DB having the option to not wait for all the latest results from all its distributed nodes, but to "improvise" in some way
The user can request a full summary after the job array is finished which causes the DB to ensure that it returns an accurate summary of all data from all its nodes and further, that all nodes are not still receiving data from jobs (quiescent for a stated time).
In other words, I want a fast DB and an accurate DB when I tell it, receiving lots of data from thousands of jobs in an LSF job array. I need to monitor the progress of the LSF job array's results but am willing to forego some accuracy when monitoring to increase speed, but need an accurate result when all is done.
The data being stored for each job is small job ID, small PASS/FAIL, large how job fails. The how is likely to be only spot-checked for a very few jobs until the end of all jobs in the job-array when triage scripts will need access to all DB data for the job array at speed.