We're being asked to spec out production database hardware for an ASP.NET web application that hasn't been built yet.
The specs we need to determine are:
Database CPU
Database I/O
Database RAM
Here are the metrics I'm currently looking at:
Estimated number of future hits to
website - based on current IIS logs.
Estimated worst-case peak loads to
website.
Estimated number of DB queries per
page, on average.
Number of servers in web farm that
will be hitting database.
Cache polling traffic from database
(using SqlCacheDependency).
Estimated data cache misses.
Estimated number of daily database transactions.
Maximum acceptable page render time.
Any other metrics we should be taking into account?
Also, once we have all those metrics in place, how do they translate into hardware requirements?
What I have been doing lately for server planning is using some free tools that HP provides, which are collectively referred to as the "server sizers". These are great tools because they figure out the optimal type of RAID to use, and the correct number of disk spindles to handle the load (very important when planning for a good DB server) and memory processor etc. I've provided the link below I hope this helps.
http://h71019.www7.hp.com/ActiveAnswers/cache/70729-0-0-225-121.html?jumpid=reg_R1002_USEN
What I am missing is a measure for the needed / required / defined level of reliability.
While you could probably spec out a big honking machine to handle all the load, depending on your reliabiltiy requirements, you might rather want to invest in smaller, but multiple machines, and into safer disk subsystems (RAID 5).
Marc
In my opinion, estimating hardware for an application that hasn't been built and designed yet is more of a political issue than a scientific issue. By the time you finish the project, current hardware capability and their price, functional requirements, expected number of concurrent users, external systems and all other things will change and this change is beyond your control.
However this question comes up very often since you need to put numbers in a proposal or provide a report to your manager. If it is a proposal, what you are trying to accomplish is to come up with a spec that can support the proposed sofware system. The only trick is to propose a system that will not increase your cost for competiteveness while not puting yourself at the risk of a low performance system.
If you can characterize your current workload in terms of hits to pages, then you can then:
1) calculate the typical type of query that will be done for each page
2) using the above 2 pieces of information, estimate the workload on the database server
You also need to determine your performance requirements - what is the max and average response time you want for your website?
Given the workload, and performance requirements, you can then calculate capacity. The best way to make this estimate is to use some existing hardware, run a simulated database workload on a database on that hardware, and then extrapolate your hardware requirements based on your data from the first steps.
Related
I'm building a web service, consisting of many different components, all of which could conceivably be bottlenecks. I'm currently trying to figure out what metrics I should be looking for, when deciding whether or not my database (on AWS RDS) is the bottleneck in the chain.
Looking at AWS Cloudwatch, I see a number of RDS metrics given. Full list:
CPUCreditBalance
CPUCreditUsage
CPUUtilization
DatabaseConnections
DiskQueueDepth
FreeStorageSpace
FreeableMemory
NetworkReceiveThroughput
NetworkTransmitThroughput
ReadIOPS
ReadLatency
ReadThroughput
SwapUsage
WriteIOPS
WriteLatency
WriteThroughput
The key metrics that I think I should be paying attention to:
Read/Write Latency
CPU-Utilization
Freeable Memory
With the latency metrics, I'm thinking that I should set up alerts if it exceeds >300ms (for fast website responsiveness), though I recognize that this is very much workload dependent.
With the CPU/memory-util, I have no idea what numbers to set these to. I'm thinking I should set an alert for 75% CPU-utilization, and 75% drop in Freeable Memory.
Am I on the right track with the metrics I've shortlisted above, and the thresholds I have guessed? Are there any other metrics I should be paying attention to?
The answer is totally dependent on your application. Some applications will require more CPU, some will need more RAM. There is no definitive answer.
The best thing is to monitor your database (with the metrics you list above). Then, when performance is below desired, take a look at which metrics are showing problems. These should be the first ones you track for scaling your database.
The key idea that if your customers are experiencing problems, it should be appearing in your metrics somewhere. If this isn't the case, then you're not collecting sufficient metrics.
I think you are on the right track - especially with the latency metrics; for a typical application with database back-end, the read/write latency is going to be what the user notices most if it degrades. Sure the memory or cpu usage may spike, but does any user care? No, not unless it then causes the latency to go up.
I'd start with the metrics you listed as the low-hanging fruit and adjust accordingly.
I'm working on a system that will need to store lots of temperature data. I could potentially store 5 samples per second or more.
I've done this in the past with a relatively simple mysql database and performance became unbearable. Inserts were not too bad, but had a noticeable load. Queries, however, could take minutes.
At that time, I had something like 50 gb of data, which is ridiculous. I can think of many ways to compress or discard data without losing critical information, but that's a completely different problem.
I'd like to pick a tool/database that is optimized for this kind of data, preferably cross platform (at a minimum, linux/c++).
RRD (Round Robin Database) seems built for this kind of thing, but it seems designed more for processing data than for storing it.
What other tools are available?
Edit: more info...
This will be running on an embedded system (a Raspberry Pi) so an ideal tool has low computing overhead, low memory footprint, and few library dependencies.
The storage may not necessarily be on the same device.
I suppose growth could reach as much as 500k samples per hour in a contrived, extreme case. More likely it will be about 20k samples per hour.
Internet access should not be presumed.
Looks like you're looking for a time series DB.
I know of two candidates:
http://tempo-db.com/
http://opentsdb.net/
If you could be a bit more specific about your requirements (req/s, daily data growth, API type, self-hosted or a fully managed solution, etc), I'd be able to go into more details or recommend other solutions.
Good luck.
We're working on an application that's going to serve thousands of users daily (90% of them will be active during the working hours, using the system constantly during their workday). The main purpose of the system is to query multiple databases and combine the information from the databases into a single response to the user. Depending on the user input, our query load could be around 500 queries per second for a system with 1000 users. 80% of those queries are read queries.
Now, I did some profiling using the SQL Server Profiler tool and I get on average ~300 logical reads for the read queries (I did not bother with the write queries yet). That would amount to 150k logical reads per second for 1k users. Full production system is expected to have ~10k users.
How do I estimate actual read requirement on the storage for those databases? I am pretty sure that actual physical reads will amount to much less than that, but how do I estimate that? Of course, I can't do an actual run in the production environment as the production environment is not there yet, and I need to tell the hardware guys how much IOPS we're going to need for the system so that they know what to buy.
I tried the HP sizing tool suggested in the previous answers, but it only suggests HP products, without actual performance estimates. Any insight is appreciated.
EDIT: Main read-only dataset (where most of the queries will go) is a couple of gigs (order of magnitude 4gigs) on the disk. This will probably significantly affect the logical vs physical reads. Any insight how to get this ratio?
Disk I/O demand varies tremendously based on many factors, including:
How much data is already in RAM
Structure of your schema (indexes, row width, data types, triggers, etc)
Nature of your queries (joins, multiple single-row vs. row range, etc)
Data access methodology (ORM vs. set-oriented, single command vs. batching)
Ratio of reads vs. writes
Disk (database, table, index) fragmentation status
Use of SSDs vs. rotating media
For those reasons, the best way to estimate production disk load is usually by building a small prototype and benchmarking it. Use a copy of production data if you can; otherwise, use a data generation tool to build a similarly sized DB.
With the sample data in place, build a simple benchmark app that produces a mix of the types of queries you're expecting. Scale memory size if you need to.
Measure the results with Windows performance counters. The most useful stats are for the Physical Disk: time per transfer, transfers per second, queue depth, etc.
You can then apply some heuristics (also known as "experience") to those results and extrapolate them to a first-cut estimate for production I/O requirements.
If you absolutely can't build a prototype, then it's possible to make some educated guesses based on initial measurements, but it still takes work. For starters, turn on statistics:
SET STATISTICS IO ON
Before you run a test query, clear the RAM cache:
CHECKPOINT
DBCC DROPCLEANBUFFERS
Then, run your query, and look at physical reads + read-ahead reads to see the physical disk I/O demand. Repeat in some mix without clearing the RAM cache first to get an idea of how much caching will help.
Having said that, I would recommend against using IOPS alone as a target. I realize that SAN vendors and IT managers seem to love IOPS, but they are a very misleading measure of disk subsystem performance. As an example, there can be a 40:1 difference in deliverable IOPS when you switch from sequential I/O to random.
You certainly cannot derive your estimates from logical reads. This counter really is not that helpful because it is often unclear how much of it is physical and also the CPU cost of each of these accesses is unknown. I do not look at this number at all.
You need to gather virtual file stats which will show you the physical IO. For example: http://sqlserverio.com/2011/02/08/gather-virtual-file-statistics-using-t-sql-tsql2sday-15/
Google for "virtual file stats sql server".
Please note that you can only extrapolate IOs from the user count if you assume that cache hit ratio of the buffer pool will stay the same. Estimating this is much harder. Basically you need to estimate the working set of pages you will have under full load.
If you can ensure that your buffer pool can always take all hot data you can basically live without any reads. Then you only have to scale writes (for example with an SSD drive).
When improving database performance, What is the fastest way?
should I go straight to my target (5000 users)?
Or should I go through artificial milestones - (for example - first 1000 users, only then 2500...)?
we have a performance problem. For the purpose of the question, suppose the only bottleneck is the Database and all other resources consume zero time.
Facts:
Database performance for 50 concurrent users is good
Database performance for more then 50 is not acceptable and by that I mean database requests takes too long. Instead of ~0.5 sec they usually take between ~2 and ~7 sec, and sometimes even ~30 sec
On about 70 users, the system stop working at all, most database requests takes more then ~30 sec
Database is implemented in SQL SERVER 2005\8
Requirement:
The application need to support 5000 concurrent users.
From my understanding, and please correct me if I'm wrong or there is something missing, the best practise for database performance improvement is:
Define performance objective targets.
Establish testing environment identical (as much as possible) to production environment
Check if the performance targets is OK. if not do the following until performance targets is OK
3.1 Improve performance (on at a time) by - server re-configuration, indexes improvements ,code improvements, database structure improvements and more, using trace and monitor tools such as - performance counters, SQL Server profiler, logs and more
3.2 Test and analyse the affects of the change and accept or reject it
My question is:
Assuming that getting the system to work on anything else then 5000, say for example 1000, or even 2500 users Doesn't provide any business value, What is the fastest way to get to 5000 users?
Getting there directly?
Or first get to smaller milestones?
for example - first target 500 users, then 1000, then 2500 ....
Something else??
Do I get any technical value from using milestones, that will eventually take me to 5000 users faster?
Tuning up to 500 users, for example, can lead you to solutions that won't be effective when you increase to 1000 users. Example: let's imagine that you can tune cache for 500 users, perhaps you won't have enought RAM to get a good result for 5,000.
Go for your goal.
I don't see that artificial milestones really add any value, if the requirement if 5000 concurrent users, go straight for that, otherwise you run the risk of wasting time making optimisations which don't scale.
I would say that getting small milestone would make you a problem, imagine that you get a small number of users (or a very big one) you would have to find a point where the milestones are not too big nor too small. That can be tricky.
We are looking at stress testing a NAS system for a Database, basically was want to see how much abuse it can take and how much it affects the Database performance. here is what we have planed
I have a test Tool I'm building that will kickoff a configurable number of
threads that run a sql query (also configurable, and thinking about
having it able to run multiple quires)
using the SQLIOSim utility to simulate SQL Server activity
Copying very large amounts of Data onto and off of the device (at the same time)
Can anyone think of anything else we could do (that's repeatable) for placing a load on the system.
You'll also want to simulate network conditions between your database and the NAS. As more traffic hits a network, its realizable utilization drops, and this will seriously affect your performance.
By way of example. If you have 50 machines on a 1Gbps network and the network is approaching 100% utilization, packet collisions and retries at the data link layer mean that your effective total transfer is a fraction of the network potential that would be realized if you had only two communicators on the net. Worse yet, as retries increase, so does effective load. You get an ugly feedback loop in the face of peak demand.
There are a number of network traffic simulators and generators out there, though I'm afraid I've never used any of them.
You could look at Pole Position, a generic database performance testing suite.
http://www.polepos.org/
Depending on the goals you wish to achieve with your load testing you may also want to look at using SQLIO & not just SQLIOSim. SQLIOSim is very good for stress testing & simulating SQL Server load & will go from green to red if it detects any IO errors. It's output is a bit cryptic though although KKline gives some insight.
SQLIO is useful if you want to perform one operation continuously such as large random reads, or large sequential writes & anything in between. It will also give you some useful output stats which you can graph & use as a comparison.
You can try IoMeter from Intel