We're working on an application that's going to serve thousands of users daily (90% of them will be active during the working hours, using the system constantly during their workday). The main purpose of the system is to query multiple databases and combine the information from the databases into a single response to the user. Depending on the user input, our query load could be around 500 queries per second for a system with 1000 users. 80% of those queries are read queries.
Now, I did some profiling using the SQL Server Profiler tool and I get on average ~300 logical reads for the read queries (I did not bother with the write queries yet). That would amount to 150k logical reads per second for 1k users. Full production system is expected to have ~10k users.
How do I estimate actual read requirement on the storage for those databases? I am pretty sure that actual physical reads will amount to much less than that, but how do I estimate that? Of course, I can't do an actual run in the production environment as the production environment is not there yet, and I need to tell the hardware guys how much IOPS we're going to need for the system so that they know what to buy.
I tried the HP sizing tool suggested in the previous answers, but it only suggests HP products, without actual performance estimates. Any insight is appreciated.
EDIT: Main read-only dataset (where most of the queries will go) is a couple of gigs (order of magnitude 4gigs) on the disk. This will probably significantly affect the logical vs physical reads. Any insight how to get this ratio?
Disk I/O demand varies tremendously based on many factors, including:
How much data is already in RAM
Structure of your schema (indexes, row width, data types, triggers, etc)
Nature of your queries (joins, multiple single-row vs. row range, etc)
Data access methodology (ORM vs. set-oriented, single command vs. batching)
Ratio of reads vs. writes
Disk (database, table, index) fragmentation status
Use of SSDs vs. rotating media
For those reasons, the best way to estimate production disk load is usually by building a small prototype and benchmarking it. Use a copy of production data if you can; otherwise, use a data generation tool to build a similarly sized DB.
With the sample data in place, build a simple benchmark app that produces a mix of the types of queries you're expecting. Scale memory size if you need to.
Measure the results with Windows performance counters. The most useful stats are for the Physical Disk: time per transfer, transfers per second, queue depth, etc.
You can then apply some heuristics (also known as "experience") to those results and extrapolate them to a first-cut estimate for production I/O requirements.
If you absolutely can't build a prototype, then it's possible to make some educated guesses based on initial measurements, but it still takes work. For starters, turn on statistics:
SET STATISTICS IO ON
Before you run a test query, clear the RAM cache:
CHECKPOINT
DBCC DROPCLEANBUFFERS
Then, run your query, and look at physical reads + read-ahead reads to see the physical disk I/O demand. Repeat in some mix without clearing the RAM cache first to get an idea of how much caching will help.
Having said that, I would recommend against using IOPS alone as a target. I realize that SAN vendors and IT managers seem to love IOPS, but they are a very misleading measure of disk subsystem performance. As an example, there can be a 40:1 difference in deliverable IOPS when you switch from sequential I/O to random.
You certainly cannot derive your estimates from logical reads. This counter really is not that helpful because it is often unclear how much of it is physical and also the CPU cost of each of these accesses is unknown. I do not look at this number at all.
You need to gather virtual file stats which will show you the physical IO. For example: http://sqlserverio.com/2011/02/08/gather-virtual-file-statistics-using-t-sql-tsql2sday-15/
Google for "virtual file stats sql server".
Please note that you can only extrapolate IOs from the user count if you assume that cache hit ratio of the buffer pool will stay the same. Estimating this is much harder. Basically you need to estimate the working set of pages you will have under full load.
If you can ensure that your buffer pool can always take all hot data you can basically live without any reads. Then you only have to scale writes (for example with an SSD drive).
Related
Logical Reads - Reading the data from the data cache
Physical Reads - Reading the data from Disk
Usually, we are using the caches for better performance, it will reduce the time to access the data from the disk.
But in SQL, why reducing the logical reads improve performance?
If data cache affects the performance, then why we need that data cache in middle?
I'm new to SQL tuning, please clarify.
Thanks.
A logical read is a request to read a block. It may be served from cache or it may be served from disk. If it is served from disk, it is also a physical read. All physical reads are also logical reads.
What SQL Server (or any database) calls a physical read may or may not actually require going to spinning disk. The operating system might have a cache, the SAN array might have a cache, etc. A physical read from SQL Server simply means that SQL Server didn't have the block in cache.
Normally, when we're tuning a query, we focus on logical reads because that is a relatively stable value and a query that does less logical I/O will generally be faster than one that does more logical I/O. When we're tuning a query or when we're running it from our application, what fraction of our logical reads are actually physical reads is somewhat up to chance. If you're working on tuning a query, it is very likely that the data you're relying on is quickly going to get cached so your physical I/O will go down if you just keep running the query. When the query runs in production, you might get lucky and find that most of the blocks are in cache and there is very little I/O or you might get unlucky and find that very few of the blocks are in cache. If you focus on physical I/O when you're tuning, you'll be chasing a constantly moving target. And since SQL Server can't differentiate between a physical read that was actually served from the operating system cache or a physical read that was served from the SAN cache or a physical read that was served from a SSD in the SAN or a physical read that was served by going to actual physical spinning disk, you're mixing a bunch of different things with very different performance profiles together. If you run the query 100 times, you'll get roughly the same number of logical I/Os every time. You'll get wildly different numbers of physical I/Os and those physical I/Os are likely to have wildly different performance characteristics because some will be hitting physical disk and some will just be hitting the operating system cache.
As a very rough first approximation for most OLTP systems, the odds that a block you want is in cache is going to be roughly constant (most queries are reading relatively recent rows in most tables that are mostly cached). If your system keeps 95% of the blocks that you're reading in cache, you can reasonably guess that if you have a query that does 1000 logical I/O's per execution, on average it's going to do 50 physical I/O's per execution. Sometimes you'll get lucky and it'll do 0 physical I/Os, sometimes you'll get luck and it'll do 250 physical I/Os but on average you'll get 50. If you reduce the logical I/O, you'll probably reduce the physical I/O by the same fraction.
Of course, this is a very rough approximation. If you've got a poorly performing query that does a table scan of a multi-million row table, the odds that all the 10 year old rows are cached is a lot lower than the odds that the recent blocks that you actually mostly want to read are cached. If you get rid of the full-scan's logical I/O, you'll probably get a much larger percentage-wise reduction in physical I/O because you're focusing the query on the recent blocks that SQL Server is really good at caching.
And, of course, sometimes we're focused on things other than logical I/O. Sometimes our systems are CPU-bound not I/O-bound and we want to focus on how much CPU a query is using. Sometimes our systems are under memory pressure and we want to tune how much memory a query uses. But normally database systems are I/O bound and starting with a focus on logical I/O is normally reasonable when we tune a query.
Databases usually are a storage for most applications. Our company also makes a lot of calculations and data manipulations with that data on daily basis.
As soon as we get more and more data, data generation became an issue cause takes too long. And I think it can make sense to separate database to at least two :
for storing data with focus on read/write performance;
for calculations with focus on data aggregation performance.
Does anybody has similar experience and can tell if this idea is good and what will be design differences for mentioned two points?
Maybe it is worth to look for noSQL solution for calculating data e.g. in-memory databases?
it can make sense to separate database to at least two
If the databases are in different Disks (with different spindles ), it may help otherwise you get no gain because disk IO is shared between these databases.
For best practice,read Storage Top 10 Best Practices
Maybe it is worth to look for noSQL solution for calculating data e.g. in-memory databases?
No need to go to noSQL solution, you can use in-memory tables
In-Memory OLTP can significantly improve the performance of transaction processing, data load and transient data scenarios.
For more details, In-Memory OLTP (In-Memory Optimization)
Other Strategies
1) Tune tempdb
Tempdb is common for all databases and heavily used in calculations.
A more pragmatic approach, is to have a 1:1 mapping between files and logical CPUs(cores) up to eight.
for more details: SQL Server TempDB Usage, Performance, and Tuning Tips
2) Evaluate life expectancy (PLE) Counter and take actions for enhancement
To evaluate data cache, run the following query
SELECT [object_name],
[counter_name],
[cntr_value] FROM sys.dm_os_performance_counters
WHERE [object_name] LIKE '%Manager%'
AND [counter_name] = 'Page life expectancy'
The recommended value of the PLE counter (in seconds ) is greater than:
total_memory_dedicated_for_sql_server / 4 * 300
Page Life Expectancy is the number of seconds a page will stay in the buffer pool without references. In simple words, if your page stays longer in the buffer pool (area of the memory cache) your PLE is higher, leading to higher performance as every time request comes there are chances it may find its data in the cache itself instead of going to the hard drive to read the data.
If PLE is't enough Increase memory and tune indexes and statistics.
3) Use SSD disks
With the cost of solid state disks (SSDs) going down, use the SSDs as a second tier of cache.
4) Use RAID 5 for the databases; and RAID 10 for the transaction logs and tempdb.
In general, the SQL optimizer game is moving data from disk (low speed) to cache (memory- high speed).
Increase memory and enhance diskIo speed, you gain high performance
I have a simple ssis ETL moving data from one server to another, a couple of the tables are 150gb+. For these larger tables how do I optimize the MaximumInsertCommitSize, on the server that is being loaded, I see the the the ram utilization is near 100% (64 gb) for the duration of the load for these tables. I am also getting pageiolatch_ex suspensions on the server that is being loaded.
This leads me to believe that the MaximumInsertCommitSize needs to be brought down.
my first question is whether you agree with this?
my other questions are more open ended
How do I optimize it by trial and error when it takes an hour to load the table (and table size matters for this operation so I would have to load all of it) ?
would network speed ever play into this optimization due to the increased bandwidth with multiple ?partitions? of data
would the hard drive speed affect it as this is the servers cheapest component (facepalm) - my thinking here is page io latch memory waits indicate that the number of disk operation is minimized but since the ram is over utilized sending tasks to the drive instead of waiting would be better.
Finally is there a calculator for this, I feel that even vague approximation of MaximumInsertCommitSize would be great (just based on network, disk, ram, file size, with assumptions that the destination is partitioned and has ample disk space)
I have been monitoring the performance of an OLTP database (approx. 150GB); the average disk sec/read and average disk sec/write values are exceeding 20 ms over a 24hr period.
I need to arrive at a clear explanation as to why the business application has no influence over the 'less-than-stellar' performance on these counters. I also need to exert some pressure to have the storage folk re-examine their configuration as it applies to the placement of the mdf, ldf and tempdb files on their SAN. At present, my argument is shaky but I am pressing my point with people who don't understand the difference between IOPs and disk latency.
Beyond the limitations of physical hardware and the placement of data files across physical disks, is there anything else that would influence these counter values? For instance: the number of transactions per second, the size of the query, poorly written queries or missing indexes? My readings say 'no' but I need a voice of authority in this debate.
There are "a lot" of factors that can affect the overall latency. To truly rule it as SAN or not, you will want to look at the "Avg. Disk sec/Read counter" and the "Avg. Desk sec/Write Counter", that you mentioned. Just make sure you are looking at the "Physical Disk" object, and not the "Logical Disk" object. The logical disk counter includes the file system overhead, and may be different, depending on different factors.
Once you have the counters for the physical disks, you will want to compare them to the latency counters for the Storage unit, the server is connected to. You mentioned "storage folk" So I'm going to assume that is a different team, hopefully they will be nice and provide the info to you.
If it is a Storage unit issue, then both of these counters should match up pretty good. That indicates the storage unit is truly running slow. If the storage unit counters show significantly better, then it's something in between. Depending on what type of storage network you are using this would be the HBA/NIC/Switches that connect the server and storage together. Or if it's a VM then the host machine stats would prove useful as well.
Apart from obvious reasons such as "not enough memory for buffer pool", latency mostly depends on how your storage is actually implemented.
If your server has external SAN, usually its problem is that it might give you stellar throughput, but it will never (again, usually) give you stellar latency. It's just the way things are. It might become a real headache for heavy loaded OLTP systems, sure.
So, if you are about to squeeze every last microsecond from your storage, most probably you will need local drives. That, and your RAID 10 should have enough spindles to cope with the load.
I'd like to know if when I run a query, the database's contents are in my system's RAM. The dataset is approx 4.1 gb, my machine has 8gb of RAM. Am I reading from disk every time I run a SELECT or UPDATE query?
Aside from monitoring IO activity as others have suggested, you can also run a query to take advantage of PostgreSQL's stats tracking.
The following query will show your cache hit rate. If you hitting only cache, the hit rate should be somewhere around the .99 or higher range, if your doing a lot of disk reads, it'll be lower.
SELECT
sum(heap_blks_read) as heap_read,
sum(heap_blks_hit) as heap_hit,
sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as ratio
FROM
pg_statio_user_tables;
This query, and other performance queries can be found here
All systems allows to track a IO activity. So you can use a system monitoring tools - there is a iotop for linux for example.
If that query is the only thing actively running on the system, use the system tools (vmstat, sar) to see if there is a spike in IO when it executes. If there is a lot of other things going on, it can be very hard to figure out what you want, as there is no easy way to distinguish data actually read from disk from data read from the OS's file system cache. You can turn on track_io_timing and see if the resulting times are consistent with the data coming from RAM.