I am trying to investigate the IO performance overhead of docker so I created a mysql docker container on a specific machine and I ran the sysbench mysql benchmark to measure IO performance. Sysbench basically executes some read/write transactions over a period of time and then outputs the number of completed transactions and the transactions/second rate.
When I run the benchmark on the native machine, I get a 779.5 transactions per second. When I run the benchmark in a mysql container, I get 336 transactions/second. Almost half the number of transactions per second. Is this a normal performance overhead of docker? This is a huge disadvantage for running the database in a container in production systems, especially for IO/database intensive applications
Are you using a Docker volume for the database files? By default, file writes inside the container will be made to the copy-on-write filesystem, which will be inefficient for database files. Using a volume will mean you can write directly to the host filesystem.
Related
I would like to know the effect of Network Performance (also the exact name for the metric used by AWS RDS instance types) on a DB.
I am loading Graph database with data at scale using parallel processing (multiple Pods in Kubernetes).
I noticed that by simply changing from one RDS instance type to one more powerful, and monitoring the DB metrics in AWS console, performance is doubled.
Performance figures that are doubled are:
VolumeWriteIOPs - doubled
Network Throughput - doubled
VolumeReadIOPs - tripled
As the better instance type has more CPU, RAM, Disk, and possibly network performance (i believe there is an 'invisible' network performance tiering that is not shown in the instance specs), I suppose my question really is -- if there is a (hypothetical) instance with same CPU, same RAM, same disk performance, what difference does network performance alone make to a DB?
Do DBs and RDS DBs process everything slower if the network performance is lower?
Or does it respond at the same speed, but only serve less connections (making the others wait)?
In my use case they are Kubernetes Pods which are writing to the DB, so does it serve each Pod more slowly, or is it non-responsive above a certain point?
Having an issue with DB2OLEDB performance, using sql server 2017 performing a data load from IBM i 7.3 .
The client is a VMware VM, network settings seem ok and have been tweaked up to the best of my ability (vmxnet3 driver 1.8). Load from other VMs or from www flies at over 100mbits.
Troubleshooting so far:
DB2OLEDB (Microsoft) performs substantially faster (3-5x) than IBMDASQL.
Setting I/O Affinity mask to one core doubles performance, but additional cores have no impact.
RSS is on.
DB2OLEDB inprocess on/off has no effect on throughput but off introduces substantial spool up time at the beginning of each query.
Performance currently around 15 mbit. Same table from another SQL server (cached) loads about 3x faster at 50mbit+ (different provider obviously).
Interestingly, enabling rowcache spikes network throughput at the beginning to 100-150mbits. I.e. I'm inferring that there is plenty of network bandwidth available.
Finally, we are using in-memory table as destination in order to eliminate disk i/o as a culprit.
Cpu is burning up one core and the remaining are at ~20% ish.
Any thoughts?
I suspect that DB2OLEDB driver or some part of COM is the bottleneck at this point.
edit: #MandyShaw (too long for comment) Windows Side. IBM i never breaks 1% for my particular workload and it generaly runs 25%-50% load depending on TOD. SQL statements are varied. Everything from straight four part query to 7 table snowflake as a passthrough. One interesting thing: throughput (network) varies based on row length. Wider tables appear to pump at roughly the same row rate as thinner tables. This is true for both the IBM and Microsoft driver. Reducing network latency had a great impact on performance (see RSC issues with Vmxnet3 driver 1.6.6.0). If I understand correctly the OLEDB drivers fetch one row at a time (except possibly when loading the rowset cache).
In other words, for every row we're issuing a request from SQL server to COM/OLEDB driver to Supervisor Network Driver, to Hypervisor Network driver, to physical NIC, through fiber and landing at the IBM i. Then back again. We have successfully been able to multiplex large table loads using the service broker (but this is impractical for most applications). This, as well as other metrics, suggests that the IBM i has plenty of cpu and bandwidth to spare. The fiber fabric is mostly idle, we've tuned the bejeezus out of the hypervisor (VMware) and the supervisor (tcp/ip stack) as well as the SQL server itself.
This is why I'm looking at the COM/OLEDB provider for answers. Something in this model seems to stink. It's either not configured properly or simply doesn't support multiple threads of execution.
I'm also willing to accept that it's something in SQL server, but for the life of me I can't find a way to make a linked server query run multi-threaded using any combination of configuration, options or hints. Again, it may just not be possible by design.
At this point in time, the few known leads that I have involve (1) tuning network interrupt request coalescing and frequency to minimize interrupts to the OLEDB driver thread and (2) throwing the HIS gateway on a consumer x86 box with a high single core frequency (5ghz) in order to maximize single threaded performance.
These are both shitty options.
If you've got something particular in mind with the EBCIDIC/ASCII conversion performance, I'd be happy to try it and report back. Shoot me a link/info.
We are running Dynamics GP 2010 on 2 load balanced citrix servers. For the past 3 weeks we have had severe performance hits when users are running Fixed Assets reporting.
The database is large in size, but when I run the reports locally on the SQL server, they run great. The SQL server seems to be performing adequately even when users are seeing slow performance.
Any ideas?
Just because your DB seems un-stressed, it does not mean that it is fine. It could contain other bottlenecks. Typically, if a DB server is not maxing-out its CPUs occasionally, it means there is a much bigger problem.
Standard process for troubleshooting performance problems on a data driven app go like this:
Tune DB indexes. The Tuning Wizard in SSMS is a great starting point. If you haven't tried this yet, it is a great starting point.
Check resource utilization: CPU, RAM. If your CPU is maxed-out, then consider adding/upgrading CPU or optimize code or split your tiers. If your RAM is maxed-out, then consider adding RAM or split your tiers.
Check HDD usage: if your queue length goes above 1 very often (more than once per 10 seconds), upgrade disk bandwidth or scale-out your disk (RAID, multiple MDF/LDFs, DB partitioning).
Check network bandwidth
Check for problems on your app (Dynamics) server
Shared report dictionaries are the bane of reporting in GP. they do tend to slow things down. also, modifying reports becomes impossible as somebody has it open all the time.
use local report dictionaries and have a system to keep them synced with a "master" reports.dic
We have a new local server that processes data using sql server 2008 and ssis. in this server i have dedicated drive to do different things. the c drive is for the os and software. the d drive is for db storage , and the ssis . The e drive is a ssd drive that we restore each database that is being used by the ssis.
our idea was that we process allot of data and since the ssd drive is only 500gb( because of the cost) we would have everything on a regular drive and transfer the databases in use to the ssd drive to have the process run faster.
when i run the ssis without the ssd drive it takes about 8 hrs and when i run the process restoring the databases on the ssd drive the process takes about the same amount of time to process( in fact if i include the restoring of the data bases the process takes longer)
As of right now i cannot move the os and software to the ssd drive to test to see if that would help the process.
is there a way to utilze the ssd drive to process the data and to speed up the process.
If you want to speed up a given process, you need to find the bottleneck.
Generally speaking (since you give no details of the SSIS-Process) at each point in the operation, one of the systems components (CPU, RAM, I/O, Network) is operating at maximum speed. You need to find the component that contributes the most to your run time and then either speed that component up by replacing it with a faster component or reduce the load on it by redesigning the process.
Since you ruled out already the I/O to the user database(s), you need to look elsewhere. For a general ad hoc look, use the systems resource monitor (available through the task manager). For a deeper look, there are lots of performance counters available via perfmon.exe, both for OS (CPU, I/O), SSIS and SQL Server.
If you have reason to believe that DB-I/O is your bottleneck, try moving the tempdb to the SSD (if you generally have lots of load on the tempdb that is a good idea anyway). Instructions here.
Unless you give us some more details about the SSIS process in question, that's all I can say for now.
Are there any ways to determine what the differences in databases are that affect a SSIS package load performance ?
I've got a package which loads and does various bits of processing on ~100k records on my laptop database in about 5 minutes
Try the same package and same data on the test server, which is a reasonable box in both CPU and memory, and it's still running ... about 1 hour so far :-(
Checked the package with a small set of data, and it ran through Ok
I've had similar problems over the past few weeks, and here are several things you could consider, listed in decreasing order of importance according to what made the biggest difference for us:
Don't assume anything about the server.
We found that our production server's RAID was miscconfigured (HP sold us disks with firmware mismatches) and the disk write speed was literally a 50th of what it should be. So check out the server metrics with Perfmon.
Check that enough RAM is allocated to SQL Server. Inserts of large datasets often require use of RAM and TempDB for building indices, etc. Ensure that SQL has enough RAM that it doesn't need to swap out to Pagefile.sys.
As per the holy grail of SSIS, avoid manipulating large datasets using T-SQL statements. All T-SQL statements cause changed data to write out to the transaction log even if you use Simple Recovery Model. The only difference between Simple and Full recovery models is that Simple automatically truncates the log file after each transactions. This means that large datasets, when manipulated with T-SQL, thrash the log file, killing performance.
For large datasets, do data sorts at the source if possible. The SSIS Sort component chokes on reasonably large datasets, and the only viable alternative (nSort by Ordinal, Inc.) costs $900 for a non-transferrable per CPU license. So... if you absolutely have to a large dataset then consider loading it into a staging database as an intermediate step.
Use the SQL Server Destination if you know your package is going to run on the destination server, since it offers roughly 15% performance increase over OLE DB because it shares memory with SQL Server.
Increase the network packaet size to 32767 on your database connection managers. This allows large volumes of data to move faster from the source server/s, and can noticably improve reads on large datasets.
If using Lookup transforms, experiment with cache sizes - between using a Cache connection or Full Cache mode for smaller lookup datasets, and Partial / No Cache for larger datasets. This can free up much needed RAM.
If combining multiple large datasets, use either RAW files or a staging database to hold your transformed datasets, then combine and insert all of a table's data in a single data flow operation, and lock the destination table. Using staging tables or RAW files can also help relive table locking contention.
Last but not least, experiment with the DefaultBufferSize and DefaulBufferMaxRows properties. You'll need to monitor your package's "Buffers Spooled" performance counter using Perfmon.exe, and adjust the buffer sizes upwards until you see buffers being spooled (paged to disk), then back off a little.
Point 8 is especially important on very large datasets, since you can only achieve a minimally logged bulk insert operation if:
The destination table is empty, and
The table is locked for the duration of the load operation.
The database is in Simply / Bulk Logged recovery mode.
This means that subesquent bulk loads a table will always be fully logged, so you want to get as much data as possible into the table on the first data load.
Finally, if you can partition you destination table and then load the data into each partition in parallel, you can achieve up to 2.5 times faster load times, though this isn't usually a feasible option out in the wild.
If you've ruled out network latency, your most likely culprit (with real quantities of data) is your pipeline organisation. Specifically, what transformations you're doing along the pipeline.
Data transformations come in four flavours:
streaming (entirely in-process/in-memory)
non-blocking (but still using I/O, e.g. lookup, oledb commands)
semi-blocking (blocks a pipeline partially, but not entirely, e.g. merge join)
blocking (blocks a pipeline until it's entirely received, e.g. sort, aggregate)
If you've a few blocking transforms, that will significantly mash your performance on large datasets. Even semi-blocking, on unbalanced inputs, will block for long periods of time.
In my experience the biggest performance factor in SSIS is Network Latency. A package running locally on the server itself runs much faster than anything else on the network. Beyond that I can't think of any reasons why the speed would be drastically different. Running SQL Profiler for a few minutes may yield some clues there.
CozyRoc over at MSDN forums pointed me in the right direction ...
- used the SSMS / Management / Activity Monitor and spotted lots of TRANSACTION entries
- got me thinking, read up on the Ole Db connector and unchecked the Table Lock
- WHAM ... data loads fine :-)
Still don't understand why it works fine on my laptop d/b, and stalls on the test server ?
- I was the only person using the test d/b, so it's not as if there should have been any contention for the tables ??