I find that Sybase is only able to send 2 or 3 UDPs per second using sp_sendmsg (or the function it calls, syb_sendmsg.)
If I run a little perl script on the same host, I can send 1000s per second, as I might expect.
So Sybase is doing something funny when sending udps.
Has anyone tried this?
Anyone know what limits this?
It may be a mis-match between the Sybase ASE setting for default packet size, and the pysical packet size for the OS.
From Jeff Garbus' Administrators Guide to Sybase 15:
When you are sending large amounts of data across the network frequently, you can get a significant performance benefit by matching up the ASE default network packet size ... with the physical network packet size.
The excerpt here talks about tuning the network settings.
Related
I am currently trying to find a way to read large geometry objects from my PostgreSQL/PostGIS database more efficiently. I've analyzed the plan for my query and the run time if I only pull back the primary key is very acceptable, but when I retrieve the - sometimes very large - geometry objects, the return time can go into the minutes for a single query.
I am wondering if there is any more efficient way to read large objects from the database besides just a typical query (maybe some kind of streaming, where I can process the input as I'm retrieving it, to cut down on the effective processing time?). I've looked into cursors, but I'm not sure that's really what I'm looking for since those seem to be mostly related to PL/pgsql scripts as opposed to a Java application.
I am running PostgreSQL 9.5, application is written in Java/Scala and uses JDBC. Any help is appreciated.
EDIT - I should also add that the database is local to the machine running the application.
You should check at what stage is Postgres spending the most time:
executing the query
retriving data from hardware
sending data over the network
If you retrieve a lot of data then a lot of time can be spend when sending data over the network.
I never tried this but maybe you can use Postgresql SSLCompresion option:
If set to 1 (default), data sent over SSL connections will be compressed (this requires OpenSSL version 0.9.8 or later). If set to 0, compression will be disabled (this requires OpenSSL 1.0.0 or later). This parameter is ignored if a connection without SSL is made, or if the version of OpenSSL used does not support it.
Compression uses CPU time, but can improve throughput if the network is the bottleneck. Disabling compression can improve response time and throughput if CPU performance is the limiting factor.
[https://www.postgresql.org/docs/9.2/static/libpq-connect.html]
The application used by a group of 100+ users was made with VB6 and RDO. A replacement is coming, but the old one is still maintained. Users moved to a different building across the street and problems began. My opinion regarding the problem has been bandwidth, but I've had to argue with others who say it's database. Users regularly experience network slowness using the application, but also workstation tasks in general. The application moves large audio files and indexes them on occasion as well as others. Occasionally the database becomes hung. We have many top end, robust SQL Servers, so it is not a server problem. What I figured out is, a transaction is begun on a connection, but fails to complete properly because of a communication error. Updates from other connections become blocked, they continue stacking up, and users are down half a day. What I've begun doing the moment I'm told of a problem, after verifying the database is hung, is set the database to single user then back to multiuser to clear connections. They must all restart their applications. Today I found out there is a bandwidth limit at their new location which they regularly max out. I think in the old location there was a big pipe servicing many people, but now they are on a small pipe servicing a small number of people, which is also less tolerant of momentary high bandwidth demands.
What I want to know is exactly what happens to packets, both coming and going, when a bandwidth limit is reached. Also I want to know what happens in SQL Server communication. Do some packets get dropped? Do they start arriving more out of sequence? Do timing problems occur?
I plan to start controlling such things as file moves through the application. But I also want to know what configurations are usually present on network nodes regarding transient high demand.
This is a very broad question. Networking is very key (especially in Availability Groups or any sort of mirroring set up) to good performance. When transactions complete on the SQL server, they are then placed in the output buffer. The app then needs to 'pick up' that data, clear it's output buffer and continue on. I think (without knowing your configuration) that your apps aren't able to complete the round trip because the network pipe is inundated with requests, so the apps can't get what they need to successfully finish and close out. This causes havoc as the network can't keep up with what the apps and SQL server are trying to do. Then you have a 200 car pileup on a 1 lane highway.
Hindsight being what it is, there should have been extensive testing on the network capacity before everyone moved across the street. Clearly, that didn't happen so you are kind of left to do what you can with what you have. If the company can't get a stable networking connection, the situation may be out of your control. If you're the DBA, I highly recommend you speak to your higher ups and explain to them the consequences of the reduced network capacity. Often times, showing the consequences of inaction can lead to action.
Out of curiosity, is there any way you can analyze what waits are happening when the pileup happens? I'm thinking it will be something along the lines of ASYNC_NETWORK_IO which is usually indicative that SQL is waiting on the app to come back and pick up it's data.
I have CAIDA internet trace dataset and it contain more than 200000 unique IPv4 addresses and almost 1 million flows.
Im currently using mininet to emulate my SDN project and i wish that i could use this dataset in my simulation.
What i plan is tou use the Tcpreplay to replay the dataset in my mininet. The question is,
1. Do i have to manually configure more than 200000 unique ipv4 hosts in order to mimic the real network as in the dataset?
2. Or there are another way
Appreciate whoever has this experience of using tcpreplay together with real internet dataset could share the knowledge.
thanks
You can use the use tcpreplay with the --unique-ip option, which will create unique IP addresses for each transmission of the pcap file. This option is very fast, and allows you to run at a controlled rate, all the way up to 10GigE full wire speed.
tcpreplay -i eth7 -tK --loop 50 --unique-ip bigFlows.pcap
I suggest that you read my article How to do a Performance Test for an IP Flow Appliance. Also suggest using my bigFlows.pcap file which contains a large capture of actual traffic on a corporate WAN.
I want to replicate data from a boat offshore to an onshore site. The connection is sometimes via a satellite link and can be slow and have a high latency.
Latency in our application is important, the people on-shore should have the data as soon as possible.
There is one table being replicated, consisting of an id, datetime and some binary data that may vary in length, usually < 50 bytes.
An application off-shore pushes data (hardware measurements) into the table constantly and we want these data on-shore as fast as possible.
Are there any tricks in MS SQL Server 2008 that can help to decrease the bandwith usage and decrease the latency? Initial testing uses a bandwidth of 100 kB/s.
Our alternative is to roll our own data transfer and initial prototyping here uses a bandwidth of 10 kB/s (while transferring the same data in the same timespan). This is without any reliability and integrity checks so this number is artificially low.
You can try out different replication profiles or create your own. Different profiles are optimized for different network/bandwidth scenarios.
MSDN talks about replication profiles here.
Have you considered getting a WAN accelerator appliance? I'm too new here to post a link, but there are several available.
Essentially, the appliance on the sending end compresses the outgoing data, and the receiving end decompresses it, all on the fly and completely invisibly. This has the benefit of increasing the apparent speed of the traffic and not requiring you to change your server configurations. It should be entirely transparent.
I'd suggest on the fly compression/decompression outside of SQL Server. That is, SQL replicates the data normally but something in the network stack compresses so it's much smaller and bandwidth efficient.
I don't know of anything but I'm sure these exist.
Don't mess around with the SQL files directly. That's madness if not impossible.
Do you expect it to always be only one table that is replicated? Are there many updates, or just inserts? The replication is implemented by calling an insert/update sproc on the destination for each changed row. One cheap optimization is to force the sproc name to be small. By default it is composed from the table name, but IIRC you can force a different sproc name for the article. Given an insert of around 58 bytes for a row, saving 5 or 10 characters in the sproc name is significant.
I would guess that if you update the binary field it is typically a whole replacement? If that is incorrect and you might change a small portion, you could roll your own diff patching mechanism. Perhaps a second table that contains a time series of byte changes to the originals. Sounds like a pain, but could have huge savings of bandwidth changes depending on your workload.
Are the inserts generally done in logical batches? If so, you could store a batch of inserts as one customized blob in a replicated table, and have a secondary process that unpacks them into the final table you want to work with. This would reduce the overhead of these small rows flowing through replication.
We are looking at stress testing a NAS system for a Database, basically was want to see how much abuse it can take and how much it affects the Database performance. here is what we have planed
I have a test Tool I'm building that will kickoff a configurable number of
threads that run a sql query (also configurable, and thinking about
having it able to run multiple quires)
using the SQLIOSim utility to simulate SQL Server activity
Copying very large amounts of Data onto and off of the device (at the same time)
Can anyone think of anything else we could do (that's repeatable) for placing a load on the system.
You'll also want to simulate network conditions between your database and the NAS. As more traffic hits a network, its realizable utilization drops, and this will seriously affect your performance.
By way of example. If you have 50 machines on a 1Gbps network and the network is approaching 100% utilization, packet collisions and retries at the data link layer mean that your effective total transfer is a fraction of the network potential that would be realized if you had only two communicators on the net. Worse yet, as retries increase, so does effective load. You get an ugly feedback loop in the face of peak demand.
There are a number of network traffic simulators and generators out there, though I'm afraid I've never used any of them.
You could look at Pole Position, a generic database performance testing suite.
http://www.polepos.org/
Depending on the goals you wish to achieve with your load testing you may also want to look at using SQLIO & not just SQLIOSim. SQLIOSim is very good for stress testing & simulating SQL Server load & will go from green to red if it detects any IO errors. It's output is a bit cryptic though although KKline gives some insight.
SQLIO is useful if you want to perform one operation continuously such as large random reads, or large sequential writes & anything in between. It will also give you some useful output stats which you can graph & use as a comparison.
You can try IoMeter from Intel