In Microsoft SQL server, I can retrieve the number of bytes sent / received with provider statistics on the connection, as described here: https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/provider-statistics-for-sql-server
We are switching to PostgreSQL and more specifically, we are using Npgsql to access the database.
I cannot figure out how to retrieve the number of bytes sent over the network to answer our query.
I looked into some Npgsql code, but my understanding of the codebase is lacking and I only see performance counters for .NET Standard there. And these would not return the exact number of bytes that were needed to read the rows for a query. Also, we are still using .NET 4.x and cannot upgrade for several reasons.
Is there a way to get the number of bytes transferred over the network for the result of a query?
No, unfortunately there's nothing like that Npgsql at the moment.
Related
We are developping an application in C# that uses ODBC and the "Adaptive server enterprise" driver to extract data from a Sybase DB.
We have a long SQL batch query that create a lot of intermediate temporary tables and returns several DataTable objects to the application. We are seeing exceptions saying TABLENAME not found where TABLENAME is one of our intermediate temporary tables. When I check the status of the OdbcConnection object in the debugger it is Closed.
My question is very general. Is this the price you pay for having long-running complicated queries? Or is there a reliable way to get rid of such spurious disconnects?
Many thanks in advance!
There's a couple of ODBC timeout parameters - see SDK docs at:
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc20116.1550/html/aseodbc/CHDCGBEH.htm
Specifically CommandTimeOut and ConnectionTimeOut which you can set accordingly.
But much more likely is you're being blocked or similar when the process is running - maybe ask your DBA to check your query plan for the various steps in your batch and look for specific problem areas such as tablescans etc which could be masking your timeout issue.
First of all, here is our setup:
SQL Server 2014 SP2 Standard Edition 64-bit
128GB of RAM (maximum allowed by Standard Edition), of which 120GB is allocated to SQL Server
The server currently hosts ~5000 databases which are all similar (same tables, stored proc, etc.) for a total of 690GB of data (mdf files only)
Now what happens:
Every now and then, after the server has been up for some time, we receive this error when executing queries on some databases:
.NET Framework execution was aborted by escalation policy because of out of memory
This error happens more often when we perform an update of all client databases (when we launch a feature) using Red Gate SQL Multi Script. Of the ~5000 DBS, we have the error on 70 of them. Running the update script again, the error happens on a portion, and so on until we have all databases updated correctly. This is just annoying.
We have this error since a long time. Our server had 64GB of RAM, so we just added more memory to max out SQL Server Standard Editor, but still, the error came back a few days later. We think the error might be a symptom of something else.
A few things that might help to get an answer:
Our version of SQL Server is 64-bit, so we think we don't have to deal with Virtual Address Space Reservation
The error also happens when running from a client app written in PHP/Linux, so we're not talking about .NET framework of the client code
In our databases, the only usage of .NET framework we make is GROUP_CONCAT, a.NET CLR Assembly with User Defined Functions which help us simulate MySQL GROUP_CONCAT aggregate. We have a copy of the assembly in each of our 5000 client databases.
We already tried to lower the max server memory setting (to 96GB), but we were still getting those errors
If more info is needed I will update my question.
It's been 4 months since I tried a fix, and I did not experienced the bug again. Still, I don't have the exact explanation for the bug, but here is what I tried and it seems to work:
My guess was that having the same .NET CLR Assembly in each of our 5000+ databases might be the problem and increased memory usage for .NET in some way.
I created a new database named something like DotNetClrUtils
I copied the .NET CLR Assembly for GROUP_CONCAT in this database
I changed all usage of GROUP_CONCAT in all our client code and stored procedures to reference the single instance in DotNetClrUtils database (calling it like this: DotNetClrUtils.dbo.GROUP_CONCAT_D(col, ','))
That's all, and now this problem is gone!
We're in the middle of doing a new data warehouse roll-out using SQL Server 2014. One of my data sources is Oracle, and unfortunately the recommended Attunity component for quick data access is not available for SSIS 2014 just yet.
I've tried avoiding using OLEDB, as that requires installation of specific Oracle client tools that have caused me a lot of frustration before, and with the Attunity stuff supposedly being in the works (MS promised they'd arrive in August already), I'm reluctant to go through the ordeal again.
Therefore, I'm using ADO.NET. All things considered, performance is acceptable for the time being, with the exception of 1 particular table.
This particular table in Oracle has a bunch of varchar columns, and I've come to the conclusion that it's because of the width of the selected row that this table performs particularly slow. To prove that, rather than selecting all columns as they exist in Oracle (which is the original package I created), I truncated all widths to the maximum length of the values actually stored (CAST(column AS varchar(46)). This reduced the time to run the same package to 17 minutes (still way below what I'd call acceptable, and it's not something I'd put in production because it'll open up a world of future pain, but it proves the width of the columns are definitely a factor).
I increased the network packet size in SQL Server, but that did not seem to help much. I have not managed to figure out a good way to alter the packet size on the ADO.NET connector for Oracle (SQL Server does have that option). I attempted to see if adding Packet size=32000;to the connection string for the Oracle connector, but that just threw an error, indicating it simply won't be accepted. The same applies to FetchSize.
Eventually, I came up with a compromise where I split the load into three different parts, dividing the varchar columns between these parts, and using two MERGE JOIN objects to well, merge the data back into a single combined dataset. Running that and doing some extrapolation leads me to think that method would have taken roughly 30 minutes to complete (but without the potential of data loss using the CAST solution from above). However, that's still not acceptable.
I'm currently in the process of trying some other options (not using MERGE JOIN but dumping into three different tables, and then merging those on the SQL Server itself, and splitting the package up into even more different loads in an attempt to further speed up the individual parts of the load), but surely there must be something easier.
Does anyone have experience with how to load data from Oracle through ADO.NET, where wide rows would cause delays? If so, are there any particular guidelines I should be aware of, or any additional tricks you might have come across that could help me reduce load time while the Attunity component is unavailable?
Thanks!
The updated Attunity drivers have just been released by Microsoft:
Hi all, I am pleased to inform you that the Oracle and TeraData connector V3.0 for SQL14
SSIS is now available for download!!!!!
Microsoft SSIS Connectors by Attunity Version 3.0 is a minor release.
It supports SQL Server 2014 Integration Services and includes bug
fixes and support for updated Oracle and Teradata product releases.
For details, please look at the download page.
http://www.microsoft.com/en-us/download/details.aspx?id=44582
Source: https://connect.microsoft.com/SQLServer/feedbackdetail/view/917247/when-will-attunity-ssis-connector-support-sql-server-2014
We are testing JDBC drivers from jTDS and Microsoft, and we are suffering from unwanted pauses in query execution. Our application opens many ResultSets and fetches only a few rows from each. Each query selects about 100k rows, but we fetch only about 50 (which is enough to fill a page). The problem is that every query after the first contains a pause of about 2s, on which the driver loads all rows from the previous ResultSet to a temporary storage (memory or disk), so they can be traversed later. Because we have about 6 queries in worst scenarios, there will be a pause of about 10s, which makes the application unresponsive to the user. MSSQL version is 2005.
To remove such pauses, we've tried to enable MARS (Multiple Active Result Sets) via connection string parameters of Microsoft JDBC driver (due to lack of documentation, we tried everything that is listed on https://sites.google.com/site/sqlconnect/sql2005strings). Example of connection string:
jdbc:sqlserver://TESTDBMACHINE;instanceName=S2005;databaseName=SampleDB;MarsConn=yes
But none of them solves the problem. Microsoft JDBC driver seems to accept anything at connection string - if you replace MarsConn=yes by PleaseBeFast=yes, MS driver ignores the parameter and doesn't even log the fact. I don't know if MARS is a client-only feature that just caches rows from a previously active result set, or if it's a server feature. I don't even know how to detect, from the server side, if a given connection is using MARS. If you can comment on this, it will be welcome.
Another solution for the pause was to use scrollable (bidirectional) result sets. This removes the pause, but makes fetch time 80% slower and more network consuming. We are now considering to implement a JDBC connection wrapper that keeps a pool of actual connections and automatically issue queries to distinct "ResultSet free" connections. But this is somewhat cumbersome because we need to keep a link between each connection and its active ResultSet. Also it would consume more connections from the server and may cause troubles to DBAs. And this solution doesn't help if there is an active transaction, on which all queries must be issued on the same connection.
Do you know some parameter, configuration, specific API, link or trick that can remove the pause from the second and subsequent query executions?
fix your SQL queries! why only use the first 50 or so from 100k rows?? use TOP 100 or something like that! There is no reason that the application should be filtering 100k rows, that is the job of the database.
Far more important that your client woes is what happens on the server side. Since you issue queries and then you stop reading the result the server will have to suspend the query in the middle of the execution because the network buffers are full and has no room to write the result into. A query suspended in the middle of the execution is consuming a lot of resources: memory, locks and, most importantly, a worker thread (there are very few of these lying around).
Issue queries for only the data you need, consume all the data, free the connection and, more importantly, the server resources. If your queries are to complex, go back to the drawing board and redesign your data model to properly answer, efficiently, the queries your requesting from it. Right now you are totally barking up the wrong tree, you're simply asking how to make a bad situation worse.
We've created an ODBC data source using the SQL Server driver (Native Client 10 - sqlncli10.dll). This ODBC data source was configured to enable MARS (key HKEY_CURRENT_USER\Software\ODBC\ODBC.INI\ datasource, value of MARS_Connection must be Yes). Then we used Sun's JDBC-ODBC bridge, and voilá! The pauses were gone, and surprisingly, fetch time became faster than JTDS and MSSQL pure Java JDBC drivers!
According to http://msdn.microsoft.com/en-us/library/ms345109(SQL.90).aspx, MARS is a server-aided feature. The pure Java drivers (from JTDS and MSSQL) doesn't seem to support served-based MARS (at least we couldn't enable it after many configuration changes). Because most of our user base that uses MSSQL Server runs on Windows (no surprise), we are about to make the switch from JTDS to JDBC-ODBC. Both Native Client ODBC driver and JDBC-ODBC bridge seems to be mature, full featured and up-to-date solutions, so I guess there should be no problems. If you know some, please comment!
Linux based users will still use JTDS. Since now we know that MARS is a server-aided feature, we'll fill a feature request for JTDS and Microsoft to support MARS in their pure Java JDBC drivers.
I'm working on a project where a PHP dialog system is communicating with a Microsoft SQL Server 2008 and I need more speed on the PHP side.
After profiling my PHP scripts, I discovered that a call to mssql_connect() needs about 200 milliseconds on that particular system. For some simple dialogs this is about 60% of the whole script runtime. So I could gain a huge performance boost by speeding up this call.
I already assured that only one single connection handle is produced for every request to my PHP scripts.
Is there a way to speed up the initial connection with SQL Server? Some restrictions apply, though:
I can't use PDO (there's a lot of legacy code here that won't work with it)
I don't have access to the SQL Server configuration, so I need a PHP-side solution
I can't upgrade to PHP 5.3.X, again because of crappy legacy code.
Hm. I don't know much about MS SQL, but optimizing that single call may be tough.
One thing that comes to mind is trying mssql_pconnect(), of course:
First, when connecting, the function would first try to find a (persistent) link that's already open with the same host, username and password. If one is found, an identifier for it will be returned instead of opening a new connection.
But you probably already have thought of that.
The second thing, you are not saying whether MS SQL is running on the same machine as the PHP part, but if it isn't, maybe there's a basic networking issue at hand? How fast is a classic ping between one host and the other? The same would go for a virtual machine that is not perfectly configured. 200 milliseconds really sounds very, very slow.
Then, in the User Contributed Notes to mssql_connect(), there is talk about a native PHP driver for MS SQL. I don't know anything about it, whether it will pertain the "old" syntax and whether it is usable in your situation, but it might be worth a look.
The User Contributed Notes are always worth a look, there are tidbits like this one:
Just in case it helps people here... We were being run ragged by extremely slow connections from IIS6 --> SQL Server 2000. Switching from CGI to ISAPI fixed it somewhat, but the initial connection still took along the lines of 10 seconds, and eventually the connections wouldn't work any more.
The solution was to add the database server IP address to the HOST file on the server, pointing it to the internal machine name. Looks like some kind of DNS lookup was the culprit.
Now connections and queries are flying, and the world is once again right.
We have been going through quite a bit of optimization between php 5.3, FreeTDS and mssql lately. Assuming that you have adequate server resources, we are finding that two changes made the database interaction much faster and much more reliable.
Using mssql_pconnect() instead of mssql_connect() eliminated an
intermittent "cannot connect to server" issue. I read a lot of posts
that indicated negative issues associated with persistent
connections but so far we haven't seen anything to suggest that it's
an issue. The php server seems to keep between 20 and 60 persistent
connections open to the db server depending upon load.
Using the IP address of the database server in the freetds.conf file
instead of the hostname also lent a speed increase.
The only thing i could think of is to use an ip adress instead of an hostname for the sql connect to spare the dns lookup. Maybe persistant connections are an option and a little bit faster.