DataReader and large datasets - sql-server

Question

I have a query that returns between 10k and 20k rows. I'm dumping this data into a IEnumerable<T> in what as far as I know is the fastest possible way:
using (var rdr = cmd.ExecuteReader()) {
var trackIdCol = rdr.GetOrdinal("TrackId");
var dateTimeCol = rdr.GetOrdinal("ActDateTime");
var clicksCol = rdr.GetOrdinal("Clicks");
var ipCol = rdr.GetOrdinal("IPAddress");
while (rdr.Read()) {
yield return new SiteClick() {
TrackId = (int)rdr[trackIdCol],
DateTime = (DateTime)rdr[dateTimeCol],
Clicks = (int)rdr[clicksCol],
IPAddress = rdr[ipCol] as string
};
}
}
The query takes about 11s to return all the results in SSMS and SSDT, but the code above takes over 2 minutes. There has to be something I'm doing wrong here. SqlDataAdapter.Fill() also takes about 2 minutes to run, if that helps.
It is worth noting that our database is horribly unoptimized. Just the fact that it takes 11 seconds to get results from that query in SSMS is ridiculous, but I gotta work with what I got. If the query executes quickly in SSMS, but an empty while(rdr.Read()){} still takes 2 minutes, it is possible that the DB is still the issue?

Answer 1

You can't dump into IEnumerable, because it is interface that must have implementation behind the scene. In your sample yield used as such implementation. But yield is rather expensive. For every iteration its state is saved, then restored and this is slow for database objects.
Database reader is the fastest iterator itself. So if you can use it directly - this will
be the fastest way. But there is a drawback, because such database cursors might lock
db tables for access from other threads. So use it in single-user environments or
when you can read uncommitted data.
To reduce locking time you can dump data into in-memory IEnumerable implementations.
For example, List. Just set larger capacity to avoid frequent allocations, so it
will be as fast as underlying array. The drawback of such solution is memory usage. But
in you case this will be less than 1M.
List<SiteClick> list = new List<SiteClick>(20000);
Also you can improve performance and memory usage if define SiteClick as struct, not class.
In this case list will contain objects, not references.
A little bit additional performance you can get if use typified reader methods.
TrackId = rdr.GetInt32(trackIdCol),
DateTime = rdr.GetDateTime(dateTimeCol),
Clicks = rdr.GetInt32(clicksCol),
IPAddress = rdr.GetString(ipCol)
UPDATE: For SQL Server it's a frequent case, when queries are tested in SSMS. But there are some issues, because SMSS has different options compared to database defaults. For example, ARITHABORT. So you can set it manually for connection to test.
// Use the same connection as for data reading
var cmdOptions = connection.CreateCommand();
cmdOptions.CommandType = CommandType.Text;
cmdOptions.CommandText = "SET ARITHABORT ON";
cmdOptions.ExecuteNonQuery();
According to MS recommendations - http://msdn.microsoft.com/en-us/library/aa933126%28SQL.80%29.aspx, its better to ensure that you have these options set:
SET ANSI_NULLS ON
SET ANSI_PADDING ON
SET ANSI_WARNINGS ON
SET CONCAT_NULL_YIELDS_NULL ON
SET NUMERIC_ROUNDABORT ON
SET QUOTED_IDENTIFIER ON
SET NUMERIC_ROUNDABORT OFF