Large Data streaming from sql server through WCF to client - sql-server

Question

What are possible ways of storing large data file( CSV filesaround 1 GB) using SQL database and streaming that data from Database using WCF to the client(without fetching the complete data in memory )?

Answer 1

I think there are a few issues to take into account here:
The size of the data you actually want to return
The structure of that data (or lack thereof)
The place to store that data behind your NLB
Returning that data to the consumer.
From your question, it sounds like you want to store 1 GB of structured (CSV) data and stream it to the client. If you really are generating and then serving a 1GB file (and don't have much metadata around it), I'd go for using a FTP/SFTP server (or perhaps a Network file share, which can certainly be secured in a variety of ways).
If you need to store metadata about the file that goes beyond its file name/create time/location, then SQL might be a good option, assuming you could do one of the following:
store the CSV data in tabular format in the database
Use FILESTREAM and store the file itself
Here is a decent primer on FILESTREAM from SimpleTalk. You could then use the SqlFileStream to help stream the data from the file itself (and SQL Server will help maintain transactional consistency for you, which you may or may not want), an example of which is present in the documentation. The relevant section is here:
private static void ReadFilestream(SqlConnectionStringBuilder connStringBuilder)
{
using (SqlConnection connection = new SqlConnection(connStringBuilder.ToString()))
{
connection.Open();
SqlCommand command = new SqlCommand("SELECT TOP(1) Photo.PathName(), GET_FILESTREAM_TRANSACTION_CONTEXT() FROM employees", connection);
SqlTransaction tran = connection.BeginTransaction(IsolationLevel.ReadCommitted);
command.Transaction = tran;
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
// Get the pointer for the file
string path = reader.GetString(0);
byte[] transactionContext = reader.GetSqlBytes(1).Buffer;
// Create the SqlFileStream
using (Stream fileStream = new SqlFileStream(path, transactionContext, FileAccess.Read, FileOptions.SequentialScan, allocationSize: 0))
{
// Read the contents as bytes and write them to the console
for (long index = 0; index < fileStream.Length; index++)
{
Console.WriteLine(fileStream.ReadByte());
}
}
}
}
tran.Commit();
}
}
Alternatively, if you do choose to store it in tabular format you can use typical SqlDataReader methods, or perhaps some combination of bcp and .NET helpers.
You should be able to combine that last link with Microsoft's remarks on streaming large data over WCF to get the desired result.