How to return FileStreamResult with SqlDataReader - sql-server

I have an ASP.NET Core project that downloads large files which are stored in SQL Server. It works fine for small files, but large files often time out as they are read into memory before getting downloaded.
So I am working to improve that.
Based on SQL Client streaming support examples I have updated the code to the following:
public async Task<FileStreamResult> DownloadFileAsync(int id)
{
ApplicationUser user = await _userManager.GetUserAsync(HttpContext.User);
var file = await this._attachmentRepository.GetFileAsync(id);
using (SqlConnection connection = new SqlConnection(this.ConnectionString))
{
await connection.OpenAsync();
using (SqlCommand command = new SqlCommand("SELECT [Content] FROM [Attachments] WHERE [AttachmentId] = #id", connection))
{
command.Parameters.AddWithValue("id", file.AttachmentId);
SqlDataReader reader = await command.ExecuteReaderAsync(CommandBehavior.SequentialAccess);
if (await reader.ReadAsync())
{
if (!(await reader.IsDBNullAsync(0)))
{
Stream stream = reader.GetStream(0);
var result = new FileStreamResult(stream, file.ContentType)
{
FileDownloadName = file.FileName
};
return result;
}
}
}
}
return null;
}
But when I test, it throws this exception:
Cannot access a disposed object. Object name: 'SqlSequentialStream'
Is there a way to fix this exception?

Your using statements are all triggering when you do your return, thus disposing your connection and command, but the whole point of this is to leave the stream copy to happen in the background after your function is done.
For this pattern you're going to have to remove the using calls and let garbage collection trigger when the stream copy is done. FileStreamResult should at the very least call Dispose on the stream you give it, which should un-root the command and connection to be later finalized and closed.

This is the working code, which is dramatically faster than without the streaming:
[HttpGet("download")]
public async Task<FileStreamResult> DownloadFileAsync(int id)
{
var connectionString = _configuration.GetConnectionString("DefaultConnection");
ApplicationUser user = await _userManager.GetUserAsync(HttpContext.User);
var fileInfo = await this._attachmentRepository.GetAttachmentInfoByIdAsync(id);
SqlConnection connection = new SqlConnection(connectionString);
await connection.OpenAsync();
SqlCommand command = new SqlCommand("SELECT [Content] FROM [Attachments] WHERE [AttachmentId]=#id", connection);
command.Parameters.AddWithValue("id", fileInfo.Id);
// The reader needs to be executed with the SequentialAccess behavior to enable network streaming
// Otherwise ReadAsync will buffer the entire BLOB into memory which can cause scalability issues or even OutOfMemoryExceptions
SqlDataReader reader = await command.ExecuteReaderAsync(CommandBehavior.SequentialAccess);
if (await reader.ReadAsync())
{
if (!(await reader.IsDBNullAsync(0)))
{
Stream stream = reader.GetStream(0);
var result = new FileStreamResult(stream, fileInfo.ContentType)
{
FileDownloadName = fileInfo.FileName
};
return result;
}
}
return null;
}

Related

Script task in SSIS package is executing but not performing the action

I have the two SSIS packages which basically has two action like below
First it truncates the contents of the table and then it executes the script task like basically call an API and inserts the response in to the table
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public async void Main()
{
try
{
var sqlConn = new System.Data.SqlClient.SqlConnection();
ConnectionManager cm = Dts.Connections["SurplusMouse_ADONET"];
string serviceUrl = Dts.Variables["$Project::RM_ServiceUrl"].Value.ToString();
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
HttpClient client = new HttpClient();
client.BaseAddress = new Uri(serviceUrl);
client.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("application/json"));
string APIUrl = string.Format(serviceUrl + "/gonogo");
var response = await client.GetAsync(APIUrl);
if (response.IsSuccessStatusCode)
{
var result = await response.Content.ReadAsStringAsync();
try
{
sqlConn = (System.Data.SqlClient.SqlConnection)cm.AcquireConnection(Dts.Transaction);
const string query = #"INSERT INTO [dbo].[RM_Approved_Room_State]
(APPROVED_ROOM_STATEID,SOURCE_ROOMID,DEST_ROOMID,ENTITY_TYPEID)
SELECT id, sourceRoomRefId, destinationRoomRefId,entityRefId
FROM OPENJSON(#json)
WITH (
id int,
sourceRoomRefId int,
destinationRoomRefId int,
entityRefId int
) j;";
using (var sqlCmd = new System.Data.SqlClient.SqlCommand(query, sqlConn))
{
sqlCmd.Parameters.Add("#json", SqlDbType.NVarChar, -1).Value = result;
await sqlCmd.ExecuteNonQueryAsync();
}
}
catch (Exception ex)
{
Dts.TaskResult = (int)ScriptResults.Failure;
}
finally
{
if (sqlConn != null)
cm.ReleaseConnection(sqlConn);
}
}
}
catch (Exception ex)
{
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
#region ScriptResults declaration
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
} }
Similar to the above package I have another one which does more likely does insert records into different table the response from a different endpoint. When I execute the packages locally/ execute them separately after deploying it in to the server it works fine. But when I add them in to the SQL Server Agent Job like below and run them on a schedule
The Jobs run successfully and dont show any errors but I can see only one table with data from one package but the other one truncates the records but I dont think the script task is getting executed / I dont see any records inserted. I dont think there are any issues with access because when I run them seperate manually the data are getting inserted, Just when it is running on a schedule it is not working as expected. Any idea what could be happening here.. Any help is greatly appreciated

SQL Server DAC FX SchemaComparison SchemaCompareDatabaseEndpoint fails with no IntegratedSecuirty

I am trying to find newly created tables in the target database on publishing. Using DAC Fx I am able to find the differences and delete the tables after moving the newly created table to another db.
I developed and tested the code with IntegratedSecurity. Started failing on machines with SQLServer logins.
The moment I toggle the IntegratedSecurity to true it works. Is it a bug?
private void Analyse()
{
try
{
var sourceDacpac = new SchemaCompareDacpacEndpoint(DacPacSrc);
var csb = new SqlConnectionStringBuilder(ConnectionString);
csb.IntegratedSecurity = false;
var targetDatabase =new SchemaCompareDatabaseEndpoint(csb.ToString());
var comparison = new SchemaComparison(sourceDacpac, targetDatabase);
comparison.Options.DropObjectsNotInSource = true;
var result = comparison.Compare();
if (result.GetErrors().Any())
{
throw new Exception("Compare failed " + result.GetErrors().FirstOrDefault().Message);
}
var delta = new List<string>();
if (result.Differences != null && result.Differences.Any())
{
var deltaTables = result.Differences.Where(x => x.Name == "Table" && x.UpdateAction == SchemaUpdateAction.Delete);
delta = deltaTables.Select(x => x.TargetObject.Name.ToString()).ToList();
}
FindingDeltaCompleted?.Invoke(this, new DeltaEventArgs(delta));
}
catch (Exception ex)
{
Logging.HandleException(ex);
}
}
Try setting Persist Security Info=True in the SQL Authentication connection string.
SSDT/DAC Fx saves connection strings in registry under HKEY_CURRENT_USER\SOFTWARE\Microsoft\SSDT\ConnectionStrings. When Persist Security Info=True is not set, it won't restore the password when loading the connection strings from registry.

SqlConnection.Open fails randomly

I have been facing a strange issue. I have a code snippet like below:
SqlTransaction transaction = null;
SqlDataReader reader = null;
using (SqlConnection sqlConnection = new SqlConnection(_sqlEndpoint.ConnectionString))
{
sqlConnection.Open(); **// Randomly fails here**
try
{
switch (_sqlEndpoint.ResultType)
{
case ResultSetType.None:
transaction = sqlConnection.BeginTransaction();
using (SqlCommand command = new SqlCommand(_sqlEndpoint.Query, sqlConnection, transaction))
{
command.CommandTimeout = _sqlEndpoint.CommandTimeout;
int rowsAffected = command.ExecuteNonQuery();
exchange.Message.SetHeader("RowsAffected", rowsAffected);
}
break;
case ResultSetType.Single:
using (SqlCommand command = new SqlCommand(_sqlEndpoint.Query, sqlConnection))
{
command.CommandTimeout = _sqlEndpoint.CommandTimeout;
exchange.Message.Body = command.ExecuteScalar();
}
break;
default:
using (SqlCommand command = new SqlCommand(_sqlEndpoint.Query, sqlConnection))
{
command.CommandTimeout = _sqlEndpoint.CommandTimeout;
reader = command.ExecuteReader(CommandBehavior.CloseConnection);
exchange.Message.Body = reader;
}
break;
}
}
catch (SqlException ex)
{
Log.ErrorFormat("[{0}] Error occured while fetching data from sql server: {1}", _sqlEndpoint.RouteName, ex.Message);
}
}
The connection opens 9 out of 10 times and then fails with the following exception:
"The operation is not valid for the state of the transaction"
Again is able to connect the 11th time and then fails randomly the 19th time or so. This doesnt happen in the dev environment, only in production.
I have tried/checked the following:
Changing the connection string to use UserId and Password creds instead of SSPI to eliminate any AD related issues
Tried Clearing the specific connection from the pool by calling SqlConnection.ClearPool(conn) so that a new connection is made rather than getting from the pool
Checked that MSDTC service is running on the production server
I am running out of ideas on this one. I am not even sure how to reproduce this error on the dev machine.
Any help or pointers will be appreciated.
Thanks

Nunit database rollback

I'm fairly new to using Nunit as a test framework and have come across something I don't really understand.
I am writing an integration test which inserts a row into a database table. As I want to repeatedly run this test I wanted to delete the row out once the test has completed. The test runs fine, however when I look in the database table the row in question is still there even though the delete command ran. I can even see it run when profiling the database whilst running the test.
Does Nunit somehow rollback database transaction? If anyone has any ideas why I am seeing this happen please let me know. I have been unable to find any information about Nunit rolling back transactions which makes me think I'm doing something wrong, but the test runs the row appears in the database it is just not being deleted afterwards even though profiler shows the command being run!
Here is the test code I am running
[Test]
[Category("Consultant split integration test")]
public void Should_Save_Splits()
{
var splitList = new List<Split>();
splitList.Add(new Split()
{
UnitGroupId = 69,
ConsultantUserId = 1,
CreatedByUserId = 1,
CreatedOn = DateTime.Now,
UpdatedByUserId = 1,
UpdatedOn = DateTime.Now,
Name = "Consultant1",
Unit = "Unit1",
Percentage = 100,
PlacementId = 47
});
var connection = Helpers.GetCpeDevDatabaseConnection();
var repository = new Placements.ConsultantSplit.DAL.PronetRepository();
var notebookManager = Helpers.GetNotebookManager(connection);
var userManager = Helpers.GetUserManager(connection);
var placementManager = Helpers.GetPlacementManager(connection);
var sut = new Placements.ConsultantSplit.ConsultantSplitManager(repository, connection, notebookManager, userManager, placementManager);
IEnumerable<string> errors;
sut.SaveSplits(splitList, out errors);
try
{
using (connection.Connection)
{
using (
var cmd = new SqlCommand("Delete from ConsultantSplit where placementid=47",
connection.Connection))
{
connection.Open();
cmd.CommandType = CommandType.Text;
connection.UseTransaction = false;
cmd.Transaction = connection.Transaction;
cmd.ExecuteNonQuery();
connection.Close();
}
}
}
catch (Exception exp)
{
throw new Exception(exp.Message);
}

SqlServer Converting XML to varbinary and parsing it in .NET (C#)

Consider the following code:
[Test]
public void StackOverflowQuestionTest()
{
const string connectionString = "enter your connection string if you wanna test this code";
byte[] result = null;
using (var connection = new SqlConnection(connectionString))
{
connection.Open();
using (var sqlCommand = new SqlCommand("declare #xml as xml = '<xml/>' SELECT convert(varbinary(max), #xml) as value"))
//using (var sqlCommand = new SqlCommand("SELECT convert(varbinary(max), N'<xml/>') as value"))
{
sqlCommand.Connection = connection;
using (SqlDataReader reader = sqlCommand.ExecuteReader())
{
while (reader.Read())
{
result = (byte[])reader["value"];
}
reader.Close();
}
}
}
string decodedString = new UnicodeEncoding(false, true).GetString(result);
var document = XElement.Parse(decodedString);
}
If I run this test I get an XmlException with message : "Data at the root level is invalid. Line 1, position 1." As it turns out the problem is "0xFFFE" preamble which is considered as invalid character.
Note that if I use commented string instead, everything works just fine, which is strange as per me. Looks like SqlServer stores XML strings in UCS-2 with a BOM, and at the same time it stores nvarchar values without it.
The main question is: how can I decode this byte array to string which will not contain this preamble (BOM)?
In case anyone will need this in future, the following code works:
using(var ms = new MemoryStream(result))
{
using (var sr = new StreamReader(ms, Encoding.Unicode, true))
{
decodedString = sr.ReadToEnd();
}
}

Resources