Testing EF SQL Server based application with in-memory SQLite? - sql-server

I am using SQL Server with Entity Framework for a development of web app in .NET 4 with VS2010 RC. I would like to prepare testing database with sample data.
Should I prepare a copy of the real database (say another SQL Server database) for testing, or can I use SQLite in memory for better performance?
If using SQLite, can I use the same model EF has created for SQL Server database? How to migrate schema from SQL Server to in-memory SQLite?
How are you testing your code that uses EF with SQL Server?
Thanks for sharing.

I use LINQ to Objects and DI.
So let's say I have a service which uses a repository:
public FooService : Service, IFooService
{
private IFooRepository Repository { get; set; }
public GetSpecialFoos()
{
return from f in Repository.SelectAll()
where f.IsSpecial
select f;
}
public FooService(IFooRepository repository)
{
this.Repository = repository;
}
}
Now I can use constructor injection to inject a mock repository for testing. Generally, you'd use a DI framework for this. But the important thing is the mock repository can use LINQ to Objects:
public MockFooRepository : IFooRepository
{
public IList<Foo> Data { get; set; }
public IQueryable<Foo> SelectAll()
{
return Data.AsQueryable();
}
}
Now I can test:
[TestMethod]
public void GetSpecialFoos_returns_only_special_foos()
{
var specialId = 1;
var notSoSpecialId = 2;
var foos = new List<Foo>
{
new Foo
{
Id = specialId,
IsSpecial = true
},
new Foo
{
Id = notSoSpecialId,
IsSpecial = false
}
}
// use a DI framework here instead, in the real world
var repository = new MockFooRepository
{
Data = foos
};
var service = new FooService(repository);
var actual = service.GetSpecialFoos();
var returned = actual.First();
Assert.AreEqual(true, returned.IsSpecial);
Assert.AreEqual(specialId, returned.Id);
}

Related

Dynamic NamedDatabase in Play Framework

I'm running a java play framework setup where I would like to have several databases depending on what customer is making the call. I have a jwt setup where there is a tenant id. However I can't get my head around what's best practise in Play regarding this. As for now I have this code:
public class JavaNamedDatabase {
private Database db;
private DatabaseExecutionContext executionContext;
private static final Logger.ALogger LOGGER = Logger.of(JavaNamedDatabase.class);
#Inject
public JavaNamedDatabase(
#NamedDatabase("xxx") Database db, DatabaseExecutionContext executionContext) {
this.db = db;
this.executionContext = executionContext;
}
where I would like to make "xxx" dynamic depending on which tenant is making the request.
Is it possible to pass this parameter or do I need to have separate classes?
Or maybe the best solution is just to have one instance running per customer and have the #NamedDatabase as a runtime config parameter?
I found DBApi where there is a getter for Database.
public class JavaNamedDatabase {
private DBApi dbApi;
private DatabaseExecutionContext executionContext;
private static final Logger.ALogger LOGGER = Logger.of(JavaNamedDatabase.class);
#Inject
public JavaNamedDatabase(
DBApi dbApi, DatabaseExecutionContext executionContext) {
this.dbApi = dbApi;
this.executionContext = executionContext;
}
public CompletionStage<Integer> addGenreToPlayItem(Integer playItemId, String genre) {
return CompletableFuture.supplyAsync(
() ->
dbApi.getDatabase("xxx").withConnection(...```

Redis vs SQL Server performance

Application performance is one of the main reason of using cache over relational database. Because it stores data in memory in the form of key value pair, we can store frequently accessed data in cache which are not changes very frequently. Reading from cache is much faster than database. Redis is one of the best solution in distributed cache market.
I was doing a performance test between Azure Redis cache and Azure SQL Server. I have created a simple ASP.NET Core application and inside that I have read data from SQL Server database as well as Redis multiple times and compare the read time duration between them. For database reading I have used Entity Framework Core and for Redis reading I have used 'Microsoft.Extensions.Caching.StackExchangeRedis'.
Model
using System;
namespace WebApplication2.Models
{
[Serializable]
public class Student
{
public int Id { get; set; }
public string Name { get; set; }
public int Age { get; set; }
public string Subject { get; set; }
public Student()
{
Name = string.Empty;
Subject = string.Empty;
}
}
}
Entity Framework Core data context.
using Microsoft.EntityFrameworkCore;
using WebApplication2.Models;
namespace WebApplication2.Data
{
public class StudentContext : DbContext
{
public StudentContext(DbContextOptions<StudentContext> options)
: base(options)
{
}
public DbSet<Student>? Students { get; set; }
}
}
Startup class
public void ConfigureServices(IServiceCollection services)
{
services.AddControllersWithViews();
string studentDbConnectionString = Configuration.GetConnectionString("StudentDbConnectionString");
services.AddDbContext<StudentContext>(option => option.UseSqlServer(studentDbConnectionString));
string redisConnectionString = Configuration.GetConnectionString("RedisConnectionString");
services.AddStackExchangeRedisCache(options =>
{
options.Configuration = redisConnectionString;
});
}
appsettings.json
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
}
},
"AllowedHosts": "*",
"ConnectionStrings": {
"StudentDbConnectionString": "[Azure SQL Server connection string]",
"RedisConnectionString": "[Azure Redis cache connection string]"
}
}
Home controller
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Caching.Distributed;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Runtime.Serialization.Formatters.Binary;
using WebApplication2.Data;
using WebApplication2.Models;
namespace WebApplication2.Controllers
{
public class HomeController : Controller
{
private readonly StudentContext _studentContext;
private readonly IDistributedCache _cache;
public HomeController(StudentContext studentContext, IDistributedCache cache)
{
_studentContext = studentContext;
_cache = cache;
}
public IActionResult Index()
{
List<Student>? students = null;
var counter = 10000;
var sw = Stopwatch.StartNew();
for (var i = 0; i < counter; i++)
{
students = _studentContext.Students.OrderBy(student => student.Id).ToList();
}
sw.Stop();
ViewData["DatabaseDuraion"] = $"Database: {sw.ElapsedMilliseconds}";
if (students != null && students.Count > 0)
{
List<Student> studentsFromCache;
var key = "Students";
_cache.Set(key, ObjectToByteArray(students));
sw.Restart();
for (var i = 0; i < counter; i++)
{
studentsFromCache = (List<Student>)ByteArrayToObject(_cache.Get(key));
}
sw.Stop();
ViewData["RedisDuraion"] = $"Redis: {sw.ElapsedMilliseconds}";
}
return View();
}
private byte[] ObjectToByteArray(object obj)
{
var bf = new BinaryFormatter();
using var ms = new MemoryStream();
bf.Serialize(ms, obj);
return ms.ToArray();
}
private object ByteArrayToObject(byte[] arrBytes)
{
using var memStream = new MemoryStream();
var binForm = new BinaryFormatter();
memStream.Write(arrBytes, 0, arrBytes.Length);
memStream.Seek(0, SeekOrigin.Begin);
object obj = binForm.Deserialize(memStream);
return obj;
}
}
}
Home\Index.cshtml view
#{
ViewData["Title"] = "Home Page";
}
<div class="text-center">
<p>#ViewData["DatabaseDuraion"]</p>
<p>#ViewData["RedisDuraion"]</p>
</div>
I have found SQL Server is faster than Redis.
The ASP.NET Core application is hosted in Azure App Service with the same location with Azure SQL Server and Azure Redis.
Please let me know why Redis is slower than SQL Server?
I have used github.com/dotnet/BenchmarkDotNet to benchmark the Azure SQL Server database and Azure cache for Redis for 10000 reads. SQL Server database mean: 16.48 sec and Redis mean: 29.53 sec.
I have used JMeter and connects 100 users each reading SQL Server database/Redis 1000 times. There is not much difference between total time it took to finish reading SQL Server database vs Redis (both are near about 3 mins and 30 sec), but I saw load on Azure SQL Server database DTU. The DTU goes near 100% during the test.
As a conclusion, I think speed is not the only reason to use Redis cache over SQL Server database but another reason is Redis cache reduces good amount of load from the database.
You don't only see performance difference here BTW. For cache, Redis is also giving you cache invalidation logic, which you need to build up in SQL In memory table. So Redis all the way when it comes to cache
Think about what's happening here
In SQL
Process -> TCP -> read optimised store (single table) -> Serialisation into application models
In Redis
Process -> check for cache hit -> TCP -> read optimised store (single table) -> Serialisation into application models
Redis is great, but don't mistake its purpose, if you are doing a read from an indexed table on a well optimised index then SQL is going to be quick, why would Redis be any quicker? The power of distributed cache comes in when your authoritive store or your process have to do some computations to gain to result, so effectively what you are saving by caching is CPU disk / time (be it on sql or in proc).
If you want to really increase speed it's in memory cache that you want, this however isn't as simple as it first sounds, the real trick here is a way to invalidate in memory cache across a distributed cluster upon a change to the authoritive store.
Hope this helps

net core 1 (dnx 4.5.1) with enterpriselibrary 6 - setting up the connection string

i ve big problems running enterprise library data access block with net core 1 (dnx 4.5.1)
How can i setup the default connection string for entlib
my appsettings.json
"ConnectionString": "Server=localhost\sqlexpress;Initial Catalog=blind;User Id=blind;Password=blind"
Here is my problem (no default connectionstring)
Database db = DatabaseFactory.CreateDatabase();
how can i pass the appsettings ConnectionString to the entlib databasefactory
any help would be greatly appreciated
I know it's an old question, but I have a similar setup (but using .NET Core 2.0) and it took me awhile to figure out how to set the default database connection without using the web.config to manage it.
What I did was include the default database and all of the connection strings in the appsettings.json and then in my Startup class I read the appsettings.json into an object that I defined to store the default db name and the connection strings and configure the default + named database using DatabaseFactory.SetDatabase.
DatabaseFactory.SetDatabases() Definition
public class DataConfiguration
{
public string DefaultDatabase { get; set; }
public List<ConnectionStringSettings> ConnectionStrings { get; set; }
}
public class Startup
{
public Startup(IConfiguration configuration)
{
//Get the Database Connections from appsettings.json
DataConfig = configuration.Get<DataConfiguration>();
var defaultDb = DataConfig.ConnectionStrings?.Find(c => c.Name == DataConfig.DefaultDatabase);
DatabaseFactory.SetDatabases(() => new SqlDatabase(defaultDb.ConnectionString), GetDatabase);
Configuration = configuration;
}
public Database GetDatabase(string name)
{
var dbInfo = DataConfig.ConnectionStrings.Find(c => c.Name == name);
if (dbInfo.ProviderName == "System.Data.SqlClient")
{
return new SqlDatabase(dbInfo.ConnectionString);
}
return new MySqlDatabase(dbInfo.ConnectionString);
}
}
Whenever there is documentation, I always suggest reading it as it is usually good. This is one of those examples, check out the "Getting Started with ASP.NET 5 and Entity Framework 6". There are several things that you need to do to ensure that you are correctly configured.
Setup your connection string and DI.
public class ApplicationDbContext : DbContext
{
public ApplicationDbContext(string nameOrConnectionString)
: base(nameOrConnectionString)
{
}
}
Also, notice the path in the configuration, it seems to differ from yours.
public void ConfigureServices(IServiceCollection services)
{
services.AddScoped((_) =>
new ApplicationDbContext(
Configuration["Data:DefaultConnection:ConnectionString"]));
// Configure remaining services
}

Parse JSON data within SQL Server Integration Services Package?

I'm trying to set up an SSIS job that will pull a JSON-encoded mailing list from MailChimp, compare it to a list of customers in our CRM database (SQL Server), and upload via JSON any new customers not already there. I can't seem to find anything on serializing/deserializing JSON within SSIS, other than writing a script task, and it seems that I can't import the .Net serialization libraries into a script. Any suggestions? Thanks in advance!
Couple things to address here:
First, your problem with adding new libraries in the scripting component. I assume you're using VS 2008 to do your SSIS development and want to use the .net 3.5 library to do this. You go to project, add reference and you don't see any of the dll's you need. This may be in part that you're using windows 7 and the compact 3.5 framework. .net 3.5.1 comes with Windows 7, you just have to enable it. Go to control panel, programs and features. In that screen you will see Turn Windows features on or off, click on that. In that window check Microsoft .NET Framework 3.5.1, this way take a few minutes to run. Once it finishes look for a directory similar to these C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework.NETFramework\v3.5\Profile\Client and C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5. Between these 2 directories, you will find any dll you will need for serialization/deserializtion of JSON. These can be added to your project by going to Project-->Add Reference-->Browse Tab, then navigate to the v3.5 directory and select the dlls you need(System.Web.Extensions.dll(v3.5.30729.5446)is used in this example).
To get JSON from a web service, deserialize it, and send the data to your CRM database, you will have to use a script component as a source in your data flow and add columns to your output buffer that will be used to hold the data coming from the JSON feed(on the Input and Output screen). In the code, you will need to override the CreateNewOutputRows method. Here is an example of how to do this:
Say Your JSON looked like this...[{"CN":"ALL","IN":"Test1","CO":0,"CA":0,"AB":0},{"CN":"ALL","IN":"Test2","CO":1,"CA":1,"AB":0}]
I would fist define a class to mirror this JSON feed attributes (and the columns you defined on the inputs and outputs screen) that will eventually hold these values once you deserialize...as such:
class WorkGroupMetric
{
public string CN { get; set; }
public string IN { get; set; }
public int CO { get; set; }
public int CA { get; set; }
public int AB { get; set; }
}
Now you need to call your web service and get the JSON feed using an HttpWebRequest and a Stream:
string wUrl = "YOUR WEB SERVICE URI";
string jsonString;
HttpWebRequest httpWReq = (HttpWebRequest)WebRequest.Create(wUrl);
HttpWebResponse httpWResp = (HttpWebResponse)httpWReq.GetResponse();
Stream responseStream = httpWResp.GetResponseStream();
using (StreamReader reader = new StreamReader(responseStream))
{
jsonString = reader.ReadToEnd();
reader.Close();
}
Now we deserialize our json into an array of WorkGroupMetric
JavaScriptSerializer sr = new JavaScriptSerializer();
WorkGroupMetric[] jsonResponse = sr.Deserialize<WorkGroupMetric[]>(jsonString);
After deserializing, we can now output the rows to the output buffer:
foreach (var metric in jsonResponse)
{
Output0Buffer.AddRow();
Output0Buffer.CN = metric.CN;
Output0Buffer.IN = metric.IN;
Output0Buffer.CO = metric.CO;
Output0Buffer.CA = metric.CA;
Output0Buffer.AB = metric.AB;
}
Here is what all the code put together would look like(I have a step by step example here):
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Net;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
using System.Web.Script.Serialization;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void CreateNewOutputRows()
{
string wUrl = "YOUR WEB SERVICE URI";
try
{
WorkGroupMetric[] outPutMetrics = getWebServiceResult(wUrl);
foreach (var metric in outPutMetrics)
{
Output0Buffer.AddRow();
Output0Buffer.CN = metric.CN;
Output0Buffer.IN = metric.IN;
Output0Buffer.CO = metric.CO;
Output0Buffer.CA = metric.CA;
Output0Buffer.AB = metric.AB;
}
}
catch (Exception e)
{
failComponent(e.ToString());
}
}
private WorkGroupMetric[] getWebServiceResult(string wUrl)
{
HttpWebRequest httpWReq = (HttpWebRequest)WebRequest.Create(wUrl);
HttpWebResponse httpWResp = (HttpWebResponse)httpWReq.GetResponse();
WorkGroupMetric[] jsonResponse = null;
try
{
if (httpWResp.StatusCode == HttpStatusCode.OK)
{
Stream responseStream = httpWResp.GetResponseStream();
string jsonString;
using (StreamReader reader = new StreamReader(responseStream))
{
jsonString = reader.ReadToEnd();
reader.Close();
}
JavaScriptSerializer sr = new JavaScriptSerializer();
jsonResponse = sr.Deserialize<WorkGroupMetric[]>(jsonString);
}
else
{
failComponent(httpWResp.StatusCode.ToString());
}
}
catch (Exception e)
{
failComponent(e.ToString());
}
return jsonResponse;
}
private void failComponent(string errorMsg)
{
bool fail = false;
IDTSComponentMetaData100 compMetadata = this.ComponentMetaData;
compMetadata.FireError(1, "Error Getting Data From Webservice!", errorMsg, "", 0, out fail);
}
}
class WorkGroupMetric
{
public string CN { get; set; }
public string IN { get; set; }
public int CO { get; set; }
public int CA { get; set; }
public int AB { get; set; }
}
This can now be used as an input for a data destination (your CRM database). Once there you can use SQL to compare the data and find mismatches, send the data to another script component to serialize, and send any updates you need back to the web service.
OR
You can do everything in the script component and not output data to the output buffer. In this situation you would still need to deserialze the JSON, but put the data into some sort of collection. Then use the entity framework and LINQ to query your database and the collection. Determine what doesn't match, serialize it, and send that to the web service in the same script component.

Integration tests with Entity Framework 4 Code First using SQL Server CE 4 or SQLite

I would like to start with integration testing. I am using an ASP.NET MVC 3 app. And I am using Entity Framework 4 Code First CTP5. My integration tests to the database is in a separate project something like MyProject.Data.IntegrationTests.
I am planning on using SQL Server CE 4 or SQLite. Any recommendations/tips/opinions on using any one of these for what I am trying to accomplish?
Does anyone know of any decent articles that I can read on what I am trying to accomplish? And help/feedback would be appreciated.
I am using SQL Server 2008 for my database. But when testing my repositories I would like to test them against one of these database mentioned above, so I will need to specify the connection string.
UPDATE
I work from a service layer (called from my controller) and then the service layer will call my repository. For examples, below is how I would add a news item:
Service class:
public class NewsService : INewsService
{
private INewsRepository newsRepository;
public NewsService(INewsRepository newsRepository)
{
this.newsRepository = newsRepository;
}
public News Insert(News news)
{
// Insert news item
News newNews = newsRepository.Insert(news);
// Insert audit entry
// Return the inserted news item's unique identifier
return newNews;
}
}
Repository class:
public class NewsRepository : INewsRepository
{
MyContext context = new MyContext();
public NewsRepository()
{
}
public News Insert(News news)
{
int newsId = context.Database.SqlQuery<int>("News_Insert #Title, #Body, #Active",
new SqlParameter("Title", news.Title),
new SqlParameter("Body", news.Body),
new SqlParameter("Active", news.Active)
).FirstOrDefault();
news.NewsId = newsId;
// Return the inserted news item
return news;
}
}
I am using Entity Framework 4 Code First CTP5 and NUnit. Does NUnit has something similar to the roll back in XUnit?
If you use a testing framework like XUnit (http://xunit.codeplex.com/), it comes with a feature called [AutoRollback] and that will rollback the transaction ran in the test so none of your data will change!
As far as how to setup the tests, I need to see more of how you setup your data access. Did you use the Repository Pattern? (Entity Framework 4 CTP 4 / CTP 5 Generic Repository Pattern and Unit Testable). If I could see some of your code it would help. Below is a sample integration test with XUnit:
private readonly IUserRepository _repository;
public UserRepositoryTests()
{
_repository = new UserRepository(base._databaseFactory);
}
[Fact, AutoRollback]
public void Should_be_able_to_add_user()
{
var user = new User{Name = "MockName"};
_repository.Add(user);
base._unitOfWork.Commit();
Assert.True(user.Id > 0);
}
So the above test adds a user to my database, then checks its Id property to check that SQL Server auto generated an Id for it. Because the method is decorated with the AutoRollback attribute, the data is then removed from my database after the method ends!

Resources