I am aware that Microsoft has not specified any plan to support Always Encrypted from within Core, yet.
However, this is supported by classical .NET Framework (4.5 onward).
Our Core 2.1 project's usage of Always Encrypted is limited to two simple queries and we are willing to take alternative routes to use it other than downgrading from Core 2.1.
There are many remote ways to solve these problems but they involve remoting (WCF or REST for another classical .NET) or through a Service Fabric Actor (we were using this before).
Is there any clean in-process way to do it? Connecting through ODBC, for example, for these two queries or something like that?
N.B. We are running on Windows.
According Microsoft Docs latest ODBC drivers supports Always Encrypted. The following code works.
using System;
using System.Data.Odbc;
namespace ConsoleApp2
{
class Program
{
static void Main(string[] args)
{
using (var connection = new OdbcConnection(
"Driver={ODBC Driver 17 for SQL Server};DSN=SQL2016;Server={SQL2016};Trusted_Connection=yes;ColumnEncryption=Enabled;Database=AndreyTest;"))
{
using (var cmd = new OdbcCommand("select SSN from TestTable", connection))
{
connection.Open();
Console.WriteLine("Connected");
Console.WriteLine("SSN: " + Convert.ToString(cmd.ExecuteScalar()));
Console.ReadLine();
connection.Close();
}
}
}
}
}
If I remove "ColumnEncryption=Enabled" from the connection string, the output becomes "SSN: System.Byte[]", i.e. the SSN isn't decrypted.
I hope this will help!
I'm enjoying the new Build tool in Visual Studio Online. Allows me to do almost everything that I do my local build server. But one thing that I'm missing is integration database tests: for every build run I re-create test database from scripts and run DB-tests against it.
In Visual Studio Online I can't seem to find any database instance available for my needs.
I have tried creating Azure SQL database (via PowerShell) for every build run and then delete it after the build is complete. But it takes forever (comparing to the rest of the build process) to create a database. And even when PowerShell scripts are done, database is not yet ready to accept requests - I need to constantly check if it is actually ready. So this scenario becomes too complex and not reliable.
Are there other options to do database (SQL Server) integration tests in Visual Studio Online?
Update: I think I'm not very clear of what I need - I need a free (very cheap) SQL Server instance to connect to that runs on build agent in VSO. Something like SQL Express or SQL CE or LocalDB, where I can connect to and re-create database to run C# tests against. Re-creating database or running tests is not a problem, having a valid connection string is a problem.
Update Oct 2016: I've blogged about how I do integration testing in VSTS
The TFS build servers come with MSSQL Server 2012 and MSSQL Server 2014 LocalDBs preinstalled.
Source: TFS Service - Software on the hosted build server
So, just put the following one-liner into your solution's post-build event to create a MYTESTDB LocalDB instance for your needs. This will allow you to connect to (LocalDB)\MYTESTDB an run the database integration tests just fine.
"C:\Program Files\Microsoft SQL Server\120\Tools\Binn\SqlLocalDB.exe" create "MYTESTDB" 12.0 -s
Source: SqlLocalDB Utility
In Azure DevOps, with .net Core and EF Core, I use a different technique.
I use a SQLite in memory database to execute both Integration and End to End tests.
Currently in .net Core you can use both InMemory database and SQLite with in memory option, to run any integration test in the default Azure DevOps CI Agent.
InMemory: https://learn.microsoft.com/en-us/ef/core/miscellaneous/testing/in-memory
Note that the InMemory database is not a relational database, it is a multi-purpose one, and just to mention one limitation:
InMemory will allow you to save data that would violate referential
integrity constraints in a relational database
SQLite in memory mode https://learn.microsoft.com/en-us/ef/core/miscellaneous/testing/sqlite
This approach offers a more realistic platform to test against.
Now, I went a bit further, I didn't want to just be able to run integration tests with database dependency in Azure DevOps, I wanted to also be able to host my WebAPIs in the CI Agent, and to share the database between the API DBcontext and my Persister object (Persister object is a helper class that allow me to automatically generate any kind of entity and save them to the database).
A quick note on Integration Tests and Ent to End tests:
Integration Tests
An example of integration test involving a database, could be a test of the Data Access Layer. In this case, normally, you would create a DBContext when starting a test, fill the target database with some data, use the component under test to manipulate the data, and again use the DBContext to make sure the assertions are satisfied.
This scenario is quite straight forward, in the same code you can share the same DBContext to generated the data and inject it to the component.
End to End Tests
Imagine you have like in my case a RESTful .net Core WebAPI you want to test, making sure all your CRUD operations are working as expected, and you want to test that filtering, pagination and so on are also correct.
In this case, it much more complex share the same DBContext between test (data setup and/or verification) and the WebAPI stack.
Before .net EF Core and WebHostBuilder
So far, the only way I knew was possible, was to have a dedicated server, VM or docker image, responsible serve the API, which had to be also accessible from the web or Azure DevOps.
Setup my integration tests to either re-create the DB, or be clever/limited enough to ignore completely the existing data, and make sure that each test was resilient to data corruption and fully reliable (no false negative or positive results).
Then I had to configure my build definition to run the tests.
Leveraging SQLite in memory with cache=shared and WebHostBuilder
Below I first describe the two majour technologies I use, then I add some code to show how to do it.
SQLite file::memory:?cache=shared
SQLite allow you to work in memory, instead of using a traditional file, this already gives us a huge performance boost, removing the I/O bottleneck, but on top of this, using the option cache=shared, we can use multiple connections within the same process to access the same data. If you need more than one database you can specify a name.
More info: https://www.sqlite.org/inmemorydb.html
WebHostBuilder
.net Core offers Host builders, WebHostBuilder allow us to create a server that startup and host our WebAPI, so that can be reached like if they were hosted on a real server.
When you use the WebHostBuilder in a test class, this two, are living within the same process.
More info: https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.hosting.webhostbuilder?view=aspnetcore-2.2
The Solution
When initialising an E2E test, create a new client to connect the api, create a dbcontext that you will use to seed the database and maybe assert.
Test initialisation:
[TestClass]
public class CategoryControllerTests
{
private TestServerApiClient _client;
private Persister<Category> _categoryPersister;
private Builder<Category> _categoryBuilder;
private IHouseKeeperContext _context;
protected IDbContextTransaction Transaction;
[TestInitialize]
public void TestInitialize()
{
_context = ContextProvider.GetContext();
_client = new TestServerApiClient();
ContextProvider.ResetDatabase();
_categoryPersister = new Persister<Category>(_context);
_categoryBuilder = new Builder<Category>();
}
[TestCleanup]
public void Cleanup()
{
_client?.Dispose();
_context?.Dispose();
_categoryPersister?.Dispose();
ContextProvider.Dispose();
}
[...]
}
TestServerApiClient class:
public class TestServerApiClient : System.IDisposable
{
private readonly HttpClient _client;
private readonly TestServer _server;
public TestServerApiClient()
{
var webHostBuilder = new WebHostBuilder();
webHostBuilder.UseEnvironment("Test");
webHostBuilder.UseStartup<Startup>();
_server = new TestServer(webHostBuilder);
_client = _server.CreateClient();
}
public void Dispose()
{
_server?.Dispose();
_client?.Dispose();
}
}
ContextProvider class is used to generate the DBContext which can be used to seed data or perform database queries for assertions.
public static class ContextProvider
{
private static bool _requiresDbDeletion;
private static IConfiguration _applicationConfiguration;
public static IConfiguration ApplicationConfiguration
{
get
{
if (_applicationConfiguration != null) return _applicationConfiguration;
_applicationConfiguration = new ConfigurationBuilder()
.AddJsonFile("Config/appsettings.json", optional: false, reloadOnChange: true)
.AddEnvironmentVariables()
.Build();
return _applicationConfiguration;
}
}
private static ServiceProvider _serviceProvider;
public static ServiceProvider ServiceProvider
{
get
{
if (_serviceProvider != null) return _serviceProvider;
var serviceCollection = new ServiceCollection();
serviceCollection.AddSingleton<IConfiguration>(ApplicationConfiguration);
var databaseType = ApplicationConfiguration?.GetValue<DatabaseType>("DatabaseType") ?? DatabaseType.SQLServer;
_requiresDbDeletion = databaseType == DatabaseType.SQLServer;
IocConfig.RegisterContext(serviceCollection, null);
_serviceProvider = serviceCollection.BuildServiceProvider();
return _serviceProvider;
}
set
{
_serviceProvider = value;
}
}
/// <summary>
/// Generate the db context
/// </summary>
/// <returns>DB Context</returns>
public static IHouseKeeperContext GetContext()
{
return ServiceProvider.GetService<IHouseKeeperContext>();
}
public static void Dispose()
{
ServiceProvider?.Dispose();
ServiceProvider = null;
}
public static void ResetDatabase()
{
if (_requiresDbDeletion)
{
GetContext()?.Database?.EnsureDeleted();
GetContext()?.Database?.EnsureCreated();
}
}
}
IocConfig class is an helper class I use in my framework to setup the dependency injection. The menthod used above, RegisterContext, is responsile to register the DBContext and set it up as desired, and because this is the same class used by the WebAPI, uses the configuration DatabaseType to determine what to do.
Inside this class probably you can find most of the "complexity".
When using SQLite in memory, you have to remember that:
The connection is not opened and closed automatically like when using SQL Server (that's why i used: context.Database.OpenConnection();)
If no connection is active, the database is deleted (that's why I used services.AddSingleton<IHouseKeeperContext>(s ... it is important that one connection is left open so that the database is not destroyed, but on the other hand you have to be careful to close all connections when a test ends, so that the database is eventually destroyed and the next test will correctly create a new empty one.
The rest of the class handles the SQL Server configuration for both Production and Testing setup. I can at any time setup the tests to use a real instance of SQL Server, all tests will keep being fully independend from the others but it will definitely be slow, and maybe suitable only for a nightly build (if needed, and it depends on the size of your system).
public class IocConfig
{
public static void RegisterContext(IServiceCollection services, IHostingEnvironment hostingEnvironment)
{
var serviceProvider = services.BuildServiceProvider();
var configuration = serviceProvider.GetService<IConfiguration>();
var connectionString = configuration.GetConnectionString(Constants.ConfigConnectionStringName);
var databaseType = DatabaseType.SQLServer;
try
{
databaseType = configuration?.GetValue<DatabaseType>("DatabaseType") ?? DatabaseType.SQLServer;
}catch
{
MyLoggerFactory.CreateLogger<IocConfig>()?.LogWarning("Missing or invalid configuration: DatabaseType");
databaseType = DatabaseType.SQLServer;
}
if(hostingEnvironment != null && hostingEnvironment.IsProduction())
{
if(databaseType == DatabaseType.SQLiteInMemory)
{
throw new ConfigurationErrorsException($"Cannot use database type {databaseType} for production environment");
}
}
switch (databaseType)
{
case DatabaseType.SQLiteInMemory:
// Use SQLite in memory database for testing
services.AddDbContext<HouseKeeperContext>(options =>
{
options.UseSqlite($"DataSource='file::memory:?cache=shared'");
});
// Use singleton context when using SQLite in memory if the connection is closed the database is going to be destroyed
// so must use a singleton context, open the connection and manually close it when disposing the context
services.AddSingleton<IHouseKeeperContext>(s => {
var context = s.GetService<HouseKeeperContext>();
context.Database.OpenConnection();
context.Database.EnsureCreated();
return context;
});
break;
case DatabaseType.SQLServer:
default:
// Use SQL Server testing configuration
if (hostingEnvironment == null || hostingEnvironment.IsTesting())
{
services.AddDbContext<HouseKeeperContext>(options =>
{
options.UseSqlServer(connectionString);
});
services.AddSingleton<IHouseKeeperContext>(s => {
var context = s.GetService<HouseKeeperContext>();
context.Database.EnsureCreated();
return context;
});
break;
}
// Use SQL Server production configuration
services.AddDbContextPool<HouseKeeperContext>(options =>
{
// Production setup using SQL Server
options.UseSqlServer(connectionString);
options.UseLoggerFactory(MyLoggerFactory);
}, poolSize: 5);
services.AddTransient<IHouseKeeperContext>(service =>
services.BuildServiceProvider()
.GetService<HouseKeeperContext>());
break;
}
}
[...]
}
Sample Test, where first I use the persister to generated data which is seeded in the database, then I use the API to get data, the test can be also reversed, using a POST request to set data and then using the DBContext to read the db and make sure the creation was successful.
[TestMethod]
public async Task GET_support_orderBy_Id()
{
_categoryPersister.Persist(3, (c, i) =>
{
c.Active = 1 % 2 == 0;
c.Name = $"Name_{i}";
c.Description = $"Desc_i";
});
var response = await _client.GetAsync("/api/category?&orderby=Id");
var categories = response.To<List<Category>>();
Assert.That.All(categories).HaveCount(3);
Assert.IsTrue(categories[0].Id < categories[1].Id &&
categories[1].Id < categories[2].Id);
response = await _client.GetAsync("/api/category?$orderby=Id desc");
categories = response.To<List<Category>>();
Assert.That.All(categories).HaveCount(3);
Assert.IsTrue(categories[0].Id > categories[1].Id &&
categories[1].Id > categories[2].Id);
}
Conclusions
I love the fact that I can run E2E tests in Azure DevOps for free, performances are incredibly good and this gives me a lot of confidence, ideal when you want to setup a continuous delivery environment.
Here is a screenshot of part of the build execution of this code in Azure DevOps (free version).
Sorry this ended up being longer than expected.
There is a "Redgate SQL CI" extension for VSTS in the marketplace you may want to try. See this link for details:
Within the extension, there are four actions available:
•Build – builds your database into a NuGet package from the database
scripts folder in source control
•Test – runs your tSQLt tests against the database
•Sync – synchronizes the package to an integration database
•Publish – publishes the package to a NuGet stream
You should push the integration tests (anything that needs an instance of your application) to be run in an environment as part of your release pipeline.
In your build just do compile and unit tests. If that competes you should trigger a Release and as part of your release pipeline your first step should be to deploy your database to an azure server.
Instead of trying to use SQL Azure you can create a VM in azure that already exists that has SQL server installed. Use remote scripting to deploy the database and execute your tests.
Even if you are not using the release tools to release this would work for you.
I want to run my ASP.NET MVC application on the local IIS 8 server but I can't get access to SQL Server 2014. Both IIS and SQL Server run on the same host and I am using Windows 8. This is what I have done so far:
In the application I created a model called Employee:
public class Employee
{
public int Id { get; set; }
public string Name { get; set; }
}
I want to connect to the database with using Entity Framework so I created another class called EmployeeDbContext:
public class EmployeeDbContext : DbContext
{
public DbSet<Employee> Employees { get; set; }
}
I have only one Home controller:
public string Index()
{
EmployeeDbContext employeeDbContext = new EmployeeDbContext();
return "Hello world";
}
I am using SQL Server 2014 Express engine with Windows Authentication and the name of my computer is ZSÓTÉ. Here is the connection string from the web.config:
<add name="EmployeeDbContext"
connectionString="Server=ZSÓTÉ\SQLEXPRESS;Initial Catalog=Test;Integrated Security=SSPI"
providerName="System.Data.SqlClient" />
In the Global.asax I didn't set any initializer strategy so by default if the Test database doesn't exist, then the application should create it automatically.
After that I set on the properties window to use IIS Local Server and I create a virtual directory for the application. So far everything is ok, I can see the application in the IIS manager and I set to run under the DefaultAppPool. After that I set all the permissions of the project folder for the IIS APPPOOL\DefaultAppPool object. The settings are fine I can access the application from the IIS server perfectly.
Finally I created a login for IIS APPPOOL\DefaultAppPool in SQL Server Management Studio and I set these two roles: public and dbcreator. But even after all these setups the application doesn't work correctly via IIS.
Though I get the "Hello world" message on the browser, but the Test database is never created. The strangest thing is if I make some malicious change in the connection string, I don't even get any compilation error, just the "Hello world" message in the browser.
What am I doing wrong? Thank you in advance!
The database is created the first time you access to entities.
Try to do this:
public string Index()
{
EmployeeDbContext employeeDbContext = new EmployeeDbContext();
employeeDbContext.Employees.ToList();
return "Hello world";
}
At this point you will receive an error if db creation does not work.
Next steps if something still does not work:
In the connection string use .\SQLEXPRESS because could happen everything to accents.
Assign to the user in SQL Server system administration role. If it work you then start to reduce rights (or, better, change approach).
I ran ELMAH sql scripts in test DB(It created ELmah_Error table and 3 stored procedures) and configured ELMAH in MVC application using Nuget.
I modified web.config as specified and I'm able to log exceptions into
http://mysite/elmah.axd
But, instead i want to log the exceptions into Sql Server.
I added below class to achieve that
public class ElmahHandleErrorAttribute : System.Web.Mvc.HandleErrorAttribute
{
public override void OnException(System.Web.Mvc.ExceptionContext context)
{
LogException(e);
}
private static void LogException(Exception e)
{
// Call to Database and insert the exception info
}
}
Final step was to:
public static void RegisterGlobalFilters(GlobalFilterCollection filters)
{
filters.Add(new ElmahHandleErrorAttribute ());
}
Is it the correct way to use ELMAH to log all exceptions or AM I missing something?
Once you have the database setup, all you need to do is add the following to the <elmah> section your web.config to setup the Elmah to log to the SQL Database:
<elmah>
<errorLog type="Elmah.SqlErrorLog, Elmah" connectionStringName="<DBConnString>"
applicationName="<YourApp>"
</elmah>
Replace <DBConnString> and <YourApp> with appropriate values for your configuration.
Once you have done this you will not need to use your custom ElmahHandleErrorAttribute class.
I am not sure which NuGet package you installed, but I would recommend using the Elmah.MVC package as it integrates Elmah into MVC exceptionally well by setting up all of the ErrorHandlers and ErrorFilters for you.
We are trying to run a Silverlight 4.0 with RIA Services SP1 on an older server without SP1. We copied all of the DLL to a local BIN folder, Copy Local is set to True AND Specific Version is set to True yet we are still getting a "complex type" error below.
WebHost failed to process a request.
Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/7339810
Exception: System.ServiceModel.ServiceActivationException: The service '/Linebacker/Services/FCSAmerica-Linebacker-Web-DomainServices-LinebackerDomainService.svc' cannot be activated due to an exception during compilation. The exception message is: Operation named 'SearchCustomers' does not conform to the required signature. Return types must be an entity, collection of entities, or one of the predefined serializable types.. ---> System.InvalidOperationException: Operation named 'SearchCustomers' does not conform to the required signature. Return types must be an entity, collection of entities, or one of the predefined serializable types.
at System.ServiceModel.DomainServices.Server.DomainServiceDescription.ValidateMethodSignature(DomainOperationEntry method)
at System.ServiceModel.DomainServices.Server.DomainServiceDescription.AddInvokeOperation(DomainOperationEntry method)
at System.ServiceModel.DomainServices.Server.DomainServiceDescription.Initialize()
at System.ServiceModel.DomainServices.Server.DomainServiceDescription.CreateDescription(Type domainServiceType)
at System.ServiceModel.DomainServices.Server.DomainServiceDescription.<>c__DisplayClass8.<GetDescription>b__7(Type type)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at System.ServiceModel.DomainServices.Server.DomainServiceDescription.GetDescription(Type domainServiceType)
at System.ServiceModel.DomainServices.Hosting.DomainServiceHost..ctor(Type domainServiceType, Uri[] baseAddresses)
at System.ServiceModel.DomainServices.Hosting.DomainServiceHostFactory.CreateServiceHost(Type serviceType, Uri[] baseAddresses)
at System.ServiceModel.Activation.ServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses)
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String normalizedVirtualPath)
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath)
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
--- End of inner exception stack trace ---
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath)
This is what our code looks like in the domain service...its a wrapper around a WCF call not an Entity object.
[Invoke]
public IEnumerable<Customer> SearchCustomers(string searchValue)
{
return new List<Customer>();
}
Do we need to install SP1 on the Host Server?
Will this impact older versions of Silverlight that are running there?
Are we missing an attribute or something?
We have basically copied every dll local to the hosts bin folder and referenced the same in a library folder on our development machines.
Things run fine on our developer machines but not on the Server.
Thanks
Qui_Jon
I've had much the same issue attempting to use a WCF RIA Service using ComplexObjects on a server which didn't have WCF RIA Services 1.0 SP1 installed on it. The short answer to your question is that yes, you will need to install WCF RIA Services V1.0 SP1 on the server. It shouldn't impact on anything else running there.
When you run the installer, it might complain that it can't find Visual Studio. If so, quit the installer, open a Command Prompt, change to the directory containing the RiaServices.msi installer and run the following command:
msiexec /i RiaServices.msi SERVER=TRUE
So here was the problem.
We had built and were running Complex Types with RIA with no issues on our local development machines which had RIA SP1.
Our deployment wrapped all of these DLL and deployed them to IIS on our Development Web Server and then we got the error that it must be a supported type like Entity.
So we looked in the GAC on the Deveopment Server and the OLD version of RIA was installed and in the GAC and it has the same version that SP1 has that were were deploying with our IIS install.
Thus the GAC was over-riding our DLL and we were never referencing the SP1 dlls were were deploying with our solution.
Since we DONT want to have to install RIA on our web servers and this was a "junk" install of the original RIA Services we just uninstalled it and then our deployed SP1 dll were correctly referenced and the problem was fixed.
The OTHER solution would have been to install RIA SP1 with the command line options shown above.
Thanks for the response everyone...