We are developing a .NET Core application on Linux.
We use JetBrains Rider.
I wanna to profile SQL-Server (remote or local) from my Linux dev machine.
How can I do that (from Linux) ?
You can download SQL-operations studio to manage SQL server on Linux:
However, a profiler is not included (at least not yet - AFAIK).
But I had the same problem earlier on with no profiler being present when having SQL-Server Express.
So I forked ExpressProfiler from Codeplex and fixed some bugs.
The resulting windows-forms application is here.
However, you can only run that with mono.
But because it uses a RichText-Control, it crashes frequently on mono.
Since I had the same problem as you, I ripped ExpressProfiler appart and created a console profiler.
The console-profiler works on .NET Core.
See here.
It captures all SQLs sent to the server, highlights syntax on the console (some problems with background switching that I haven't figured out, yet), but that's all it does. If you need execution duration or more filters than just the database, you need to add that yourselfs. Otherwise, you'll be fine.
The command line syntax is:
./sql_profiler --server {computername\instance} --username WebAppWebServices --password TOP_SECRET --db "The DB you want to profile";
or from the project:
dotnet run sql_profiler --server {computername\instance} --username WebAppWebServices --password TOP_SECRET --db "The DB you want to profile";
If you omit the username, it will attempt to connect with integrated security.
The command-line project is here.
If you don't want to compile it yourselfs, you can find a release here.
The beauty of this solution is that you can also run it directly on SSH, if you need to profile a sql-server that's not in your network.
If you standalone compile it, it's also not required that mono/.net/.net-core is installed on the server.
So you can skip CITRIX/RDP.
The other answer was helpful, but slightly outdated. Here's updated instructions.
Note that this doesn't work for all Entity Framework Core queries, see issue #2125 on GitHub.
Prerequisites:
Running the mssql-server-linux docker image
Install SQL Operations Studio (a fork of VS Code, cross-platform successor to SSMS)
Install the SQL Server Profiler Extension
(If you know what you're doing you can skip straight to step 12 below.)
I'm on Windows but have colleagues on Linux so tried to use cross-platform tools to set this up. As a baseline reference, here's a repro for checking if it works in your own environment:
Create new ASP.NET Core Web API Application (2.1) named FooBarBaz with this code added:
public class Item
{
public int Id { get; set; }
public string Name { get; set; }
}
public class MyDbContext : DbContext
{
public DbSet<Item> Items { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlServer("Server=localhost;Database=FooBarBaz;User=sa;Password=yourStrong(!)Password;");
}
}
Install Microsoft.EntityFrameworkCore
Install Microsoft.EntityFrameworkCore.SqlServer
Install Microsoft.EntityFrameworkCore.Tools
Run Add-Migration InitialCreate
Run Update-Database to create and scaffold the database
Register context in ConfigureServices using services.AddDbContext<MyDbContext>();
Change the controller to this:
[Route("api/[controller]")]
[ApiController]
public class ValuesController : ControllerBase
{
private readonly MyDbContext _context;
public ValuesController(MyDbContext context)
{
_context = context;
}
[HttpGet]
public ActionResult<IEnumerable<Item>> Get()
{
return _context.Items.ToList();
}
}
Open SQL Operations studio
Connect to your Sql Server in Docker
Click on the database that was created (FooBarBaz)
Press ALT+p to start profiling
Run your web application and access /api/values
You should now see something like this in the profiler:
Again, the screenshots and instructions were written from Windows, but my colleague uses a Debian Linux distro where things work similarly.
The best option is now to use Azure Data Studio (which Microsoft releases as a native Linux application):
https://learn.microsoft.com/en-us/sql/azure-data-studio/download-azure-data-studio
Then install the SQL Server Profiler extension:
https://learn.microsoft.com/en-us/sql/azure-data-studio/extensions/sql-server-profiler-extension
I'm enjoying the new Build tool in Visual Studio Online. Allows me to do almost everything that I do my local build server. But one thing that I'm missing is integration database tests: for every build run I re-create test database from scripts and run DB-tests against it.
In Visual Studio Online I can't seem to find any database instance available for my needs.
I have tried creating Azure SQL database (via PowerShell) for every build run and then delete it after the build is complete. But it takes forever (comparing to the rest of the build process) to create a database. And even when PowerShell scripts are done, database is not yet ready to accept requests - I need to constantly check if it is actually ready. So this scenario becomes too complex and not reliable.
Are there other options to do database (SQL Server) integration tests in Visual Studio Online?
Update: I think I'm not very clear of what I need - I need a free (very cheap) SQL Server instance to connect to that runs on build agent in VSO. Something like SQL Express or SQL CE or LocalDB, where I can connect to and re-create database to run C# tests against. Re-creating database or running tests is not a problem, having a valid connection string is a problem.
Update Oct 2016: I've blogged about how I do integration testing in VSTS
The TFS build servers come with MSSQL Server 2012 and MSSQL Server 2014 LocalDBs preinstalled.
Source: TFS Service - Software on the hosted build server
So, just put the following one-liner into your solution's post-build event to create a MYTESTDB LocalDB instance for your needs. This will allow you to connect to (LocalDB)\MYTESTDB an run the database integration tests just fine.
"C:\Program Files\Microsoft SQL Server\120\Tools\Binn\SqlLocalDB.exe" create "MYTESTDB" 12.0 -s
Source: SqlLocalDB Utility
In Azure DevOps, with .net Core and EF Core, I use a different technique.
I use a SQLite in memory database to execute both Integration and End to End tests.
Currently in .net Core you can use both InMemory database and SQLite with in memory option, to run any integration test in the default Azure DevOps CI Agent.
InMemory: https://learn.microsoft.com/en-us/ef/core/miscellaneous/testing/in-memory
Note that the InMemory database is not a relational database, it is a multi-purpose one, and just to mention one limitation:
InMemory will allow you to save data that would violate referential
integrity constraints in a relational database
SQLite in memory mode https://learn.microsoft.com/en-us/ef/core/miscellaneous/testing/sqlite
This approach offers a more realistic platform to test against.
Now, I went a bit further, I didn't want to just be able to run integration tests with database dependency in Azure DevOps, I wanted to also be able to host my WebAPIs in the CI Agent, and to share the database between the API DBcontext and my Persister object (Persister object is a helper class that allow me to automatically generate any kind of entity and save them to the database).
A quick note on Integration Tests and Ent to End tests:
Integration Tests
An example of integration test involving a database, could be a test of the Data Access Layer. In this case, normally, you would create a DBContext when starting a test, fill the target database with some data, use the component under test to manipulate the data, and again use the DBContext to make sure the assertions are satisfied.
This scenario is quite straight forward, in the same code you can share the same DBContext to generated the data and inject it to the component.
End to End Tests
Imagine you have like in my case a RESTful .net Core WebAPI you want to test, making sure all your CRUD operations are working as expected, and you want to test that filtering, pagination and so on are also correct.
In this case, it much more complex share the same DBContext between test (data setup and/or verification) and the WebAPI stack.
Before .net EF Core and WebHostBuilder
So far, the only way I knew was possible, was to have a dedicated server, VM or docker image, responsible serve the API, which had to be also accessible from the web or Azure DevOps.
Setup my integration tests to either re-create the DB, or be clever/limited enough to ignore completely the existing data, and make sure that each test was resilient to data corruption and fully reliable (no false negative or positive results).
Then I had to configure my build definition to run the tests.
Leveraging SQLite in memory with cache=shared and WebHostBuilder
Below I first describe the two majour technologies I use, then I add some code to show how to do it.
SQLite file::memory:?cache=shared
SQLite allow you to work in memory, instead of using a traditional file, this already gives us a huge performance boost, removing the I/O bottleneck, but on top of this, using the option cache=shared, we can use multiple connections within the same process to access the same data. If you need more than one database you can specify a name.
More info: https://www.sqlite.org/inmemorydb.html
WebHostBuilder
.net Core offers Host builders, WebHostBuilder allow us to create a server that startup and host our WebAPI, so that can be reached like if they were hosted on a real server.
When you use the WebHostBuilder in a test class, this two, are living within the same process.
More info: https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.hosting.webhostbuilder?view=aspnetcore-2.2
The Solution
When initialising an E2E test, create a new client to connect the api, create a dbcontext that you will use to seed the database and maybe assert.
Test initialisation:
[TestClass]
public class CategoryControllerTests
{
private TestServerApiClient _client;
private Persister<Category> _categoryPersister;
private Builder<Category> _categoryBuilder;
private IHouseKeeperContext _context;
protected IDbContextTransaction Transaction;
[TestInitialize]
public void TestInitialize()
{
_context = ContextProvider.GetContext();
_client = new TestServerApiClient();
ContextProvider.ResetDatabase();
_categoryPersister = new Persister<Category>(_context);
_categoryBuilder = new Builder<Category>();
}
[TestCleanup]
public void Cleanup()
{
_client?.Dispose();
_context?.Dispose();
_categoryPersister?.Dispose();
ContextProvider.Dispose();
}
[...]
}
TestServerApiClient class:
public class TestServerApiClient : System.IDisposable
{
private readonly HttpClient _client;
private readonly TestServer _server;
public TestServerApiClient()
{
var webHostBuilder = new WebHostBuilder();
webHostBuilder.UseEnvironment("Test");
webHostBuilder.UseStartup<Startup>();
_server = new TestServer(webHostBuilder);
_client = _server.CreateClient();
}
public void Dispose()
{
_server?.Dispose();
_client?.Dispose();
}
}
ContextProvider class is used to generate the DBContext which can be used to seed data or perform database queries for assertions.
public static class ContextProvider
{
private static bool _requiresDbDeletion;
private static IConfiguration _applicationConfiguration;
public static IConfiguration ApplicationConfiguration
{
get
{
if (_applicationConfiguration != null) return _applicationConfiguration;
_applicationConfiguration = new ConfigurationBuilder()
.AddJsonFile("Config/appsettings.json", optional: false, reloadOnChange: true)
.AddEnvironmentVariables()
.Build();
return _applicationConfiguration;
}
}
private static ServiceProvider _serviceProvider;
public static ServiceProvider ServiceProvider
{
get
{
if (_serviceProvider != null) return _serviceProvider;
var serviceCollection = new ServiceCollection();
serviceCollection.AddSingleton<IConfiguration>(ApplicationConfiguration);
var databaseType = ApplicationConfiguration?.GetValue<DatabaseType>("DatabaseType") ?? DatabaseType.SQLServer;
_requiresDbDeletion = databaseType == DatabaseType.SQLServer;
IocConfig.RegisterContext(serviceCollection, null);
_serviceProvider = serviceCollection.BuildServiceProvider();
return _serviceProvider;
}
set
{
_serviceProvider = value;
}
}
/// <summary>
/// Generate the db context
/// </summary>
/// <returns>DB Context</returns>
public static IHouseKeeperContext GetContext()
{
return ServiceProvider.GetService<IHouseKeeperContext>();
}
public static void Dispose()
{
ServiceProvider?.Dispose();
ServiceProvider = null;
}
public static void ResetDatabase()
{
if (_requiresDbDeletion)
{
GetContext()?.Database?.EnsureDeleted();
GetContext()?.Database?.EnsureCreated();
}
}
}
IocConfig class is an helper class I use in my framework to setup the dependency injection. The menthod used above, RegisterContext, is responsile to register the DBContext and set it up as desired, and because this is the same class used by the WebAPI, uses the configuration DatabaseType to determine what to do.
Inside this class probably you can find most of the "complexity".
When using SQLite in memory, you have to remember that:
The connection is not opened and closed automatically like when using SQL Server (that's why i used: context.Database.OpenConnection();)
If no connection is active, the database is deleted (that's why I used services.AddSingleton<IHouseKeeperContext>(s ... it is important that one connection is left open so that the database is not destroyed, but on the other hand you have to be careful to close all connections when a test ends, so that the database is eventually destroyed and the next test will correctly create a new empty one.
The rest of the class handles the SQL Server configuration for both Production and Testing setup. I can at any time setup the tests to use a real instance of SQL Server, all tests will keep being fully independend from the others but it will definitely be slow, and maybe suitable only for a nightly build (if needed, and it depends on the size of your system).
public class IocConfig
{
public static void RegisterContext(IServiceCollection services, IHostingEnvironment hostingEnvironment)
{
var serviceProvider = services.BuildServiceProvider();
var configuration = serviceProvider.GetService<IConfiguration>();
var connectionString = configuration.GetConnectionString(Constants.ConfigConnectionStringName);
var databaseType = DatabaseType.SQLServer;
try
{
databaseType = configuration?.GetValue<DatabaseType>("DatabaseType") ?? DatabaseType.SQLServer;
}catch
{
MyLoggerFactory.CreateLogger<IocConfig>()?.LogWarning("Missing or invalid configuration: DatabaseType");
databaseType = DatabaseType.SQLServer;
}
if(hostingEnvironment != null && hostingEnvironment.IsProduction())
{
if(databaseType == DatabaseType.SQLiteInMemory)
{
throw new ConfigurationErrorsException($"Cannot use database type {databaseType} for production environment");
}
}
switch (databaseType)
{
case DatabaseType.SQLiteInMemory:
// Use SQLite in memory database for testing
services.AddDbContext<HouseKeeperContext>(options =>
{
options.UseSqlite($"DataSource='file::memory:?cache=shared'");
});
// Use singleton context when using SQLite in memory if the connection is closed the database is going to be destroyed
// so must use a singleton context, open the connection and manually close it when disposing the context
services.AddSingleton<IHouseKeeperContext>(s => {
var context = s.GetService<HouseKeeperContext>();
context.Database.OpenConnection();
context.Database.EnsureCreated();
return context;
});
break;
case DatabaseType.SQLServer:
default:
// Use SQL Server testing configuration
if (hostingEnvironment == null || hostingEnvironment.IsTesting())
{
services.AddDbContext<HouseKeeperContext>(options =>
{
options.UseSqlServer(connectionString);
});
services.AddSingleton<IHouseKeeperContext>(s => {
var context = s.GetService<HouseKeeperContext>();
context.Database.EnsureCreated();
return context;
});
break;
}
// Use SQL Server production configuration
services.AddDbContextPool<HouseKeeperContext>(options =>
{
// Production setup using SQL Server
options.UseSqlServer(connectionString);
options.UseLoggerFactory(MyLoggerFactory);
}, poolSize: 5);
services.AddTransient<IHouseKeeperContext>(service =>
services.BuildServiceProvider()
.GetService<HouseKeeperContext>());
break;
}
}
[...]
}
Sample Test, where first I use the persister to generated data which is seeded in the database, then I use the API to get data, the test can be also reversed, using a POST request to set data and then using the DBContext to read the db and make sure the creation was successful.
[TestMethod]
public async Task GET_support_orderBy_Id()
{
_categoryPersister.Persist(3, (c, i) =>
{
c.Active = 1 % 2 == 0;
c.Name = $"Name_{i}";
c.Description = $"Desc_i";
});
var response = await _client.GetAsync("/api/category?&orderby=Id");
var categories = response.To<List<Category>>();
Assert.That.All(categories).HaveCount(3);
Assert.IsTrue(categories[0].Id < categories[1].Id &&
categories[1].Id < categories[2].Id);
response = await _client.GetAsync("/api/category?$orderby=Id desc");
categories = response.To<List<Category>>();
Assert.That.All(categories).HaveCount(3);
Assert.IsTrue(categories[0].Id > categories[1].Id &&
categories[1].Id > categories[2].Id);
}
Conclusions
I love the fact that I can run E2E tests in Azure DevOps for free, performances are incredibly good and this gives me a lot of confidence, ideal when you want to setup a continuous delivery environment.
Here is a screenshot of part of the build execution of this code in Azure DevOps (free version).
Sorry this ended up being longer than expected.
There is a "Redgate SQL CI" extension for VSTS in the marketplace you may want to try. See this link for details:
Within the extension, there are four actions available:
•Build – builds your database into a NuGet package from the database
scripts folder in source control
•Test – runs your tSQLt tests against the database
•Sync – synchronizes the package to an integration database
•Publish – publishes the package to a NuGet stream
You should push the integration tests (anything that needs an instance of your application) to be run in an environment as part of your release pipeline.
In your build just do compile and unit tests. If that competes you should trigger a Release and as part of your release pipeline your first step should be to deploy your database to an azure server.
Instead of trying to use SQL Azure you can create a VM in azure that already exists that has SQL server installed. Use remote scripting to deploy the database and execute your tests.
Even if you are not using the release tools to release this would work for you.
According to Scaleout with SQL Server you can use SignalR.SqlServer to keep SignalR synced on a load balancing setup. I have an existing MVC 5 website with its own database created using Entity Code First. The article seems to use a dedicated database with service broker enabled and says not to modify the database.
So do I need to have a separate database for this? I know Entity can be picky if the database schema doesn't match and I worry that if I try to use the SignalR Sql Server package with the existing database that the tables it creates will cause a context changed error.
Also can someone provide me with more information about using the Microsoft.AspNet.SignalR.SqlServer package. The article I linked doesn't give a ton of detail and I don't know if I need to change anything in my hub and groups or if it is all handled automatically.
You should be able to, though you'd likely want to separate your entity framework definitions from signalR. You can either put SignalR in a separate database, or give the two a separate schema.
In terms of configuration, you'll need to make an addition to the Startup class of your web project:
public class Startup
{
public void Configuration(IAppBuilder app)
{
var sqlConnectionString = "connection string here";
GlobalHost.DependencyResolver.UseSqlServer(sqlConnectionString);
this.ConfigureAuth(app);
app.MapSignalR();
}
}
I am working on a web api project using EF 6.1.3 (code first) fluent API and SQL Server 2008 R2. I create migrations in code using add-migration and I have been able to create several migrations with no issues. Now I made a change to the model (increased column size for three columns in different tables) as follows:
Protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
... //previously these columns had length = 50
modelBuilder.Entity<EvalDef>().Property(t =>t.Name).HasMaxLength(70);
modelBuilder.Entity<ProgDef>().Property(t => t.Name).HasMaxLength(70);
modelBuilder.Entity<ReqDocDef>().Property(t => t.Name).HasMaxLength(70);
...
}
I am using the MigrateDatabaseToLatestVersion initialization:
protected void Application_Start()
{
Database.SetInitializer(new
MigrateDatabaseToLatestVersion<FtoContext,FTO.Model.Migrations.FtoConfiguration>());
}
Now when I run the add-migration command (from PM console), EF will not detect the changes to the model and an empty migration is created. I can't figure out why the fluent api model changes are not noticed when creating the migration.
Also, when I run the app, the OnModelCreating method is executed but I do not get any exception...which I would expect since now the DB schema would not match the current model.
This question is not about SQL Server "3xpr355" (the obfuscation
in quotes is by design; it should prevent future searches from hitting
dead-ends, as mine have done).
I am using code-first and Entity Framework to develop an MVC 4 application. The SQL Server, IIS, and Visual Studio are all running on the same machine.
I was originally using SQL Server "3xpr355" but the requirement to put it on an exposed-to-the-Internet-via-ISS machine made hooking it up to a full-featured SQL Server necessary. I have the application set up to drop and re-create the database whenever the models change:
public class XyzDBContext : DbContext
{
public XyzDBContext()
: base("XyzDBContext")
{
Database.SetInitializer<XyzDBContext>(new DropCreateDatabaseIfModelChanges<XyzDBContext>());
}
public DbSet<XyzModel> XyzModels{ get; set; }
}
Here are my connection strings:
<connectionStrings>
<add
name="DefaultConnection"
connectionString="Data Source=.;Initial Catalog=aspnet-Xyz-20150131102119;Integrated Security=SSPI"
providerName="System.Data.SqlClient" />
<add
name="XyzDBContext"
connectionString="Data Source=.;Initial Catalog=XyzDatabase;Integrated Security=True"
providerName="System.Data.SqlClient"/>
</connectionStrings>
Expectations:
Execution of the portions of the application that use XyzDBContext should cause the database "XyzDatabase" to be created (if necessary).
Execution of the portions of the application that use the Membership Provider should cause the database "aspnet-Xyz-20150131102119" to be created (if necessary).
Actual Results:
Exception is thrown: System.Data.ProviderIncompatibleException "An
error occurred while getting provider information from the database.
This can be caused by Entity Framework using an incorrect connection
string. Check the inner exceptions for details and ensure that the
connection string is correct."
Inner Exception:
System.Data.ProviderIncompatibleException "The provider did not
return a ProviderManifestToken string."
Inner Exception:
System.Data.SqlClient.SqlException "Login failed for user
'DOMAIN\SERVER$'."
Measures:
I have tried using SQL Server Management Studio to add a login for the "DOMAIN\SERVER$" user, but the login always fails.
I have researched and tried many permutations of the connection strings, all but a few of the examples I have found were for use with SQL Server "3xpr355".
Well if you map each connection string (database) to it's own DbContext the you could reach your expectations but you still have to work with two contexts (which is painful).
Your third actual result is due to IIS that can't connect to the database. You have to go to IIS manager and change the identity of the application pool under which your application runs (You can change it to LocalSystem)
Finally, try this if you have already the databases created:
public class DefaultConnectionContext : DbContext, IDisposable
{
public DefaultConnectionContext()
: base("name=DefaultConnection")
{
Database.SetInitializer<DefaultConnectionContext>(null);
}
// Some dbsets
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
}
}
public class XyzDBContext : DbContext, IDisposable
{
public XyzDBContext()
: base("name=XyzDBContext")
{
Database.SetInitializer<XyzDBContext>(null);
}
// Some dbsets
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
}
}
Hope That helps
The solution to this problem is multifaceted:
Create an Application Pool
In IIS Manager, visit the "Application Pools" node and select the "Add Application Pool..." Action:
Select the new Application Pool and select the "Advanced Settings..." Action, then select the "..." button (on the "Identity" field):
Set the Application Pool identity to "LocalSystem"
Grant Permission
In SQL Server Management Studio (when connected to the database service), visit the "NT AUTHORITY\SYSTEM" node under "Logins" and bring up the "Properties" view, then select the "Server Roles" page, and grant all of the privileges by selecting the check-boxes:
In IIS Manager, Right-click on your site and select "Manage Application/Advanced Settings..." and Set the "Application Pool" property value to "CodeFirstMVC".
Run the Application
Run the application and visit the areas that would require a new database to be created; you should now see a new database on your SQL Server instance.