We are using Entity Framework Code First, and have our models in C#.
One of our DBAs added an additional column, how do we migrate his change into our Net Core Project? Is there a command line to automatically sync this?
We know command line below will take a whole database and place into C#. We are only interested in small modified changes.
Scaffold-DbContext "Server=(localdb)\mssqllocaldb;Database=Blogging;Trusted_Connection=True;" Microsoft.EntityFrameworkCore.SqlServer -OutputDir Models
That's all you've got. Using an existing database is an all or nothing affair. Either you manage your entities via code and migrate changes there back to your database, or you scaffold the entities from your database. In other words, a database change means you need to re-scaffold everything. There's no such thing as "code migration".
Alternatively, I suppose you could just manually modify the entity class. For simple changes like adding a new column, that's probably the best approach, as long as you take care to replicate the property exactly as it should be to match the column in the database.
EDIT (based on comments on question)
Okay, so if you're doing code-first, then no one should be manually touching the database ever. Period. You have to just make that a rule. If the DBA wants to make a change, he can communicate that to your team, where you can make the appropriate code change and hand back a SQL script to perform the migration. This is where DevOps comes into play. Your DBA should be part of your team and you all should be making decisions together. If you can't do that, this isn't going to work.
Add column to the model.
Generate migration.
Migration file 20180906120000_add_dba_column.cs will be generated
Manually add new record about just created migration in table _EFMigrationHistory
record: MigrationId: 20180906120000_add_dba_column, ProductVersion: <copy from previous migration>
If you want do it automatically, means you want combine two different approaches in same time.
I think there was a reason why EF team separated "EF Code First" and "EF Database first" approaches.
When you choose EF Code first approach, that means that source of truth is your application code.
If you want stick to have code-first approach, that mean you should train your DBA to write EF migrations.
Related
I have multiple dozen of table definitions and data in text form.
The table definitions could also change from time to time.
I would like to create EF Core tables from theese definitions at runtime and fill them from these files.
Migration would be also nice if possible.
Is this Possible?
Alternatively: Is it possible to SQL"Create Table xyz" from a DatabaseContext?
Or should i better just use plain (Npg)sqlCommands to create, insert and update?
I am using Npgsql btw.
If you just want to execute your own SQL to create your tables, then it doesn't really matter if you do it with NpgsqlCommand (ADO.NET) or EF Core. EF Core doesn't really have any added value for executing that kind of raw SQL, so you're probably better off just using NpgsqlCommands. However, it's your responsibility to make sure your EF Core code model corresponds exactly to your text definitions. Either you manually make sure of that (by maintaining a C# class model that corresponds exactly), or you can reverse-engineer the model from the database created from your definition).
However, migrations are a very different thing. If the idea is to keep the external table definitions and even evolve them, then migrations don't really make sense - your external definitions are your single source of truth on the schema, and everything is derived from that. If you want to work with EF Core migrations, then your C# model must be your single source of truth - you make changes on that and the database is updated.
Note that there's an issue open on updating a code model from an existing database - this would allow you to make changes on your database (via your external definitions?) and update the code model, instead of regenerating it from scratch. However this isn't implemented yet.
I am creating an application in C# Asp.net using Code First Entity Framework that will be using a different databases for different customers (in other words every customer has its own database, that will be generated on first time use).
I am trying to figure out a way to update all these databases automatically whenever I apply changes to my objects. In other words, how would I approach a cleanstep system in Code First EF?
Currently I am using InitializerIfModelChange to define a simple database that allows me to test my application whenever a schema change occurs. However, this method drops the database, which obviously is unacceptable in case of customer databases.
I must assume hundreds of customers so updating all databases by hand is not an option
I do not mind writing code that copies the data into a new database.
I think the best solution would be a way to backup a database somehow and then reinsert all data into the newly created database. Even better would be a way that automatically updates the schema without dropping the database. However I have no idea how to approach this. Can anyone point me in the right direction?
The link posted by Joakim was helpful. It requires you to update to EF 4.3.1 (dont forget your references in other projects if you have them) after which you can run the command that enables the migration. To automatically update the schema from code you can use
Configuration configuration = new Configuration();
DbMigrator migrator = new DbMigrator(configuration);
migrator.Update();
Database.SetInitializer<DbContext>(null);
I am preferring to manually migrate my tables in Django. Because using automated tools puts me in a place where I cannot see the impact. With impact, I mean the the time it takes the db get in synch with my models. Below is a simple example:
class User(models.Model):
first_name = CharField(..)
Let's say I want to add this:
class User(models.Model):
first_name = CharField(..)
last_name = CharField(..)
I will follow the these steps in my production server:
Disable site traffic.
Manually connect to the your DB server, let's say MySQL and add a field to the User table named last_name (make sure it is sync with the SQL generated for the new Model, of course.)
Update your model.
Upload new files, restart traffic.
I have two questions for this scenario:
Is this a preferred/acceptable way for manual db migration in Django?
If I just add a field with a specific default value to the User table by SQL manually, but don't update the model, will I still get DatabaseIntegrity exception?
Thanks in advance,
With all of the schema migration tools, such as south, there are ways of explicitly defining how your models get migrated. The benefits of using a tool such as this are:
Your migrations are stored in your version control system
There's a documented procedure to roll back schema migrations
If another developer joins your project, you can refer that person to the south documentation rather than explaining your own hacky solution to documenting schema migrations.
I think I should just emphasize a point here: Though south has automigration tools, you don't have to use automigration if you're using South.
Is this a preferred/acceptable way for manual db migration in Django?
I would answer no. As #Mike said Django has a reliable and fairly versatile ecosystem of migration tools, the most prominent of which is South. #Mike's answer has the details right.
To answer your second question:
If I just add a field with a specific default value to the User table by SQL manually, but don't update the model, will I still get DatabaseIntegrity exception?
No. Your models will continue to function normally. Of course if you want to do something with the new fields using Django's ORM you'll be better off adding them to the model class.
A side effect of this is that you can migrate legacy database tables by selectively choosing the fields to use in your models.
We currently run an ecommerce solution for a leisure and travel company. Everytime we have a release, we must bring the ecommerce site down as we update database schema and the data access code. We are using a custom built ORM where each data entity is responsible for their own CRUD operations. This is accomplished by dynamically generating the SQL based on attributes in the data entity.
For example, the data entity for an address would be...
[tableName="address"]
public class address : dataEntity
{
[column="address1"]
public string address1;
[column="city"]
public string city;
}
So, if we add a new column to the database, we must update the schema of the database and also update the data entity.
As you can expect, the business people are not too happy about this outage as it puts a crimp in their cash-flow. The operations people are not happy as they have to deal with a high-pressure time when database and applications are upgraded. The programmers are upset as they are constantly getting in trouble for the legacy system that they inherited.
Do any of you smart people out there have some suggestions?
The first answer is obviously, don't use an ORM. Only application programmers think they're good. Learn SQL like everyone else :)
OK, so back to reality. What's to stop you restricting all schema changes to be additions only. Then you can update the DB schema anytime you like, and only install the recompiled application until a safe time (6am works best I find) after the DB is updated. If you must remove things, perform the steps the other way round - install the new app leaving the schema unchanged, and then remove the bits from the schema.
You're always going to have a high-pressure time as you roll out changes, but at least you can manage it better by doing it in 2 easier to understand pieces. Your DBAs will be ok with updating the schema for the existing application.
The downside is that you have to be a lot more organised, but that's not a bad thing when dealing with production servers and you should be seriously organised about it currently.
Supporting this scenario will add significant complexity to your environment and/or process and/or application.
You can run a complex update process where your application code is smart enough to run correctly on both the old schema and the new schema at the same time. Then you can update the application first and the schema second. A third step may be to migrate any data, which again, the application has to be able to work with. In that case, you only need to "tombstone" the application for the time it takes to upgrade the application, which could just be seconds, depending on how many files and machines are involved in the upgrade.
In most cases, it's best to leave the application/environment/process simple and live with the downtown during a slow time of the day/week/month. Pretty much all applications need to be "taken down" for time to time for "regularly schedule maintenance".
I know that there are a few (automatic) ways to create a data access layer to manipulate an existing database (LINQ to SQL, Hibernate, etc...). But I'm getting kind of tired (and I believe that there should be a better way of doing things) of stuff like:
Creating/altering tables in Visio
Using Visio's "Update Database" to create/alter the database
Importing the tables into a "LINQ to SQL classes" object
Changing the code accordingly
Compiling
What about a way to generate the database schema from the objects/entities definition? I can't seem to find good references for tools like this (and I would expect some kind of built-in support in at least some frameworks).
It would be perfect if I could just:
Change the object definition
Change the code that manipulates the object
Compile (the database changes are done auto-magically)
Check out DataObjects.Net - is is designed to support exactly this case. Code only, and nothing else. Its schema upgrade layer is probably the most featured one you can find, and it really fully abstracts schema upgrade SQL.
Check out product video - you'll notice nothing additional is made to sync the schema. Schema upgrade sample shows the intended usage of this feature.
You may be looking for an Object Database.
I believe this is the problem that the Microsofy Entity Framework is trying to address. Whilst not specifically designed to "Compile (the database changes are done auto-magically)" it does address the issue of handling changes to the domain model without a huge dependance on the underlying data model.
As Jason suggested, object db might be a good choice. Take a look at db4objects.
What you described is GORM. It is part of the Grails framework and is built to work with Hibernate (maybe JPA in the future). When I was first using Grails it seemed backwards. I was more comfortable with a Rails style workflow of making the tables and letting the framework generate scaffolding from the database schema. GORM persists your domain objects for you so you create and change the objects, it manages database create/update. This makes more sense now that I have gotten used to it. Sorry to tease you if you aren't looking for a new framework but it is on the roadmap for release 1.1 to make GORM available standalone.
When we built the first version of our own framework (Inon Datamanager) I had it read pre-existing SQL tables and autogenerate Java objects from them.
When my colleagues who came from a Smalltalkish background built the second version, they started from the objects and then autogenerated the tables.
Actually, they forgot about the SQL part altogether until I came back in and added it. But nowadays we just run a trigger on application startup which iterates over the object model, checks if the tables and all the right columns exist, and creates them if not. Very convenient.
This turned out to be a lot easier than you might expect - if your favourite tool doesn't support a similar process, you could probably write it in a couple of hours - assuming the relational to object mapping is relatively simple.
But the point is, it seems to depend on whether you're culturally an object person or a database person - you can regard either one as the authoritative source.
Some of the really big dogs, such as ERwin Data Modeler, will go object to DB. You need to have the big bucks to afford the product though.
I kept digging around some of the "major" frameworks and it seems that Django does exactly what I was talking about. Or so it seems from this screencast.
Does anyone have any remark to make about this? Does it work well?
Yes, Django works well.
yes, it will generate your SQL tables from your data model definitions (written in python)
It won't always alter existing tables if you update your structure, you might have to run an ALTER table manually
Ruby on Rails has an even more advanced version of these features (Rails migrations), but I don't like the framework as much, I find ruby and rails pretty idiosyncratic
Kind of a late answer, but here it goes:
I faced the exact same problem and ended up writing my own solution for it, working with .NET and SQL Server only however. It basicaly does implement the process you describe:
All DB objects are kept as embedded CREATE scripts as part of the source code
DB Objects are set up automatically (or on request) when using the data access functionality
All non-table changes are also performed automatically (or on request) at the same time
Table changes, which may require special attention to migrate data, are performend via (manually created) change scripts also upon upgrading the database
Even manual changes made to any databse object can be detected, so that schema integrity can be verified and rectified
An optional lightweight ORM can map stored procedures and objects as well as result sets (even multiple)
A command-line application helps keeping the SQL source files in sync with a development database
The library including the database are free under a LGPL license.
http://code.google.com/p/bsn-modulestore/