I have multiple dozen of table definitions and data in text form.
The table definitions could also change from time to time.
I would like to create EF Core tables from theese definitions at runtime and fill them from these files.
Migration would be also nice if possible.
Is this Possible?
Alternatively: Is it possible to SQL"Create Table xyz" from a DatabaseContext?
Or should i better just use plain (Npg)sqlCommands to create, insert and update?
I am using Npgsql btw.
If you just want to execute your own SQL to create your tables, then it doesn't really matter if you do it with NpgsqlCommand (ADO.NET) or EF Core. EF Core doesn't really have any added value for executing that kind of raw SQL, so you're probably better off just using NpgsqlCommands. However, it's your responsibility to make sure your EF Core code model corresponds exactly to your text definitions. Either you manually make sure of that (by maintaining a C# class model that corresponds exactly), or you can reverse-engineer the model from the database created from your definition).
However, migrations are a very different thing. If the idea is to keep the external table definitions and even evolve them, then migrations don't really make sense - your external definitions are your single source of truth on the schema, and everything is derived from that. If you want to work with EF Core migrations, then your C# model must be your single source of truth - you make changes on that and the database is updated.
Note that there's an issue open on updating a code model from an existing database - this would allow you to make changes on your database (via your external definitions?) and update the code model, instead of regenerating it from scratch. However this isn't implemented yet.
Related
I want to do a versioning control of my database.
I currently control my front-end applications through git, however I am creating my database and would like to have a versioning of my tables, function and procedures, how can I accomplish this for database?
That is, I will make a change in a function but I would like to save the previous one I was executing in case there is any problem I can put the previous one again.
Major tools for versioning of database structure including optional data migration are:
Flyway
Liquibase
However, the specifics of your question are beyond what existing tools offer today.
If you need to have two or more versions of some object in the database for parallel use (e.g. step-by-step migration of tables, triggers, aso.) you are better off using either:
a naming scheme embedded into the objects, e.g. my_cool_function_v2 vs. my_cool_function_v3
...or:
use a different database schema for every major version (if you adhere to the semantic version approach), e.g. CREATE FUNCTION my_schema_v2.my_cool_function will not collide with my_schema_v1.my_cool_function
In both cases, you usually have to manage referencing the newer version where wanted. For the second approach, this can be further simplified with the schema search_path, which you can modify to prefer a new schema containing new versions of the objects, e.g. with:
SET search_path TO my_schema_v2, my_schema_v1, public;
dynamically (useful for testing in the live system without affecting actual applications/users) and once you are confident the basics are set, include it into the PostgreSQL configuration (postgresql.conf) so that the new schema becomes standard for every new connection:
search_path = 'my_schema_v2, my_schema_v1, public'
After you have migrated all of the new objects and everything is working fine, you can remove the old my_schema_v1 from the search_path and also DROP ... CASCADE it to remove all the old objects at once.
However, one downside of the schema approach is that if you always create all objects (functions, triggers, tables, ...) in all schemas, you'll lose the benefits of it when combined with the search_path. Therefore, I would create different schemas for different objects, e.g. data_v1 for the data (tables, indexes, ...) and func_v1 for other things (functions, procedures, ...). That way you can evolve the structure independently of the data but at the same time, you can also start evolving the table structure and automatically benefit from fixes/improvements in functions but also test whether changes are forward-compatible.
Hi future readers of this Q, I am the maintainer of project #yuniql: a schema versioning and migration tool than you can run with zero dependencies. In scenario described, most tools treats every change in the database as immutable or atomic change and if you need to rollback your change, it will just be another minor or major version. This way you can fully reconstruct the database anywhere, anytime and at the same time tracks the evolution of your database schema.
In yuniql, your workspace will typically be organized like this
v0.00
+ create_schema.sql
+ create_table_customers.sql
v0.01
+ create_table_custmomers_with_contact.sql
v0.02
+ create_table_custmomers_with_picture.sql
You may refer to github for getting started and samples ;)
https://github.com/rdagumampan/yuniql
https://yuniql.io/docs/get-started-postgresql/
Br, Rodel
We are using Entity Framework Code First, and have our models in C#.
One of our DBAs added an additional column, how do we migrate his change into our Net Core Project? Is there a command line to automatically sync this?
We know command line below will take a whole database and place into C#. We are only interested in small modified changes.
Scaffold-DbContext "Server=(localdb)\mssqllocaldb;Database=Blogging;Trusted_Connection=True;" Microsoft.EntityFrameworkCore.SqlServer -OutputDir Models
That's all you've got. Using an existing database is an all or nothing affair. Either you manage your entities via code and migrate changes there back to your database, or you scaffold the entities from your database. In other words, a database change means you need to re-scaffold everything. There's no such thing as "code migration".
Alternatively, I suppose you could just manually modify the entity class. For simple changes like adding a new column, that's probably the best approach, as long as you take care to replicate the property exactly as it should be to match the column in the database.
EDIT (based on comments on question)
Okay, so if you're doing code-first, then no one should be manually touching the database ever. Period. You have to just make that a rule. If the DBA wants to make a change, he can communicate that to your team, where you can make the appropriate code change and hand back a SQL script to perform the migration. This is where DevOps comes into play. Your DBA should be part of your team and you all should be making decisions together. If you can't do that, this isn't going to work.
Add column to the model.
Generate migration.
Migration file 20180906120000_add_dba_column.cs will be generated
Manually add new record about just created migration in table _EFMigrationHistory
record: MigrationId: 20180906120000_add_dba_column, ProductVersion: <copy from previous migration>
If you want do it automatically, means you want combine two different approaches in same time.
I think there was a reason why EF team separated "EF Code First" and "EF Database first" approaches.
When you choose EF Code first approach, that means that source of truth is your application code.
If you want stick to have code-first approach, that mean you should train your DBA to write EF migrations.
I am creating an application in C# Asp.net using Code First Entity Framework that will be using a different databases for different customers (in other words every customer has its own database, that will be generated on first time use).
I am trying to figure out a way to update all these databases automatically whenever I apply changes to my objects. In other words, how would I approach a cleanstep system in Code First EF?
Currently I am using InitializerIfModelChange to define a simple database that allows me to test my application whenever a schema change occurs. However, this method drops the database, which obviously is unacceptable in case of customer databases.
I must assume hundreds of customers so updating all databases by hand is not an option
I do not mind writing code that copies the data into a new database.
I think the best solution would be a way to backup a database somehow and then reinsert all data into the newly created database. Even better would be a way that automatically updates the schema without dropping the database. However I have no idea how to approach this. Can anyone point me in the right direction?
The link posted by Joakim was helpful. It requires you to update to EF 4.3.1 (dont forget your references in other projects if you have them) after which you can run the command that enables the migration. To automatically update the schema from code you can use
Configuration configuration = new Configuration();
DbMigrator migrator = new DbMigrator(configuration);
migrator.Update();
Database.SetInitializer<DbContext>(null);
I am currently working on a project with the final aim of converting a legacy system to a true N-Layer architecture. The initial part of the project involves converting the underlying database to a true relational design.
The underlying database is currently running on the IBM iSeries. The tables are defined using DDS and contain mountains of redundant data, no integrity checking and poorly designed keys etc. Basically refactoring them to a fully normalized design is a non-starter.
New tables are going to be designed from scratch. They will also be on the same iSeries but will be defined with DDL. This will also involve re-writing any insert or update code throughout the application to utilise the new tables. There is however a large amount of legacy apps responsible for reports, displays etc that will not be re-written at this point and will still be reading from the original tables. So we need to keep the data in old, legacy tables in sync with new table data. I was wondering if anyone had ever done something similar or had any suggestions? I am currently thinking either:
1) The stored procedures that inserts, updates, deletes from new table A will also do the same to matching legacy table B. At some point down the line the stored procedure will need to be modified to stop syncing table B
2) Place triggers on table A that also modifies table B. Then the triggers will be removed down the line... the stored procedure don't need to be changed but will this still work fine with transaction management?
3) Remove legacy table B and recreate it as view on table A. Not sure if this will work as table B works on keyed access and I believe views don't support this?
Would be interested to hear anyone's thoughts?
Cheers
I favour number 3. I assume that you would run some conversion process that would take the existing data from legacy table B and put it into new table A (or more likely, a set of new tables since it sounds like legacy table B is denormalized).
If your legacy apps responsible for reports, displays etc don't need to do any updates then using views is the ideal situation. I think keeping two separate copies of data (one normalized, one denormalized) would turn into a headache, especially when the manager's TPS reports don't match what he entered in the new system. At least with only one copy of the data, you don't have to worry about performing data fixes in two places.
I know that there are a few (automatic) ways to create a data access layer to manipulate an existing database (LINQ to SQL, Hibernate, etc...). But I'm getting kind of tired (and I believe that there should be a better way of doing things) of stuff like:
Creating/altering tables in Visio
Using Visio's "Update Database" to create/alter the database
Importing the tables into a "LINQ to SQL classes" object
Changing the code accordingly
Compiling
What about a way to generate the database schema from the objects/entities definition? I can't seem to find good references for tools like this (and I would expect some kind of built-in support in at least some frameworks).
It would be perfect if I could just:
Change the object definition
Change the code that manipulates the object
Compile (the database changes are done auto-magically)
Check out DataObjects.Net - is is designed to support exactly this case. Code only, and nothing else. Its schema upgrade layer is probably the most featured one you can find, and it really fully abstracts schema upgrade SQL.
Check out product video - you'll notice nothing additional is made to sync the schema. Schema upgrade sample shows the intended usage of this feature.
You may be looking for an Object Database.
I believe this is the problem that the Microsofy Entity Framework is trying to address. Whilst not specifically designed to "Compile (the database changes are done auto-magically)" it does address the issue of handling changes to the domain model without a huge dependance on the underlying data model.
As Jason suggested, object db might be a good choice. Take a look at db4objects.
What you described is GORM. It is part of the Grails framework and is built to work with Hibernate (maybe JPA in the future). When I was first using Grails it seemed backwards. I was more comfortable with a Rails style workflow of making the tables and letting the framework generate scaffolding from the database schema. GORM persists your domain objects for you so you create and change the objects, it manages database create/update. This makes more sense now that I have gotten used to it. Sorry to tease you if you aren't looking for a new framework but it is on the roadmap for release 1.1 to make GORM available standalone.
When we built the first version of our own framework (Inon Datamanager) I had it read pre-existing SQL tables and autogenerate Java objects from them.
When my colleagues who came from a Smalltalkish background built the second version, they started from the objects and then autogenerated the tables.
Actually, they forgot about the SQL part altogether until I came back in and added it. But nowadays we just run a trigger on application startup which iterates over the object model, checks if the tables and all the right columns exist, and creates them if not. Very convenient.
This turned out to be a lot easier than you might expect - if your favourite tool doesn't support a similar process, you could probably write it in a couple of hours - assuming the relational to object mapping is relatively simple.
But the point is, it seems to depend on whether you're culturally an object person or a database person - you can regard either one as the authoritative source.
Some of the really big dogs, such as ERwin Data Modeler, will go object to DB. You need to have the big bucks to afford the product though.
I kept digging around some of the "major" frameworks and it seems that Django does exactly what I was talking about. Or so it seems from this screencast.
Does anyone have any remark to make about this? Does it work well?
Yes, Django works well.
yes, it will generate your SQL tables from your data model definitions (written in python)
It won't always alter existing tables if you update your structure, you might have to run an ALTER table manually
Ruby on Rails has an even more advanced version of these features (Rails migrations), but I don't like the framework as much, I find ruby and rails pretty idiosyncratic
Kind of a late answer, but here it goes:
I faced the exact same problem and ended up writing my own solution for it, working with .NET and SQL Server only however. It basicaly does implement the process you describe:
All DB objects are kept as embedded CREATE scripts as part of the source code
DB Objects are set up automatically (or on request) when using the data access functionality
All non-table changes are also performed automatically (or on request) at the same time
Table changes, which may require special attention to migrate data, are performend via (manually created) change scripts also upon upgrading the database
Even manual changes made to any databse object can be detected, so that schema integrity can be verified and rectified
An optional lightweight ORM can map stored procedures and objects as well as result sets (even multiple)
A command-line application helps keeping the SQL source files in sync with a development database
The library including the database are free under a LGPL license.
http://code.google.com/p/bsn-modulestore/