migrating django-model field-name change without losing data - django-models

I have a django project with a database table that already contains data. I'd like to change the field name without losing any of the data in that column. My original plan was to simply change the model field name in a way that would not actually alter the name of the db table (using the db_column column parameter):
The original model:
class Foo(models.Model):
orig_name = models.CharField(max_length=50)
The new model:
class Foo(models.Model):
name = models.CharField(max_length=50, db_column='orig_name')
But, running South's schemamigration --auto produces a migration script that deletes the original column, orig_name, and adds a new column, name, which would have the unwanted side effect of deleting the data in that column. (I'm also confused as to why South wants to change the name of the column in the db, since my understanding of db_column was that it enables a change to the model field name without changing the name of the database table column).
If I can't get away with changing the model field without changing the db field, I guess I could do a more straight forward name change like so:
The original model:
class Foo(models.Model):
orig_name = models.CharField(max_length=50)
The new model:
class Foo(models.Model):
name = models.CharField(max_length=50)
Regardless of which strategy I end up using (I would prefer the first, but would find the second acceptable), my primary concern is ensuring that I don't lose the data that is already in that column.
Does this require a multi-step process? (such as 1. adding a column, 2. migrating the data from the old column to the new column, and 3. removing the original column)
Or can I alter the migration script with something like db.alter_column?
What is the best way to preserve the data in that column while changing the column's name?

Changing the field name while keeping the DB field
Adding an answer for Django 1.8+ (with Django-native migrations, rather than South).
Make a migration that first adds a db_column property, and then renames the field. Django understands that the first is a no-op (because it changes the db_column to stay the same), and that the second is a no-op (because it makes no schema changes). I actually examined the log to see that there were no schema changes...
operations = [
migrations.AlterField(
model_name='mymodel',
name='oldname',
field=models.BooleanField(default=False, db_column=b'oldname'),
),
migrations.RenameField(
model_name='mymodel',
old_name='oldname',
new_name='newname',
),
]

Django 2.0.9 (and onwards) can automatically detect if a field was renamed and gives an option to rename instead of delete and create a new one
(same works for Django 2.2)
Initial answer
Posting, if it's still helpful for someone.
For Django 2.0 + simply rename the field in the model
class Foo(models.Model):
orig_name = models.CharField(max_length=50)
to
class Foo(models.Model):
name = models.CharField(max_length=50)
Now run python manage.py makemigrations
It'll generate migration with operations for removing the old field and adding the new one.
Go ahead and change that to following.
operations = [
migrations.RenameField(
model_name='foo',
old_name='orig_name',
new_name='name')
]
Now run python manage.py migrate it'll rename the column in DB without losing data.

It is quite easy to fix. But you will have to modify the migration yourself.
Instead of dropping and adding the column, use db.rename_column. You can simply modify the migration created by schemamigration --auto

Actually with Django 1.10, just renaming the field in the model and then running makemigrations, immediately identifies the operation (ie. one field disappeared, another appeared in its stead):
$ ./manage.py makemigrations
Did you rename articlerequest.update_at to articlerequest.updated_at (a DateTimeField)? [y/N] y
Migrations for 'article_requests':
article_requests/migrations/0003_auto_20160906_1623.py:
- Rename field update_at on articlerequest to updated_at

I've run into this situation. I wanted to change the field names in the model but keep the column names the same.
The way I've done it is to do schemamigration --empty [app] [some good name for the migration]. The problem is that as far as South is concerned, changing the field names in the model is a change that it needs to handle. So a migration has to be created. However, we know there is nothing that has to be done on the database side. So an empty migration avoids doing unnecessary operation on the database and yet satisfies South's need to handle what it considers to be a change.
Note that if you use loaddata or use Django's test fixture facility (which uses loaddata behind the scenes). You'll have to update the fixtures to use the new field name because the fixtures are based on the model field names, not the database field names.
For cases where column names do change in the database, I never recommend the use db.rename_column for column migrations. I use the method described by sjh in this answer:
I have added the new column as one schemamigration, then created a datamigration to move values into the new field, then a second schemamigration to remove the old column
As I've noted in a comment on that question, the problem with db.rename_column is that it does not rename constraints together with the column. Whether the issue is merely cosmetic or whether this mean a future migration may fail because it cannot find a constraint is unknown to me.

It is possible to rename a field without doing any manual migration file editing:
▶︎ Start with something like this:
class Foo(models.Model):
old_name = models.CharField(max_length=50)
▶︎ Add db_column=OLD_FIELD_NAME to the original field.
class Foo(models.Model):
old_name = models.CharField(max_length=50, db_column='old_name')
▶︎ Run: python3 manage.py makemigrations
▶︎ Rename the field from OLD_FIELD_NAME to NEW_FIELD_NAME
class Foo(models.Model):
new_name = models.CharField(max_length=50, db_column='old_name')
▶︎ Run: python3 manage.py makemigrations
You will be prompted:
Did you rename MODEL.OLD_FIELD_NAME to MODEL.NEW_FIELD_NAME (a ForeignKey)? [y/N] y
This will generate two migration files rather than just one, although both migrations are auto-generated.
This procedure works on Django 1.7+.

I ran into this situation on Django 1.7.7. I ended up doing the following which worked for me.
./manage.py makemigrations <app_name> --empty
Added a simple subclass of migrations.RenameField that doesn't touch the database:
class RenameFieldKeepDatabaseColumn(migrations.RenameField):
def database_backwards(self, app_label, schema_editor, from_state, to_state):
pass
def database_forwards(self, app_label, schema_editor, from_state, to_state):
pass

UPDATE In Django 3.1 it is quite simple for changing only one field at a time.
In my case:
The old field name was: is_admin
The new field name was: is_superuser
When I make migrations by python manage.py makemigrations it asked me do I want to rename the field or not. And I just hit y to rename. Then I migrate by python manage.py migrate. The terminal history in my case looks like:
NOTE: I did not test with more than one field at a time.

This is for Django 4.0.
Let's do this with an example.
My original field name was anticipated_end_date, I need to name it tentative_end_date. Follow the following steps to complete the operation
Change the anticipated_end_date to tentative_end_date inside the model
run python manage.py makemigrations. Ideally, it would show the following message
Was your_model_name.anticipated_end_date renamed to your_model_name.tentative_end_date (a DateField)? [y/N]
If it shows this message, then just press y and you are good to migrate, as it will generate the correct migration. However, if makemigrations command does not ask about renaming the model field, then go inside the generated migration and change the operations content the following way:
operations = [
migrations.RenameField(
model_name='your_model_name',
old_name='anticipated_end_date',
new_name='tentative_end_date',
),
]
Now you can run python manage.py migrate.
This way your model field/DB column will be renamed, and your data will not be lost.

As pointed out in the other responses, it is now quite easy to rename a field with no changes on the database using db_column. But the generated migration will actually create some SQL statements. You can verify that by calling ./manage.py sqlmigrate ... on your migration.
To avoid any impact on your database you need to use SeparateDatabaseAndState to indicate to Django that it doesn't need to do something in DB.
I wrote a small article about that if you want to know more about it.

1.Edit the field name on the django model
2.Create an empty migration like below:
$ python manage.py makemigrations --empty testApp (testApp is your application name)
Edit the empty migration file which is created recently
operations = [
migrations.RenameField('your model', 'old field’, 'new field'),
]
Apply the migration
$ python manage.py migrate
Database column name will be altered with new name.

Related

Want to replace old liquibase migration script file with new ones

I am using liquibase scripts with Cordapp. And previously the first version databaseChangeLog file was having all table creations in one single change set and in a later point of time we have split it into different databaseChangeLog having each changeset.
Now the problem is some production testing environments have the data in it with the older script, but we want to use the new scripts.
The change done is like →
Old: abc-master.xml contained abc-init.xml (usual way)|
Now: abc-master.xml contains abc-v1.xml and
abc-v1.xml contains table-v1.xml files for each table creation
Solution we were thinking is like,
create new tables with slight name change → then copy the data from old tables here → then drop old tables. So then we can remove old tables and old scripts (i assume)
Still the DATABASECHANGELOG will probably have the old scripts data, would that be a problem?
Or is there a far better way to do this?
Many thanks.
I answered this also on the Liquibase forums, but I'll copy it here for other people.
The filename is part if the unique key on the databasechangelog table (ID/Author/Filename). So when you change the filename of a changeset that has already executed, that is now in-fact a new changeset according to Liquibase.
I normally recommend that my customers never manually update the databasechangelog table, but in this case I think it might be the best course of action for you. That way your new file structure is properly reflected in the databasechangelog table.
I would run an update-sql command on the new file structure, against one of your database where you have already executed the chagesets. This will show you what changesets are pending, and also the values for the filenames that you need to update.
we are planning to go with
<preConditions onFail="MARK_RAN">
<not>
<tableExists tableName="MY_NEW_TABLE"/>
</not>
</preConditions>
For all those table creation changeset in new distributed structure ones, so our assumptions are:
We can keep this new structure alone in code & remove the old INIT file.
For environments having existing data, eventhough these new structure of changesets will be considered as new changeset to run, the preconditions will prevent it running.
For fresh db deployments, it will work as expected, by creating all the required tables.

Makemigrations and auto_now_add

Django 1.9.7
PostgreSQL
Could you help me understand what should I do?
The option auto_now_addand default are mutually exclusive.
This is just a bit confusing. I'm in favour of pressing 1? But just decided to ask for some piece of advice.
Model
creation_date = models.DateField(auto_now_add=True)
Traceback
michael#michael:~/workspace/photoarchive$ python manage.py makemigrations
You are trying to add a non-nullable field 'creation_date' to place without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows)
2) Quit, and let me add a default in models.py

Access a SQL Server 2008 View From Grails

We have a Grails project that needs to pull data from a SQL Server 2008 view. We just need to do a basic select from the view.
Is there a config setting for views such that we can map a domain class to the view?
Or should we use raw SQL like:
db.rows("SELECT foo, bar FROM my_view")
You can use SQL, e.g. with groovy.sql.Sql as suggested in the other similar question's answer, but it's also possible in a domain class. If you create a domain class (use any sensible name) and specify its table name as the name of the view, you can select from it. You'll have problems creating and updating of course, but if you only want to read then it'll be fine:
class SomeDomainClass {
String foo
Integer bar
static mapping = {
table 'my_view'
}
}
If you name the class MyView then mapping isn't needed since the naming convention applies, but this would be a bad name for the class since using it isn't related to the fact that it's backed by a view.
Note that you'll also have problems when using dbCreate set to "create", "create-drop", or "update" since Hibernate will try to create the table, but it shouldn't cause any real problem and just display an ignorable error like "could not create table 'my_view' since it already exists". And once you move to using database migrations this won't be a problem at all.

can CakePHP automatically create tables from models?

In python::Pylons i'm able to issue a setup-app command and it will look at my Models and issue the appropriate CREATE TABLE or CREATE INDEX ddl for my particular database.
it seems like this would be a feature in CakePHP, but i'm having trouble finding it.
in fact i see this in the manual:
"You can create your database tables as you normally would. When you create your Model classes, they'll automatically map to the tables that you've created."
which leads me to believe it doesn't exist?
No, it's other way around - you can create models, controllers and views by having DB schema. It's more logical to have a DB design schema first.
Check this out
Some of the comments in the accepted answer above lead me to creating this answer. You can technically create new tables on the fly using the YourModel->query() function. I am currently using this in a Behavior I am writing. This works in CakePHP 2.x, pretty sure it works in 1.3 as well.
In the setup function for the Behavior I am checking to see if the table already exists. If it doesn't I create it.
$dataSource = ConnectionManager::getDataSource('your DB');
if(!in_array($tableName, $dataSource->listSources()){
$this->createYourTableFunction();
}
In the createYourTableFunction you create a temporary model to run the YourModel->query() against. And just provide it your SQL instructions. When creating your temporary model just set the table parameter to false so you don't get a missing table error.
$YourModel = new Model(array('table' => false, 'name' => 'YourModel', 'ds' => 'Your DB'));
$YourModel->query('SQL instruction string');

Adaptive Database

Are there any rapid Database protoyping tools that don't require me to declare a database schema, but rather create it based on the way I'm using my entities.
For example, assuming an empty database (pseudo code):
user1 = new User() // Creates the user table with a single id column
user1.firstName = "Allain" // alters the table to have a firstName column as varchar(255)
user2 = new User() // Reuses the table
user2.firstName = "Bob"
user2.lastName = "Loblaw" // Alters the table to have a last name column
Since there are logical assumptions that can be made when dynamically creating the schema, and you could always override its choices by using your DB tools to tweak it later.
Also, you could generate your schema by unit testing it this way.
And obviously this is only for prototyping.
Is there anything like this out there?
Google's Application Engine works like this. When you download the toolkit you get a local copy of the database engine for testing.
Grails uses Hibernate to persist domain objects and produces behavior similar to what you describe. To alter the schema you simply modify the domain, in this simple case the file is named User.groovy.
class User {
String userName
String firstName
String lastName
Date dateCreated
Date lastUpdated
static constraints = {
userName(blank: false, unique: true)
firstName(blank: false)
lastName(blank: false)
}
String toString() {"$lastName, $firstName"}
}
Saving the file alters the schema automatically. Likewise, if you are using scaffolding it is updated. The prototype process becomes run the application, view the page in your browser, modify the domain, refresh the browser, and see the changes.
I agree with the NHibernate approach and auto-database-generation. But, if you want to avoid writing a configuration file, and stay close to the code, use Castle's ActiveRecord. You declare the 'schema' directly on the class with via attributes.
[ActiveRecord]
public class User : ActiveRecordBase<User>
{
[PrimaryKey]
public Int32 UserId { get; set; }
[Property]
public String FirstName { get; set; }
}
There are a variety of constraints you can apply (validation, bounds, etc) and you can declare relationships between different data model classes. Most of these options are parameters added to the attributes. It's rather simple.
So, you're working with code. Declaring usage in code. And when you're done, let ActiveRecord create the database.
ActiveRecordStarter.Initialize();
ActiveRecordStarter.CreateSchema();
May be not exactly responding to your general question, but if you used (N)Hibernate then you can automatically generate the database schema from your hbm mapping files.
Its not done directly from your code as you seem to be wanting but Hibernate Schema generation seems to work well for us
Do you want the schema, but have it generated, or do you actually want NO schema?
For the former I'd go with nhibernate as #tom-carter said. Have it generate your schema for you, and you are all good (atleast until you roll your app out, then look at something like Tarantino and RedGate SQL Diff or whatever it's called to generate update scripts)
If you want the latter.... google app engine does this, as I've discovered this afternoon, and it's very nice. If you want to stick with code under your control, I'd suggest looking at CouchDB, tho it's a bit of upfront work getting it setup. But once you have it, it's a totally, 100% schema-free database. Well, you have an ID and a Version, but thats it - the rest is up to you. http://incubator.apache.org/couchdb/
But by the sounds of it (N)hibernate would suite the best, but I could be wrong.
You could use an object database.

Resources