I have been working on developing the models for a Django app. I am working on it in my spare time. I have an issue when it comes to testing. Whenever I realize I have a mistake in my models, I have to go through the trivial but annoying process of dropping the corresponding database and then recreating it and running python manage.py syncdb. Apparently, this is because Django's syncdb cannot change the database schema. I do not fully understand what that means. Although, I know it is a pain to have to keep deleting and recreating my database every time I try and change something in my models.
Is there a better way of doing this? Is something wrong with my install? I feel like this is such a simple thing to fix that I must be doing something wrong.
No, nothing is wrong with your installation. It is a limitation of django.
There is a 3rd party app called django-south, which can be used for managing "migrations" (changes to database models). It is a very widely used app for managing database changes.
There are lots of documentation online to help you understand how south works, and how to use it.
This is a good tutorial on south
Related
I have a flask app that is deployed on Google's App Engine. I have noticed a minor bug and I would like to fix it but my database is already populated.
How can I make this minor code change and push / deploy back to my app without losing all my data? (which is probably a basic question but I'm not finding much. all tutorials online are focused on creating the app and deploy, not updating)
Thus far, I have been dropping and re-creating the tables whenever I redeploy, mostly out of ignorance. Here are the steps I have followed
1). make the change in my app
commit and push changes to bitbucket source code
in Google Cloud SDK: git pull
Google Cloud SDK: gcloud app deploy
These steps result in an empty database because the directory I am pushing from my local computer has an empty database. Is this where I should be using git merge?
Is this a database "migration" or is this a "git merge"? I'm not sure what the right terms are to use to research this further. Thanks.
There are a couple of angles to your question. I'm going to try to give you some information, but let me warn you, this isn't going to be a trivial change to your workflow, you'll have to change some things.
First of all, based on the way you worded your question I get the idea that you commit your database to git along with your code. If I got this right, then this is something that you need to stop doing. The database is not code, so it should not be committed to source control.
You should have a completely independent database on each installation of your application. For example, you will have a database on your own machine to do development. You will also need another database in your gcloud deployment. You may need more databases if you have other uses for your application. A very common third database for many people is one that is used for automated tests, which could also be located in your local development machine, but is not the same database that you use for day to day development.
To make changes to your database schema you will not drop and recreate tables anymore, that is clearly something that you already realized that needs an improvement. A good approach to make these changes is to use a database migration framework. These tools allow you to generate short scripts that make these changes to the database in a more focused way, without destroying and recreating everything, and for that reason, the data is in general not lost. For Flask-SQLAlchemy, the best option for database migrations is Flask-Migrate, which is a lightweight wrapper around the Alembic migration framework. (I might be biased here as I'm the author of the Flask-Migrate extension!).
Documentation for Flask-Migrate: https://flask-migrate.readthedocs.io/en/latest/.
I am currently investigating possible options of a migration framework/tool. I like the idea of ruby migrations on which the above frameworks are based.
So I am asking for your experience, opinions and maybe a comparison between them. Are you using them in production?
thanks for responses. The goal of this question was to get a feeling about which tools is used most in the developer community but it seems that migrations are not a hot topic here.
Anyway, I have decided to go with MigSharp as the codebase seem to be pretty clean and it is quite easy to handle and had build in support for MS SQL CE. Second runner up would have been FluentMigrator where I was not able to produce a working example for compact edition.
Cheers
I use FluentMigrator in production, and am a longtime contributor to FM. I think your question is to general; be more specific. Also, FM has a google group which is fairly active if you want FM information.
FM is derived from migrator.net, as I recall. It uses a fluent-syntax, and supports multiple databases. We have taken some inspiration from rails migrations, but it's definitely not a port. Worth checking out.
One thing I've learned is not to put your migrations in the same assembly as you app code. Separate them into a migration assembly, and use that for migrating your databases.
Also, you should always work on multiple environments to avoid problems with migrations run straight against production. I always have at least a development and production environment, and most of the time there is a testing environment as well.
I use mig#.
It works well, but you will need to have some guidelines for usage - as migrations can get complicated.
We use sequence number on the end of our migrations rather than a date-time stamp. This is because we don't know when the date time stamp was set (when they begun the source code change-set; just before committing; some time inbetween) different developers could use different approaches.
Names such as Migration_0000034.cs give you plenty of space.
At this point, I would stick with migrator.net. I like the promise of FluentMigrator, but it seems to not have any better active development than migrator.net (see the issues and pull requests that have languished on their github site).
There is also no easy way to do an ExecuteScalar(). I'd add it, but I don't want to create my own fork, and I see no reason that a pull request would actually land in the master. (Execute.WithConnection is an Action so it will fire on demand rather than when I need it to fire)
So for me, I'm heading back to migrator.net.
I'd like to respect the database as a "read-only" and never write to it. Is there a way to easily prevent syncdb from even bothering to check to update the database?
With Django 1.2 and the ability to have multiple databases, it'd like to be able to query a database for information. I'd never need to actually write to that database.
However, I'd be scared if syncdb ran and attempted to update that database (because I may not have a technically read-only account to that database). Mainly, I'd just like to use/abuse the Django ORM as a way to query that database.
UPDATE: Sorry, I need to be able to sync one of the databases in settings.py, just not this specific one.
Heh, I guess I'll answer my own question (RTFM!)...
http://docs.djangoproject.com/en/dev/topics/db/multi-db/#an-example
def allow_syncdb(self, db, model):
...
That's a definite start...
If you don't need syncdb, don't run it, simple as that. Updating the database is what it does, so if you don't need that, you shouldn't run it - it doesn't do anything else.
However if you're actually asking how to prevent syncdb from running at all, one possibility would be to define a 'dummy' syncdb command inside one of your apps. Follow the custom management command instructions but just put pass inside the command's handle method. Django will always find your version of the command first, making it a no-op.
This issue came up for me when working with read-only mirrors of Microsof SQL Server databases (uhhg). Since you can't selectively run syncdb on a single app or database. But you have to run syncdb when you first create a new Django project or install an new app that requires it (like south). What I did was to put my read-only database in its own Django app and then add an empty South migration to that app. That way syncdb thinks south is handling db setup for those apps, and south doesn't do anything to them!
manage.py schemamigration ap_with_read_only_database --empty initial_empty_migration_that_does_nothing
That leaves you free to manage the schema of that db outside of django.
From Ruby on Rails I've learned about a comfortable and very manageable way to do database migrations.
How would one handle such cases in Qt4? It does not need to be that elaborated as Rails' solution. I'd be fine with running SQL and code snippets to go up or down the version list of my data.
I googled a little bit but there seems to be no standard solution. There even seems to be no one who scratched his head before. It looks like everybody implements their own idea of how to do that.
Is there anyone interested or working on a standard solution and like to share efforts? Or maybe even has a working implementation? I'm thinking of the idea to have something similar like the information_schema table in Rails and let the application run a wizard to up- or downgrade the database (and maybe offer to do a backup/dump first), or as another option just run the migration without wizard, maybe just a handy progressbar.
QDjango may you like, it's mimic of django database ORM
Update:
Unfortunately QDjango doesn't support migrations. – #Riateche
I'm in the process of starting up a web site project. My plan is to roll out the site in a somewhat rudimentary form first and then add to the site functionality along the way.
I'm using Subsonic 3 for my DAL, and I'm expecting the database will go through multiple versions as the sites evolve. This means I'll need some kind of versioning and migration tools. I understand that Subsonic has built in migration possibilities, but I'm having difficulties grasping how to use these tools, in my scenario.
First there's the SimpleRepository model, where the Subsonic "automagically" handles the migrations as i develop my site. I can see how this works on my dev-machine, but I'm not sure how to handle deployments with this.
Would Subsonic run the necessary migrations on my live site as the appropriate methods are called?
Is there some way I can force all necessary migrations on a site while taking the site offline, when using the Simplerepository model? (Else I would expect random users to experience severe performance cuts, as the migration routines kick in)
Would I be better off using the ActiveRecord model, and then handling migrations with the Subsonic.Schema.Migrator? (I suspect so)
Do you know of any good resources explaining how to handle this situation with the migrator? (I read the doc, but I can't piece together how I would use this in practice)
Thanks for listening/replying.
Regards
Jesper Hauge
I would advise against ever running migrations against a live site. SubSonic's migrations are really there to make development simpler and should never be used against a live environment. To be honest even using SubSonic.Schema.Migrator you're still going to bump into the fact that refactoring databases is an incredibly hard problem. For example renaming a column in a table using management studio is trivial, but what happens in the background involves creating an entirely new table and migrating all the constraints, data etc. before renaming the new table.
The most effective way I've found for dealing with this is:
Script all database changes as you make them in your development environment (SQL Server Management Studio will do this for you) and add these scripts to your source control.
As part of deployment (obviously backup first) run the migration scripts and then deploy the updated application on success.
Whether you use ActiveRecord or SimpleRepository is then down to whether you want the extra features/complexity of ActiveRecord.
Hope this helps
i would use activerecord easy to use and any changes you just run the TT files, you would then just build or publish your slution and done ???? SVN will keep your multiple versions of the build stage so if you make a tit of it you just drop back a revision.