I am assinged for the task of Continuous deployment from development server to production server.
In my development server all the database objects will be created under the 'DBO' Schema. But in Production server based on every Tenants company list differenet SCHEMAS will be there.
for E.g in my development server if a tablename is created like
dbo.ABC
dbo.XYZ
And while i creating a tenant(Omkar---db) (Sarkur,Mathur--- schemas), the database objects will be like
Sarkur.ABC, sarkur.XYZ
Mathur.ABC, Mathur.XYZ
Now, i have to compare these two databases to check whether any changes in structure of the database objects, addition / deletion of database objects. If so that changes has tobe synchronized in the production database.
If anyone know that how to compare these two different schemas object, pls let me know..
1 option that I know is looking suitable
Flyway :
It is Easy to setup, simple to master. Flyway let's you regain control of your database migrations with pleasure and plain sql.
Solves only one problem and solves it well. Flyway migrates your database, so you don't have to worry about it anymore.
Made for continuous delivery. Let Flyway migrate your database on application startup. Releases have never been this easy.
Big Plus It's Open Source framework!
http://flywaydb.org/
Related
I have a Web application that I usually deployed using Web Deploy directly from Visual Studio (whatever branch I am currently using in VS - normally master). But now I'm introducing a second web app on Azure that will be built from the same repo but different branch. To make things simpler I will be configuring both Web apps on Azure to integrate directly with GitHub and associate them with specific branch.
I also added two additional web.config files: Web.Primary.config and Web.Secondary.config and configured app settings on Azure portal of each web app by adding additional value SCM_BUILD_ARGS and set them to
SCM_BUILD_ARGS=-p:PublishProfile=Primary // in primary web app
SCM_BUILD_ARGS=-p:PublishProfile=Secondary // in secondary web app
which I understand will transform correct config file with specific external services' configurations (DB connection, mail server, etc.).
Now the additional step that I would like to include in continuous deployment is run a set of SQL scripts that I have in my repo that I used to manually upgrade database during Web Deploy in VS. Individual scripts are actually doing specific database upgrade steps:
backup current tables - backup creates a set of Backup_OriginalTableName tables that are copied from existing ones and populated with existing data
drop whole DB model - all non-backup objects are being dropped from procedures, functions, types, views, tables...
create model - creates all tables, views and indices
create user types
create user functions
create stored procedures
restore data to new tables from backup tables - this step may occasionally break if we introduce new non-nullable columns to tables in the new model don't have defaults defined on them; I will somehow have to mitigate this problem by adding an additional script that will add missing columns to backup tables and give them some defaults, but that's a completely different issue.
I used to also have a set of batch files (BAT) in my VS solution that simply executed sqlcmd against specific database instance and executed these scripts in predefined order (as above). Hence I had batches:
Recreate Local.bat - this one used additional SQL scripts to not restore from backup but rather to recreate an empty DB with only lookup tables being populated and some default data for development purposes (like predefined test users)
Restore Local.bat - I used this script to simply restore database from backup tables discarding any invalid data I may have created while debugging/testing since last DB recreate/upgrade/restore
Upgrade Local.bat - upgrade local development DB executing scripts mentioned above
Upgrade Production.bat - upgrade production DB on Azure executing scripts mentioned above
So to support the whole deployment process I was now doing manually in VS I would now like to also execute these scripts against specific Azure SQL DB during continuous deployment. I suppose I should be running these right after code deployment because if that one fails, DB shouldn't be upgraded either.
I'm a bit confused where and how to do this? Can I configure this somewhere in Azure portal? I was looking for resources on the Web but I can't seem to find any relevant information how to do additional deployment steps to execute these scripts. I think this is some everyday scenario as it's hard to think of web apps not requiring databases these days.
Maybe it's just my process that is wrong for DB upgrade/deployment so let me also know if there is any other normal way that does DB upgrade/migration with continuous deployment on Azure... I may change my process to accommodate for this.
Note 1: I'm not using Entity Framework or any other full blown ORM. I'm rather using NPoco and all my DB logic is built in SPs that DAL is using.
Note 2: I'm aware of recently introduced staging capabilities of Azure, but my apps are on cheaper plan that doesn't support staging and I want to keep it this way as I may be introducing additional web apps along the way that will be using additional code branches and resources (DB, mail etc.)
It sounds to me like your db project is a good candidate for SSDT and inclusion in source control. You can create a MyDB.sqlproj that builds your db as a dacpac, and then you can use SqlPackage.exe Publish to accomplish your deployment to Azure.
We recently brought our databases under source control and follow a similar process to build and automatically deploy them (but not to a SQL Azure DB). We've found the source control, SSDT tooling support, and automated deployment options to be worth the effort of setting up and maintaining our project this way.
This SO question has some good notes for Azure deployment of a dacpac using SSDT:
How to publish DACPAC file to a SQL Server database project via SQLPackage.exe of SSDT?
Seeking some advice before I dig in. Let me explain the ask and my current process.
We currently have Dev teams who require refreshed data from the Prod databases placed into there DEV databases. There requirements change, sometimes they just need tables and other times they need different subsets of the below
Tables
Views
Stored Procedures
Users
Schemas
Currently the process is completely manual and is as outlined below
Disable Job responsible for replication of Prod DB (actually standby)
Highlight Prod DB and "Generate Scripts"
Select the Options that is required (see above e.g. tables views etc..)
Backup Dev DB (just in case)
sp_msforeachtable and drop each table from dev db
Execute the script that was generated from step 2 on dev db
Then use the import wizard to pull the data from the prod source
An additional sql script is often required to run post import on new Dev DB (scrubber)
Enable Job on prod for repl
The SQL Server instance hosts can change as can the databases so variables will need to be passed. The SQL Servers are 2008 on Windows 2008. The box that I host the script/instance on can be any version of windows and any version of SQL Server.
I'm hoping to automate this process, at first just for the SA teams (so could be ps or cli). Eventually (hopefully sooner than later) however present this in some type of ui to the dev teams so they can manage themselves.
I'd prefer if this all runs from a management box running SQL Server and not the SQL Server instances that host the databases. I'm not sure what options are available but I suspect SSIS could be used or PowerShell and SMO and I'm sure there are other crude ways.
I'd like this to be somewhat elegant so it's easily presentable to management. I'm comfortable with PowerShell and SQL but have no experience with SSIS.
Anyway looking for some suggestions.
EDIT:
So my Requirements have actually changed. I now need to scrub the data then backup then post to a share for dev. I'm nearing completion of my script, which is powershell using SMO. I'll give a brief description below and when I'm complete I'll post more details. Prod is over wan, as are the backups. We have have log shipping enabled to my site which is the data I have to work with. Steps are probably going to make some gringe but its necessary because by db is standby.
Create new database by looping through source DB using smo for files/file settings
Backup the newly create database with standby / readonly
Stop Job for log shipping to source db
Take source db offline
Take newly create db offline
Replace newly created DB files with source db files
Bring both DB's back on line
Start Job for log shipping to source db
Restore new created / newly copied file DB with recovery
Execute .sql to scrub new DB
Backup new DB
Copy to share for dev
So thats it, all sql related work is done through SMO. I'm pretty much done, I've built out functions for each step which all work, I just have to pull it all together.
Not pretty but does the job...damn that wan!
EDIT 2:
I ended up backing up, copying local, scrubbing, backing up again with compression than copying across that WAN overnight. I did this all through task scheduler / ps / SMO.
Thanks to all that offered advise
"Automating" what you describe sounds very ambitious - probably impractical given the complexity of the requirements. I would aim for "streamlining" a process rather than total automation.
I would use the SQL Server Data Tools (SSDT) for steps 2, 3, 5 & 6. It can generate and run alter scripts for you, based on a schema compare.
https://msdn.microsoft.com/en-au/data/tools.aspx
You typically start with SSDT by "Importing" the entire database schema into a Visual Studio Project. You can then use the "Schema Compare" tool to see what the differences are between your project and various database environments. Combined with source control (e.g. Visual Studio Online), this will give you a much better handle on what the changes are between environments, avoiding surprises.
I like the import wizard for 7 - I would save the generated SSIS packages and edit them to cover step 8 (if there is ever any reuse). SSDT's Schema Compare tool can tell you if there are any changes, implying you should regenerate the SSIS package for that table.
Needing to ship schema changes from Prod to Dev is presumably a red flag that developers/dbas are risking Prod-first changes. SSDT will help you monitor and quantify that.
I would agree with Mike Honey, there are some big red flags here that something isn't right.
To answer the specific question, if you need to have production data from all of your tables (even a subset) I would personally backup and restore the production data on whatever your schedule is (nightly?) - you should have a backup already so restoring it should be simple.
Once it has been restored you can delete off bits you don't want and if there is any personally identifiable information anonymize it!
Once you are happy with the data you can apply the latest dacpac from the SSDT project to make sure it has all of the developer changes.
This approach has two benefits, firstly it is simpler than copying all of the tables individually and secondly it tests your deployment process pretty effectively for when you do go to production.
That being said, re the big red flags! I would really question why you need production data, the way it should generally work is that each developer has a test database with little to no data in but unit tests that verify that everything works - if you need more data for performance testing then use something like the redgate sql data generator tool to generate realistic test data rather than full production data.
I have seen cases where it was thought production data was required but it actually wasn't - if it is too difficult to produce realistic test data, that in itself is a sign of some bad design etc - of course every environment it unique so maybe they do need it!
ed
We have a database in production that already has a good number of rows in the "user" table. Consider the following statement from the flyway website:
If you have an existing database that has not been filled by Flyway
this is the way to go:
Create an initial migration script that will recreate your current
state and give it a low version number.
Use flyway:init to create the metadata table and set this script as the current version.
I'd like to use flyway to manage my schema and various constants in the database, but I don't want V1__Base_version.sql to contain the account information for our current production users, especially considering it's stored in SCM. If I understand these instructions correctly though, I would need the ability to "recreate [my] current state" with V1__Base_version.sql.
So would creating an initial migration with just the schema and the constants work okay? Or do the databases on our workstations need to match those in production 100%?
You are correct. The init command is there to mark the production database with a version.
The initial migration you create (with the structure of your PROD db) is for the other environments. It will never run on PROD as its version will be below the init version. It will however align all environments so that subsequent migrations can be applied equally across all of them.
Building and maintaining a database that is then deplyed/developed further by many devs is something that goes on in software development all the time. We create a build script, and maintain further update scripts that get applied as the database grows over time. There are many ways to manage this, from manual updates to console apps/build scripts that help automate these processes.
Has anyone who has built/managed these processes moved over to a Source Control solution for database schema management? If so, what have they found the best solution to be? Are there any pitfalls that should be avoided?
Red Gate seems to be a big player in the MSSQL world and their DB source control looks very interesting:
http://www.red-gate.com/products/solutions_for_sql/database_version_control.htm
Although it does not look like it replaces the (default) data* management process, so it only replaces half the change management process from my pov.
(when I'm talking about data, I mean lookup values and that sort of thing, data that needs to be deployed by default or in a DR scenario)
We work in a .Net/MSSQL environment, but I'm sure the premise is the same across all languages.
Similar Questions
One or more of these existing questions might be helpful:
The best way to manage database changes
MySQL database change tracking
SQL Server database change workflow best practices
Verify database changes (version-control)
Transferring changes from a dev DB to a production DB
tracking changes made in database structure
Or a search for Database Change
I look after a data warehouse developed in-house by the bank where I work. This requires constant updating, and we have a team of 2-4 devs working on it.
We are fortunate because there is only the one instance of our "product", so we do not have to cater for deploying to multiple instances which may be at different versions.
We keep a creation script file for each object (table, view, index, stored procedure, trigger) in the database.
We avoid the use of ALTER TABLE whenever possible, preferring to rename a table, create the new one and migrate the data over. This means that we don't have to look through a history of ALTER scripts - we can always see the up to date version of every table by looking at its create script. The migration is performed by a separate migration script - this can be partly auto-generated.
Each time we do a release, we have a script which runs the create scripts / migration scripts in the appropriate order.
FYI: We use Visual SourceSafe (yuck!) for source code control.
I've been looking for a SQL Server source control tool - and came across a lot of premium versions that do the job - using SQL Server Management Studio as a plugin.
LiquiBase is a free one but i never quite got it working for my needs.
There is another free product out there though that works stand along from SSMS and scripts out objects and data to flat file.
These objects can then be pumped into a new SQL Server instance which will then re-create the database objects.
See gitSQL
Maybe you're asking for LiquiBase?
If I am running an organization that has 50 .net developers and all are using SQL Server, what is the best way to make a single SQL Server available to them?
Here is some of the concerns that I want to be careful about
Should I configure database users per project or per user? or both?
Should I provide single SQL Server instance?
Edit:
How can I track changes done by each user in database?
There are some more concerns but I think getting answer of these two will be a good starting point.
You should definitely configure a database per project, as only project specific items should be in that database. Also for backup and restore purposes a database per project will be a good idea.
Configuring databases for your developers depends on how many developers will actually develop for the database: create tables, views etc. Database developers should probably have some sort of test copy of the database they can use to develop their end of things, while the 'regular' developers work against a published copy of this database:
So a setup could be: 2 databases per project, one for db development and one for other development.
This way changes to the database scheme can first be developed and tested before pushed out to the rest of the developers.