We are using Visual Studio 2012, and have a database project that deploys to an instance of localDB on each developers machine. All development work uses the localDB instance, until we are ready to checkin code and publish to our development SQL Server. My problem is that after a certain amount of time, all the data/primary key values in each person's localDB is becoming outdated. I would like to periodically "sync" everyone's instance of localDB, so that we are all using the same data for various lookup/dependency tables/userIds/etc.
Is there an easy way to include a data script in the database project that would push data to each instance automatically, or perhaps to grab a backup of our main dev DB and use it to copy over each developers localDB? I just need a way to periodically update everyone to make sure we are all working off of the same data.
There's no easy way to write these scripts, but I highly recommend that you write post-deploy scripts and include them in your project so when the project is pushed, everyone gets those changes. More importantly, they should be there so the changes will be released when you publish your project. You'll need to write them in such a manner that they can be repeated (if not exists ... insert type commands).
You can definitely backup and restore the shared database regularly as long as you know you've got the local changes checked in. Nothing is worse than losing a bunch of work because you've restored over your database.
You can use the "Data Compare" option under the SQL menu as well, but you want to understand what you're doing before just running it or you'll likely experience data loss.
Related
Seeking some advice before I dig in. Let me explain the ask and my current process.
We currently have Dev teams who require refreshed data from the Prod databases placed into there DEV databases. There requirements change, sometimes they just need tables and other times they need different subsets of the below
Tables
Views
Stored Procedures
Users
Schemas
Currently the process is completely manual and is as outlined below
Disable Job responsible for replication of Prod DB (actually standby)
Highlight Prod DB and "Generate Scripts"
Select the Options that is required (see above e.g. tables views etc..)
Backup Dev DB (just in case)
sp_msforeachtable and drop each table from dev db
Execute the script that was generated from step 2 on dev db
Then use the import wizard to pull the data from the prod source
An additional sql script is often required to run post import on new Dev DB (scrubber)
Enable Job on prod for repl
The SQL Server instance hosts can change as can the databases so variables will need to be passed. The SQL Servers are 2008 on Windows 2008. The box that I host the script/instance on can be any version of windows and any version of SQL Server.
I'm hoping to automate this process, at first just for the SA teams (so could be ps or cli). Eventually (hopefully sooner than later) however present this in some type of ui to the dev teams so they can manage themselves.
I'd prefer if this all runs from a management box running SQL Server and not the SQL Server instances that host the databases. I'm not sure what options are available but I suspect SSIS could be used or PowerShell and SMO and I'm sure there are other crude ways.
I'd like this to be somewhat elegant so it's easily presentable to management. I'm comfortable with PowerShell and SQL but have no experience with SSIS.
Anyway looking for some suggestions.
EDIT:
So my Requirements have actually changed. I now need to scrub the data then backup then post to a share for dev. I'm nearing completion of my script, which is powershell using SMO. I'll give a brief description below and when I'm complete I'll post more details. Prod is over wan, as are the backups. We have have log shipping enabled to my site which is the data I have to work with. Steps are probably going to make some gringe but its necessary because by db is standby.
Create new database by looping through source DB using smo for files/file settings
Backup the newly create database with standby / readonly
Stop Job for log shipping to source db
Take source db offline
Take newly create db offline
Replace newly created DB files with source db files
Bring both DB's back on line
Start Job for log shipping to source db
Restore new created / newly copied file DB with recovery
Execute .sql to scrub new DB
Backup new DB
Copy to share for dev
So thats it, all sql related work is done through SMO. I'm pretty much done, I've built out functions for each step which all work, I just have to pull it all together.
Not pretty but does the job...damn that wan!
EDIT 2:
I ended up backing up, copying local, scrubbing, backing up again with compression than copying across that WAN overnight. I did this all through task scheduler / ps / SMO.
Thanks to all that offered advise
"Automating" what you describe sounds very ambitious - probably impractical given the complexity of the requirements. I would aim for "streamlining" a process rather than total automation.
I would use the SQL Server Data Tools (SSDT) for steps 2, 3, 5 & 6. It can generate and run alter scripts for you, based on a schema compare.
https://msdn.microsoft.com/en-au/data/tools.aspx
You typically start with SSDT by "Importing" the entire database schema into a Visual Studio Project. You can then use the "Schema Compare" tool to see what the differences are between your project and various database environments. Combined with source control (e.g. Visual Studio Online), this will give you a much better handle on what the changes are between environments, avoiding surprises.
I like the import wizard for 7 - I would save the generated SSIS packages and edit them to cover step 8 (if there is ever any reuse). SSDT's Schema Compare tool can tell you if there are any changes, implying you should regenerate the SSIS package for that table.
Needing to ship schema changes from Prod to Dev is presumably a red flag that developers/dbas are risking Prod-first changes. SSDT will help you monitor and quantify that.
I would agree with Mike Honey, there are some big red flags here that something isn't right.
To answer the specific question, if you need to have production data from all of your tables (even a subset) I would personally backup and restore the production data on whatever your schedule is (nightly?) - you should have a backup already so restoring it should be simple.
Once it has been restored you can delete off bits you don't want and if there is any personally identifiable information anonymize it!
Once you are happy with the data you can apply the latest dacpac from the SSDT project to make sure it has all of the developer changes.
This approach has two benefits, firstly it is simpler than copying all of the tables individually and secondly it tests your deployment process pretty effectively for when you do go to production.
That being said, re the big red flags! I would really question why you need production data, the way it should generally work is that each developer has a test database with little to no data in but unit tests that verify that everything works - if you need more data for performance testing then use something like the redgate sql data generator tool to generate realistic test data rather than full production data.
I have seen cases where it was thought production data was required but it actually wasn't - if it is too difficult to produce realistic test data, that in itself is a sign of some bad design etc - of course every environment it unique so maybe they do need it!
ed
I'm developing a WPF desktop application, and the database currently resides in SQL Server Express 2012. The app is undergoing acceptance testing, and periodically I have to backup my database and restore it on the tester's PC. This is setting alarm bells ringing that distribution is going to be a pain when we come to start working with customers - I don't like the idea of them having to use SSMS.
I wondered if there was any way to distribute the mdf file and have my application connect to that "directly"? I've seen examples of connection strings that specify an MDF file/path, and wondered if it was really that simple or does it still need attaching in SSMS first? (I'm guessing so, otherwise how would you configure things like users and roles?).
Most applications I've encountered run SQL CREATE scripts during the installation process. I'd prefer that way, because when you will have to change your schema some time in the future (and you will), you will have to do something similar, because you cannot exchange the old MDF (with your users' data) against an empty updated one.
I have been doing some research on the correct procedure to follow when working with both a development and live production database. The best article i have found was this one: Strategies for Database Development and Deployment, but i can not accept the idea that i have to maintain a Word document, manually, of every change i make to the DB. That seems ridiculous to me...
I use SQL Server Management Studio to manage my SQL databases both in dev and in prod. Is there a way to deploy latest changes to production WITHOUT destroying tables and data. Can someone please point me to a good procedural article on how this is done in SSMS.
Thanks
It is irresponsible to make changes to a database design without creating change scripts that are put into source control.
However, if you are already in this curcumstance, I suggest buying red_gate's SQLCompare. It will look at the two databases and script the differnces. You still can't run this willy-nilly though - sometimes you have made changes to the dev database that are not yet part of the curent version being pushed to prod and SQLCompare has no way to know this. It is far better to create the scripts as you go (Using alter table when the table currently exists so as not to disrupt existing data) and keep them in source control with the rest of code that you will be pushing at the same time.
The only right way to do it in prodiction - with or without Management Studio - prepare, check, test and run the scripts manually.
WITH FRESH BACKUP!
A common strategy is to keep an ordered set of change scripts, e.g. prefixed with date or database version, which can easily be tested on the development database by starting with a fresh backup from production. The change scripts can often be generated from SQL Server Management Studio when making changes, or could be crafted manually in case of more complex changes.
Another suggestion would be to version control the database definitions (tables, procedures etc). This can be easily achieved by using SQL Server Management Studio to generate create scripts for all objects after each update. This way you can easily compare changes made over time, or between different environments.
I have been working on a project and gotten it through the first stage. However, the requirments ended up changing and I have to add new tables and redo some of the foriegn key references in the DB.
The problem I have is my lack of knowledge of dealing with doing this kind of change to a staging then production database once I get the development done on dev database.
What are some strategies for migrating database schema changes and maintaining data in the database?
About as far as my knowledge is on doing this is open up Sql Server Management Studio and starting adding tables manually. I know this is probably a bad way to do it so looking for how to do it properly while realizing I probably started out wrong.
For maintaining schema changes you can use ApexSQL Diff, a SQL Server and SQL Azure schema comparison and synchronization tool, and for maintaining data in the database you can use ApexSQL Data Diff, a SQL Server and SQL Azure data comparison and synchronization tool.
Hope this helps
Disclaimer: I work for ApexSQL as a Support Engineer
You have to have something called as a "KIT". Obviously, if you are maintaining some kind of a source control, all the scripts for the changes that you do in the development environments should be maintained in the source control configuration tool.
Once you are done with all the scripts/changes that you deem certified to move to next higher environment. Prepare the kit with having all these scripts in folders (ideally categorized as Procedures, Tables, Functions, Bootstraps) And then have a batch files that could execute these scripts in the kit in a particular order using OSQL command line utility.
Have separate batch files for UAT/ Staging/ production so that you can just double click on the batch file to execute the kit in the appropriate server. Check for OSQL options.
This way all your environments are in sync!
I typically use something like the SQL Server Publishing Wizard to produce SQL scripts of the changes. That is a rather simple and easy approach. The major downside with that tool is that the produced will drop and recreate tables that are not changed but used by procedures that have changed (and I can't understand why), so there is some manual labour involved in going through the script and remove things that don't need to be there.
Note that you don't need to download and install this tool; you can launch it from within Visual Studio. Right-click on a connection in the Server Explorer and select "Publish to Provider" in the context menu.
Red Gate SQL Compare and SQL Data Compare all the way. Since my company bought it, it saved me tons of time staging our databases from DEV to TEST to ACCEPTANCE to PRODUCTION.
And you can have it synchronize with a scripts folder too for easy integration in a source control system.
http://www.red-gate.com
You might want to check out a tool like Liquibase: http://liquibase.org/
You can use visual studio 2015. Go to Tools=> SQL server => New Schema comparison
step 1) Select source and target Database.
Click on Compare option.
step 2) once comparison completed, you can click on icon Generate Script(Shift+alt+G)
this will generate Commit script.
step 3) To generate rollback script for database changes just swap database from step 1
There are some tools available to help you with that.
If you have Visual Studio Team edition, check database projects (aka DataDude aka Visual Studio Team for Database Professionals) See here and here
It allows you to generate a model from the dev/integration database and then (for many, but not all cases) automatically create scripts which update your prod database with the changes you made to dev/integration.
For VS 2008, make sure you get the GDR2 patches.
We have found the best way to push changes is to treat databases changes like code. All changes are in scripts, they are in source control and they are part of a version. Nothing is ever under any circumstances pushed to prod that is not scripted and in source control. That way you don't accidentally push changes that are in dev, but not yet ready to be pushed to prod. Further you can restore prod data to the dev box and rerun all the scripts not yet pushed and you have fresh data and all the dev work preserved. This also works great when you have lookup values to tables that are chaging that you don;t want pushed to prod until other things move as well. Script the insert and put it with the rest of the code for the version.
It's nice to use those tools to do a compare to see if something is missed in the scripts, but I would NEVER rely on them alone. Far too much risk of pushing something "not yet ready for prime time" to prod.
A good database design tool (such as Sybase Powerdesigner) will allow you to create the design changes to the data model, then generate the code to implement those changes. You can then store and run the code as you choose. This tool should also be able to do reverse engineering when you inherit a database you didn't build.
Finding all the changes between development and production is often difficult even in an organized, well-documented environment. Idera has a tool for SQL Server which will detect structural differences between your development and production database and another tool which detects changes in the data. In fact, I often use these to go the other direction and sync development with production to start a new project.
Say I have a website and a database of that website hosted locally on my computer (for development) and another database hosted (for production)...ie first I do the changes on the dev db and then I do the changes to the prod DB.
What is the best way to transfer the changes that I did on the local database to the hosted database?
If it matters, I am using MS Sql Server (2008)
The correct way to do this with Visual Studio and SQL Server is to add a Database Project to the web app solution. The database project should have SQL files that can recreate the entire database completely on a new server along with all the necessary tables, procedures users and roles.
That way, they are included in the source control for all the rest of the code as well.
There is a Changes sub-folder in the Database Project where I put the SQL files that apply any new alterations or additions to the database for subsequent versions.
The SQL in the files should be written with proper "if exists" blocks such that it can be run safely multiple times on an already updated database without error.
As a rule, you should never make your changes directly in the database - instead modify the SQL script in the project and apply it to the database to make sure your source code (the SQL files) is always up to date.
We do this in the (Ruby on) Rails world by writing "migrations," which capture the changes you make to the DB structure at each point. These are run with a migration tool (a task for rake), which also writes to a DB table so it knows whether a particular migration has been run or not.
You could make a structure like this for your dev platform (.Net?), but I think that in other answers to this question people will suggest available tools for handling database versioning in your development platform, or perhaps for your specific DB.
I don't know any of these, but check out this list. I see a lot of pay things out there, but there must be something free. Also check this out.
I migrate changes via change scripts written by developers when they have tested/verified their changes. (The exception being moving large data.) All scripts are stored in a Source control system. and can be verified by DBAs.
It is manual, sometime time consuming but effective, safe and controled process.
Databases are too vital to copy from dev.
There are tools to help create/verify these scripts.
See http://www.red-gate.com/
I have used their tools to compare 2 databases to create scripts.
Brian
If the changes are small, I sometimes make them by hand. For larger changes, I use Red Gate's SQL Compare to generate change scripts. These are hand-verified and run in the QA environment first to make sure they don't break anything. For large changes, we run a special backup prior to making the change both in QA and in production.
We used to use the approach provided by Ron. It makes sense for a big project with dedicated team of DBAs. But if you do not have a dedicated developers who write code only for DB this approach is time and resource expensive.
The approach to use RedGate DB compare is also not good. You still have a do a lot of manual work you can skip some step by mistake.
It needs something better. This is was the reason why we built the "Agile DB Recreation/Import/Reverse/Export tool"
The tool is free.
Advantages: your developers use any prefered tools to develop DEV DB. Then they run the DB RIRE and it makes reverseengeniring DB (tables, views, stor proc, etc) and export data into XML files. XML files you can keep in the any code repository system.
And the second step is to run DB RIRE one more time to generate difference scripts between structure and data in XML files and in Production DB.
Of course you can make as much iterations as you need.