I'm working with the Redgate SQL Comparison SDK at the moment and have got it set up to nicely diff 2 databases.
What I would like to do now is be able to diff an sqlproj from source control with a destination database.
I have tried pulling the sql files using the tfs/vsts SDKs but to no avail.
Is there any way to either build a sqlproj from source control into a dacpac and then pull this in as a source database, or to directly pull the sqlproj in as a source?
Edit:
My ultimate goal with this is to be able to basically compare the version of the database that is in source control with the the database running across many different environments and create delpoyment scripts for the diffs.
I have another couple of Redgate tools that accomplish this (SQL compare & SQL Source), but these can only be installed on 1 (maybe 2 max?) devices, the difficulty I have is in using Amazon RDS (where the endpoints are unreachable outside the VPC), I cannot connect one central install of these tools to all of my environments, and I can't buy an additional license for every environment. So I was trying to use the Comparison SDK to attempt to "roll my own" middle ground.
Many Thanks,
I also work at Redgate, please do email me via dlm#red-gate.com if you want to go into more details into your specific questions and I'll set up a call for us.
In general the process that Redgate recommends for what you're doing below would be to keep the canonised schema that you want all the database to have in version control. You could get that schema in either by each developer using the SQL Source Control product to bring their changes in from SSMS as they develop them, or by using the SQL Compare product to put a version in at the end of a sprint.
You can then use our DLM Automation tools in conjunction with a CI server to automate creating difference reports and sync scripts for your target servers. DLM Automation is a set of PowerShell commandlets and plugins for common CI servers like TeamCity, Jenkins, VSTS, TFS etc. You could also use the SQL Compare Pro command line.
If your whole team have our SQL Toolbelt product then you're licensed to install the DLM Automation tools as many times as you like on build/release agents, so you don't need additional licences per environment.
Are you doing this in the context of an automation build/ci system? You mention VSTS, so the way this normally works is that this would have already pulled the files from source control. Once the files are in the build agent's working folder, you should be able to point the SDK (or SQL Compare command line) at this. Bear in mind that a sql proj isn't an officially supported data source for Redgate tools, although it will work in many instances.
It would be good if you could edit your question and give some background on the higher level problem you're trying to solve just in case we (I work for Redgate) can recommend a more suited set of tools or techniques.
Related
My main problem is where does database go?
The project will be on SVN and is developed using asp.net mvc repository pattern. Where do I put the sql server database (mdf file)? If I put it in app_data, then my other team mates can check out the source and database and run it with the database being deployed in the vs instance.
The problem with this method are:
I cannot use SQL Management Studio with this database.
Most web hosts require me to deploy the database using their UI or SQL Management studio. Putting it in App Data will make no sense.
Connection String has to be edited each time I'm moving from testing locally to testing on the web host.
If I create the database using SQL Management studio, my problems are:
How do I keep this consistent with the source control (team mates have to re-script the db if the schema changes).
Connection string again. (I'd like to automatically use the string when on production server).
Is there a solution to all my problems above? Maybe some form of patterns of tools that I am missing?
Basically your two points are correct - unless you're working off a central database everyone will have to update their database when changes are made by someone else. If you're working off a central database you can also get into the issues where a database change is made (ie: a column dropped), and the corresponding source code isn't checked in. Then you're all dead in the water until the source code is checked in, or the database is rolled back. Using a central database also means developers have no control over when databsae schema changes are pushed to them.
We have the database installed on each developer's machine (especially good since we target different DBs, each developer has one of the supported databases giving us really good cross platform testing as we go).
Then there is the central 'development' database which the 'development' environment points to. It is build by continuous integration each checkin, and upon successful build/test it publishes to development.
Changes that developers make to the database schema on their local machine need to be checked into source control. They are database upgrade scripts that make the required changes to the database from version X to version Y. The database is versioned. When a customer upgrades, these database scripts are run on their database to bring it up from their current version to the required version they're installing.
These dbpatch files are stored in the following structure:
./dbpatches
./23
./common
./CONV-2345.dbpatch
./pgsql
./CONV-2323.dbpatch
./oracle
./CONV-2323.dbpatch
./mssql
./CONV-2323.dbpatch
In the above tree, version 23 has one common dbpatch that is run on any database (is ANSI SQL), and a specific dbpatch for the three databases that require vendor specific SQL.
We have a database update script that developers can run which runs any dbpatch that hasn't been run on their development machine yet (irrespective of version - since multiple dbpatches may be committed to source control during a single version's development).
Connection strings are maintained in NHibernate.config, however if present, NHibernate.User.config is used instead, however NHibernate.User.config is ignored from source control. Each developer has their own NHibernate.User.config, which points to their local database and sets the appropriate dialects etc.
When being pushed to development we have a NAnt script which does variable substitution in the config templates for us. This same script is used when going to staging as well as when doing packages for release. The NAnt script populates a templates config file with variable values from the environment's settings file.
Use management studio or Visual Studios server explorer. App_Data isn't used much "in the real world".
This is always a problem. Use a tool like SqlCompare from Redgate or the built in Database Compare tools of Visual Studio 2010.
Use Web.Config transformations to automatically update the connection string.
I'm not an expert by any means but here's what my partner and I did for our most recent ASP.NET MVC project:
Connection strings were always the same since we were both running SQL Server Express on our development machines, as were our staging and production servers. You can just use a dot instead of the computer name (eg. ".\SQLEXPRESS" or ".\SQL_Named_Instance").
Alternatively you could also use web.config transformations for deploying to different machines.
As far as the database itself, we just created a "Database Updates" folder in the SVN repository and added new SQL scripts when updates needed to be made. I always thought it was a good idea to have an organized collection of database change scripts anyway.
A common solution to this type of problem is to have the database versioning handled in code rather than storing the database itself in version control. The code is typically executed on app_start but could be triggered in other ways (build/deploy process). Then developers can run their own local databases or use a shared development database. The common term for this is called database migrations (migrating from one version to the next). Here is a stackoverflow question for .net tools/libraries to make this easier: https://stackoverflow.com/questions/8033/database-migration-library-for-net
This is the only way I would handle this on projects with multiple developers. I've used this successfully with teams of over 50 developers and it's worked great.
The Red Gate solution would be to use SQL Source Control, which integrates into SSMS. Its maintains a sql scripts folder structure in source control, which you can keep in the same folder/ respository that you keep your app code in.
http://www.red-gate.com/products/SQL_Source_Control/
Can we use GIT as the source control for sql management studio?
for Database source control within SSMS
Agent SVN - SCC Subversion Plug-in.
http://www.zeusedit.com/agent/ssms/ms_ssms.html
or
http://www.red-gate.com/products/sql-development/sql-source-control/
I’ve found out that ApexSQL has a tool that natively supports Git as a source control system. It comes as a SSMS add-in, and offers a wizard you can use to map database objects with the source control systems. To do that:
Download and install ApexSQL Source Control
Start SSMS and in Object explorer select the database you want to be linked to a source control
Right-click the database, and form the context menu, select the Link database to source control option, from the ApexSQL Source Control submenu
Select the source control system (in your case it is Git) and choose from 2 database development models - shared or dedicated. Shared model is recommended when you link a database on which multiple developers will work at the same time
Filter objects which you don’t want to track using source control: by schema, type or name by schema, type or name
Provide appropriate login information and repository string. For Git it is:
<protocol>://<hostname>:<portnumber>/<Git server name>/<repository> (see example below):
More detailed step-by-step instructions can be found in this article:
http://knowledgebase.apexsql.com/link-database-source-control-system-2/
You could add Git Bash as an External Tool (Tools | External Tools...):
Name: &Git (use & to specify a hot key)
Command: C:\windows\SysWOW64\cmd.exe (32-bit Command shell)
Arguments: /c ""C:\path\to\Git\bin\sh.exe" --login -i" Finding the path where Git is installed on a Windows system
Initial directory: $(ItemDir)
Try sql-source-control, a free and open source CLI used to get SQL into source control systems like Git.
https://www.npmjs.com/package/sql-source-control
Microsoft has released SQL Operations Studio. It's a free tool that runs on Windows, macOS, and Linux, for managing SQL Server, Azure SQL Database, and Azure SQL Data Warehouse; wherever they're running. It comes with native GIT support.
Since SSMS is (more or less) a custom version of Visual Studio, you might be able to use a solution intended for VS:
Using Git with Visual Studio
Alternatively, manage your DB source code in Visual Studio from the beginning, not in SSMS. That way (at least in VS2010) you have database projects, integrated deployment and unit testing etc. Or continue using SSMS and check in your code from an external tool when you're ready (not so convenient, of course).
But it depends on exactly what you're doing: SSMS is a DBA tool, VS is a developer tool. Either way, you should be using some form of source control, but it's not clear from the question exactly what sort of files you need to version.
Not yet but if you go to http://redgate.uservoice.com/forums/39019-sql-source-control/suggestions/537681-add-git-support and vote for redgate to add support for git it might get added in the next version. Yes I know it's a commercial product but some products are just good enough to pay for!
Red Gate SQL Source Control has been updated to include Git and Mercurial support (as well as Perforce and TFS). Be forewarned that their DVCS integration is not 100% feature complete in relation to their SVN product, as basic features such as viewing history of an object are not supported from within SSMS. This may be a deal-breaker if other Red Gate tools like SQL Compare are part of your workflow.
Our workaround was to install TortoiseGit or GitExtensions and navigate to the repository on disk to drill into the specifics. It works but is a bit clunky.
VersionSQL is an SSMS source control add-in I've designed to be lightweight and easy to use. In the Object Explorer panel, just right-click on a database or object and click Commit. VersionSQL will script it out to Git/SVN in a neatly organized folder structure.
Check it out at https://www.versionsql.com
There have been a number of answers around this question which you might want to look at but in a nutshell ....
The nature of version control is to store the original file and then the deltas, the difference between the original file committed and and subsequent changes (ok i have made that a bit simpler than it is perhaps) and then to manage the version number and give tools to extract any particular revision you need. This also then allows you to compare previous revisions and roll back (etc. etc.)
The RDBMS is made up of the schema and the and data these change and can change frequently in the case of the data so even if you did VCS what would you compare to do a restore and how would that help? Assuming you have a live system then reverting to an earlier revision would lose all the data stored in the interim and although i have never tried it i suspect could destroy the general integrity of the RDBMS.
The better solution is to use a backup application built for that RDBMS, MySQLdump say in the case of MySQL which makes a snapshot of the data and the structure of the data and store that in a safe place.
Dumps can be scheduled regularly and you can do things like master/slave databases (or other strategies) so you can backup live production databases on the fly without impacting on performance
We are starting a new BI project in our company. We have at least three developers working with database design and development. Our tools include Sparks EA, SQL Server 2008 EE and undetermined reporting tools. What kind of tools one can use for database version control in SQL Server? What kind of version control systems there are available for database development (managing versions of database schema, tables, stored procedures etc.)?
Visual Studio Team System includes the Database Edition features for database source control, deployment, schema comparison and more. You don't necessarily need a tool to do database versioning but if you do want a tool then VSTS is one option.
Often it's easier to export a set of scripts that would create your database from scratch, and put those in version control instead.
Then, if you want to upgrade a database to a "version" from source control, you could use a tool like SQL Compare or "Data Dude" to compare two databases (or creation scripts and a database) and apply the changes from one to the other.
We create all database changes in scripts, never use the GUI. Then save the scripts in folders that are part of our Subversion store just like any other piece of code. Since our configuration management people will not push to prod without a script, we have no rtrouble at all enforcing this. The beauty is that you have all the scripts you need for a particular change located together and those in another change not yet ready to go to prod are not pushed by accident.
We store all our SSIS packages as files as well and we store our configurations in sripts as well and push them the same way.
Should you want to do a comparison between databases to make sure nothing was missed, I highly recommend SQLCompare by Red-gate.
How do you manage your sql server database build/deploy/migrate for visual studio projects?
We have a product that includes a reasonable database part (~100 tables, ~500 procs/functions/views), so we need to be able to deploy new databases of the current version as well as upgrade older databases up to the current version. Currently we maintain separate scripts for creation of new databases and migration between versions. Clearly not ideal, but how is anyone else dealing with this?
This is complicated for us by having many customers who each have their own db instance, rather than say just having dev/test/live instances on our own web servers, but the processes around managing dev/test/live for others must be similar.
UPDATE: I'd prefer not to use any proprietary products like RedGate's (although I have always heard they're really good and will look into that as a solution).
We use Red-Gate SQLCompare and SQLDataCompare to handle this. The idea is simple. Both compare products let you maintain a complete image of the schema or data from selected tables (e.g. configuration tables) as scripts. You can then compare any database to the scripts and get a change script. We keep the scripts in our Mercurial source control and tag (label) each release. Support can then go get the script for any version and use the Redgate tools to either create from scratch or upgrade.
Redgate also has an API product that allows you to do the compare function from your code. For example, this would allow you to have an automatic upgrade function in your installer or in the product itself. We often use this for our hosted web apps as it allows us to more fully automate the rollout process. In our case, we have an MSBuild task that support can execute to do an automatic rollout and upgrade. If you distribute to third-parties, you have to pay a small additional license fee for each distribution that includes the API.
Redgate also has a tool that automatically packages a database install or upgrade. We don't use that one as we have found that the compare against scripts for a version gives us more flexibility.
The Redgate tools also help us in development because they make it trivial to source control the schema and configuration data in a very granular way (each database object can be placed in its own file)
The question was asked before SSDT projects appeared, but that's definitely the way I'd go nowadays, along with hand-crafting migration scripts for structural db changes where there is data that would be affected.
There's also the MS VSTS method (2008 description here), anyone got a good article on doing this with 2010 and the pros/cons of using these tools?
I am trying to get our company's database objects stored in source control. We are going to be stuck using Aldon LM for our source control program. What is the best way to deploy scripts that we create to our SQL Server environments automatically from source control?
For our .Net web applications, we have configured Aldon to push our code to the web server and run MSBuild.
We have programs (ant hill pro for one) which go through the source control system, and retrieve all the necessary sql scripts and compile them into one file. We have a branch for each release, so only the ones for that branch get get pulled. I think cruise control can do the same thing, but I don't know how that is done.
We then can execute all the updates and once and don't have to worry about missing one.
Open a Database project in your solution to manage all SQL code such as stored procedures. This has a few advantages:
Your SQL can be under version control just as the rest of your conde
You can do easy find/replace file operations on multiple procedures
You can keep your SQL in a meaningful and systematic folder hierarchy
Install VS Team System Database Edition, now included (available to add to) Developer Edition.
This has tools to work out which scripts to apply to a database to update it to match the reference database and will (providing your SCM integrates with VS) also handle tracking all those scripts and there status within the project.