I recently had the unfortunate experience of having the size of a field changed in a table that was being used in SSIS packages. The developer who wrote the packages had since retired, and had grouped them all in one VS BI project. She developed them solely on her local PC and moved them to a shared drive when she left.
Anyone with any knowledge of Protection Levels in SSIS knows what happened next. She saved them with the default EncryptSensitiveWithUserKey option, so as a result, I couldn't modify the package due to the fact that I wasn't her and I wasn't on her machine. Her AD account has long since been deactivated, and her machine may be checking people out at a grocery store now for all I know.
I had to recreate the packages from scratch. Fortunately, the protection level prevents you from saving or building, not from looking at things in Design, so I was able to replicate what she did, but it was a long, tedious process that took the better part of a full day to complete.
My question is: We use SourceSafe to maintain our projects, so that's where the new SSIS project will be going. Given the following:
We each have our own PCs with working folders that sync to SourceSafe
We do not have the option of saving the project on the database server itself; we can only deploy the pacakges.
Software details: MS Visual Studio 2008, VSS 8.0, MS SQL Server 2008 R2
What would be the best security option to configure our project with? I immediately see that DontSaveSensitive would be the logical approach, but I don't know where the passwords would then be supplied from. I would think a config file, but I don't know how to set that up for an SSIS deployment.
Thanks!
Use VSS as the source control provider for Visual Studio and add the SSIS project just like any other source project. After installing VSS, go to Tools\Options\Source Control and set VSS as the provider.
I've been using this setup for several months. There's no difference, as far as I can tell, between any of my C# projects and my SSIS projects in relation to VSS.
Related
I'm working with the Redgate SQL Comparison SDK at the moment and have got it set up to nicely diff 2 databases.
What I would like to do now is be able to diff an sqlproj from source control with a destination database.
I have tried pulling the sql files using the tfs/vsts SDKs but to no avail.
Is there any way to either build a sqlproj from source control into a dacpac and then pull this in as a source database, or to directly pull the sqlproj in as a source?
Edit:
My ultimate goal with this is to be able to basically compare the version of the database that is in source control with the the database running across many different environments and create delpoyment scripts for the diffs.
I have another couple of Redgate tools that accomplish this (SQL compare & SQL Source), but these can only be installed on 1 (maybe 2 max?) devices, the difficulty I have is in using Amazon RDS (where the endpoints are unreachable outside the VPC), I cannot connect one central install of these tools to all of my environments, and I can't buy an additional license for every environment. So I was trying to use the Comparison SDK to attempt to "roll my own" middle ground.
Many Thanks,
I also work at Redgate, please do email me via dlm#red-gate.com if you want to go into more details into your specific questions and I'll set up a call for us.
In general the process that Redgate recommends for what you're doing below would be to keep the canonised schema that you want all the database to have in version control. You could get that schema in either by each developer using the SQL Source Control product to bring their changes in from SSMS as they develop them, or by using the SQL Compare product to put a version in at the end of a sprint.
You can then use our DLM Automation tools in conjunction with a CI server to automate creating difference reports and sync scripts for your target servers. DLM Automation is a set of PowerShell commandlets and plugins for common CI servers like TeamCity, Jenkins, VSTS, TFS etc. You could also use the SQL Compare Pro command line.
If your whole team have our SQL Toolbelt product then you're licensed to install the DLM Automation tools as many times as you like on build/release agents, so you don't need additional licences per environment.
Are you doing this in the context of an automation build/ci system? You mention VSTS, so the way this normally works is that this would have already pulled the files from source control. Once the files are in the build agent's working folder, you should be able to point the SDK (or SQL Compare command line) at this. Bear in mind that a sql proj isn't an officially supported data source for Redgate tools, although it will work in many instances.
It would be good if you could edit your question and give some background on the higher level problem you're trying to solve just in case we (I work for Redgate) can recommend a more suited set of tools or techniques.
We originally dismissed using database projects in conjunction with TFS as our solution for our deployment and soucecontrol needs. However, in the interest of thoroughness, I'm exploring and prototyping it.
I've set up my database project (with add to source control checked). I've checked in the changes. Now, where do you develop from?
I've tried ...
connecting to the remote development server to make changes
syncing schema to (localdb)\Projects and making changes there
directly in the Source Control Explorer
With option 1 and 2 I don't see an automated way to add code to source control. Am I suppose to be working in the Source Control Explorer? (this seems a little silly)... Is there a way to commit the entire solution to source control? My apologies in advance, I'm a database developer and this concept of a "solution" is very foreign to me.
Also there were a lot of chatter about Visual Studios doing a lot of ugly things in the back ground that turned a lot of development shops off of database projects. Can someone share your experiences with me? Some of the pitfalls and gotchas.
And yes, we have looked at Redgate SourceControl (very nice tool).
Generally people do one of two things:
Develop in Visual Studio, via the Solution Explorer. Just open the project like you would any other project, add tables, indexes, etc. You even get the same GUI for editing DB objects as you get in SSMS. All changes will automatically be added to TFS Pending changes (just like any other code change), and can be checked in when you're ready.
Deploy the latest DB (using Publish in VS) to any SQL Server, make your changes in SSMS, then do a Schema Compare in Visual Studio to bring your changes back into your DB project so they can be checked into TFS.
I've been using DB projects for many years and I LOVE them! Every developer I've introduced them to, refuses to develop without them from that point on.
I'm going to explain you briefly how we use DB projects with TFS.
We basically have one DB already done and if we require any changes or new tables we create them or alter them directly in SQL Server (each developer has its own dev SQL Server).
Then in VS from the SQL Server Object Explorer we drag the tables we want into the DB project so when we check in the changes, every user in TFS would be able to get them and then publish that project that will generate and execute a script into the DB.
This is the way we use to develop when we need to add specific tables or records to the DB so we don't have to send emails with scripts or have them stored in an specific location (even with source control). This way we can get latest version of the project and publish it to ensure we have the latest DB version although it requires the user (who made the changes) to add them to the DB project.
Other way could be to do all the changes (and can be done without any problem) directly in the DB project and then publish it. That one would be a more right way to do it so you do all the changes directly in a source controlled project, but as you know, is always more comfortable to work directly through the SQLMS.
Hope this helps somehow.
We use the SSDT tools and have implemented the SQL Server Database Project Type to develop our databases:
http://www.techrepublic.com/blog/data-center/auto-deploy-and-version-your-sql-server-database-with-ssdt/
The definition of database objects and peripheral SQL Code (e.g. functions, sprocs, triggers etc) sit within the Visual Studio project and all changes are managed through VS. The interface is very similar to SSMS and, at this point doesn't cause any issues.
The benefits of this approach for us are as follows:
An existing SQL database can be imported into the SQL Server Project and managed through Visual Studio.
SQL object definitions & code can managed through the same version control system as the rest of the application code.
SQL Code can be checked for errors within Visual Studio in much the same way as you'd check your C# / VB for compilation / reference errors.
You can compare database schema's (within Visual Studio) between environments and easily identify key changes that you need to be aware of.
The SQL project can be compiled into a DACPAC file for automating deployment to different servers using a CI / Build Server (using the sqlpackage.exe utility without any custom scripts or code).
In essence developers can have a local version of the database to work on but would manage any changes through VS, then publish the changes to their local database. Once the changes are complete, the changes are committed to your version control system and then built centrally & automatically through a CI / Build server to ensure that all changes integrate and play nicely in much the same way that your other code is.
Hope that helps :)
My main problem is where does database go?
The project will be on SVN and is developed using asp.net mvc repository pattern. Where do I put the sql server database (mdf file)? If I put it in app_data, then my other team mates can check out the source and database and run it with the database being deployed in the vs instance.
The problem with this method are:
I cannot use SQL Management Studio with this database.
Most web hosts require me to deploy the database using their UI or SQL Management studio. Putting it in App Data will make no sense.
Connection String has to be edited each time I'm moving from testing locally to testing on the web host.
If I create the database using SQL Management studio, my problems are:
How do I keep this consistent with the source control (team mates have to re-script the db if the schema changes).
Connection string again. (I'd like to automatically use the string when on production server).
Is there a solution to all my problems above? Maybe some form of patterns of tools that I am missing?
Basically your two points are correct - unless you're working off a central database everyone will have to update their database when changes are made by someone else. If you're working off a central database you can also get into the issues where a database change is made (ie: a column dropped), and the corresponding source code isn't checked in. Then you're all dead in the water until the source code is checked in, or the database is rolled back. Using a central database also means developers have no control over when databsae schema changes are pushed to them.
We have the database installed on each developer's machine (especially good since we target different DBs, each developer has one of the supported databases giving us really good cross platform testing as we go).
Then there is the central 'development' database which the 'development' environment points to. It is build by continuous integration each checkin, and upon successful build/test it publishes to development.
Changes that developers make to the database schema on their local machine need to be checked into source control. They are database upgrade scripts that make the required changes to the database from version X to version Y. The database is versioned. When a customer upgrades, these database scripts are run on their database to bring it up from their current version to the required version they're installing.
These dbpatch files are stored in the following structure:
./dbpatches
./23
./common
./CONV-2345.dbpatch
./pgsql
./CONV-2323.dbpatch
./oracle
./CONV-2323.dbpatch
./mssql
./CONV-2323.dbpatch
In the above tree, version 23 has one common dbpatch that is run on any database (is ANSI SQL), and a specific dbpatch for the three databases that require vendor specific SQL.
We have a database update script that developers can run which runs any dbpatch that hasn't been run on their development machine yet (irrespective of version - since multiple dbpatches may be committed to source control during a single version's development).
Connection strings are maintained in NHibernate.config, however if present, NHibernate.User.config is used instead, however NHibernate.User.config is ignored from source control. Each developer has their own NHibernate.User.config, which points to their local database and sets the appropriate dialects etc.
When being pushed to development we have a NAnt script which does variable substitution in the config templates for us. This same script is used when going to staging as well as when doing packages for release. The NAnt script populates a templates config file with variable values from the environment's settings file.
Use management studio or Visual Studios server explorer. App_Data isn't used much "in the real world".
This is always a problem. Use a tool like SqlCompare from Redgate or the built in Database Compare tools of Visual Studio 2010.
Use Web.Config transformations to automatically update the connection string.
I'm not an expert by any means but here's what my partner and I did for our most recent ASP.NET MVC project:
Connection strings were always the same since we were both running SQL Server Express on our development machines, as were our staging and production servers. You can just use a dot instead of the computer name (eg. ".\SQLEXPRESS" or ".\SQL_Named_Instance").
Alternatively you could also use web.config transformations for deploying to different machines.
As far as the database itself, we just created a "Database Updates" folder in the SVN repository and added new SQL scripts when updates needed to be made. I always thought it was a good idea to have an organized collection of database change scripts anyway.
A common solution to this type of problem is to have the database versioning handled in code rather than storing the database itself in version control. The code is typically executed on app_start but could be triggered in other ways (build/deploy process). Then developers can run their own local databases or use a shared development database. The common term for this is called database migrations (migrating from one version to the next). Here is a stackoverflow question for .net tools/libraries to make this easier: https://stackoverflow.com/questions/8033/database-migration-library-for-net
This is the only way I would handle this on projects with multiple developers. I've used this successfully with teams of over 50 developers and it's worked great.
The Red Gate solution would be to use SQL Source Control, which integrates into SSMS. Its maintains a sql scripts folder structure in source control, which you can keep in the same folder/ respository that you keep your app code in.
http://www.red-gate.com/products/SQL_Source_Control/
What is the best way to version database objects (Trigger, SPs, and other elements) in VSS in a similar fashion to the way that we store source code in VSS and access it in Microsoft Visual Studio?
We would like to check database elements in and out in VSS or a similar tool so that we can store database objects in a central location, and also so that we can have versioning of database elements, for example, version history of stored procedures.
We currently use SQL Server 2005 as our database engine.
If you are using VS, the easiest way to control your source objects is to create a Database project using the "Database" project template in Visual Studio.
The entire database project can be associated with source control (VSS in your case) and then all your DB object scripts are versioned.
A very important point to note is to make sure that Developers get out of their old habits of directly updating / changing objects in the DB because doing this will not stop them from doing so. An easy way out of that dilemma is to ensure that the DB project is built and deployed periodically(Continuous integration) just like your code is.
This will ensure that if changes are directly done to DB, they will be lost and hence automatically inculcate the behavioural change in developers.
Refer link for a step by step tutorial as to how to get started using a DB project.
Screen shots are for VS2010 but DB projects have been around since VS 2005 and more or less on the same lines. Very easy to use.
If you're willing to use Subversion or TFS, and are using SSMS to make your database changes, Red Gate's SQL Source Control maybe a tool that would work for you.
http://www.red-gate.com/products/SQL_Source_Control/
[Edit]
We've now added VSS and SourceGear Vault support. Try the early access build:
http://www.red-gate.com/MessageBoard/viewtopic.php?t=12265
In the SO podcast episode 54 Jeff talked about using Visual Studio to save all the database objects to individual files. This sounded like just what my team needed to better implement database schema changes into TFS and I told my lead about it. He thinks it's a great idea to.
Unfortunately, so far I've had no luck getting this to work for me. One of my problems is that I don't have SQL Server installed on my local box (dept policy). I'm obviously doing something wrong.
Can someone give me a rundown of the steps or provide a decent link?
Thanks!
1) Create a connection to the database in Server Explorer.
2) Right-click on the connection and select Publish to provider...
3) Next, Script to file, Next, Types of data to publish should be Schema (unless you need some initial data), finish.
4) Add script to the project and check into source control.
He was referring to a project type in VS that supports managing databases, including (if you set it up correctly), versioning your database publishes.
Check out this article on Database Projects in VS
You want to use the GDR 2. (Sometimes called Data Dude)
This allows for a completely offline solution. You don't need to have SQL Server installed on your machine for this to work. (In fact the GDR is the first version that does not care if you have SQL Server installed.)
I use the GDR for my db (a team of 3 devs and 2 testers) and it works GREAT!
Here is a link for the GDR 2 release:
http://blogs.msdn.com/vstsdb/archive/2009/04/21/microsoft-visual-studio-team-system-2008-database-edition-gdr-r2.aspx
And this is a link to the actual bits:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=bb3ad767-5f69-4db9-b1c9-8f55759846ed
The basic use is to import from an existing DB and server. (You will probably want a server project and 1 or more DB Projects)
You can then make your changes off line. When you are ready to send you changes back to the DB you can delploy (make sure you set up your deploy options first as I think the default is to drop the db and re-deploy). You can also do a Data->Schema Compare in Visual Studio and comapre your project to your database then get a script of changes from the diff output.
It takes a bit of work, but it really allows great source control and is easy once you get the hang of it. (I have my db auto deploy in my night time build twice a week.)
If you're importing an existing database schema, its important to get the right database project. You should likely be using "Database Projects\SQL Server 2005 Wizard".
Visual Studio requires a database connection it can use to create temporary copies of the databases it is working with. It sounds like this is the issue. Do you have SQLExpress on your local box? Use that as the deployment target (server name would be .\sqlexpress if . does not work).
As an alternative, grab update GDR 2 for Visual Studio. Allegedly, it allows one to work with database projects without using a local instance of sql server to deploy temporary working copies of the database.