We have been planning migrating from shared development database to local database for each developer. Installing the database, schema and initial data should be automated and platform independent, and each developer would point his application server and DBMS to this local database instead of a shared one, to freely experiment with the schema without fear of breaking others work. Database in question is Oracle.
Database stuff is of course source controlled, and each developer should easily upgrade to the latest version. Ideal is, each developer runs some kind of platform independent container, which on boot is configured to mirror the QA database by fetching the latest schema and scripts from the source control. It should be easy resetting to the last stable state, but also preserve local changes in some persistent storage in the case of container failure.
I have been considering technologies like Vagrant, Docker and/or Ansible to ship and automate the local database setup and configuration in a platform independent way. However, I read Oracle Database doesn't officially support Docker. What does that mean? Can't I build custom docker with Oracle Database binary?
Would it be better to install Oracle database using vagrant Ansible provisioner, due to the uncertain docker support? Would docker just bring unnecessary layer of complexity, as Vagrant already provides the virtualization, and Ansible could handle the setup and configuration?
I would like to hear some real life war stories about implementing platform independent database per developer pattern.
Oracle Database doesn't officially support Docker. just means that
there is no official docker image for oracle database for now. But you
can always pull a base image like ubuntu and install your database.
Once you have setup the whole environment on top of the base image, you can push the image you created into a private repository and share it.
Private repository serveice with version control is provided by dockerhub, GCP, AWS etc...
Once everyone has docker deamon running in their systems, they can just pull and deploy the image as a container.
Related
I,
We are currently working on a .net core project that will use multiple databases with the same structure.
In short, this is a multi tenant project and each tenant will use the same web application (multiple instances behind a load balancer) BUT each tenant will have its own database.
We are searching for the best solution to ease our deployment process.
When the application (and DB) is updated, we need to update the database structure on all SQL servers and all databases (one SQL can contain x databases).
FYI, application and SQL server are hosted on AWS, our CI/CD is Azure DevOps.
And last (but not least) limitation: we are working on VSCode only (MAC & Linux laptop).
So, we looked for some solutions :
Using Database projects (.sqlproj) + DACPAC generation deployed using DevOps, but it's not available on VSCode
Using Migration: not working with multiple databases and dynamic connection strings
Using SQL script: too complicated to maintains by hand a SQL script that takes care of possible cases
So could someone give us some advice to solve this problem?
The general solution here is to generate SQL Scripts for each deployment, and integrate those into your CI/CD process.
You could use EF Migrations to generate a SQL Script, that is then tested, deployed to your repo as a first-class asset, and deployed by your CI/CD pipeline. Or you could use SSDT to manage the schema and generate change scripts. But those aren't the only reasonable ways.
If you are modifying the schema by hand without using SSDT, you would normally just use a tool to generate the change script. And go from there.
There are many tools (including SSDT) that help you to diff a development environment against a target production schema and generate the change scripts. Eg Redgate ReadyRoll
Note that if you intend to perform online schema updates you need to review the change scripts manually for offline DDL operations, and to ensure that your code/database changes have the correct forward and backward compatibility to support a rollout while the application is online.
And preparing, reviewing, testing, and editing the database change scripts is not something that everyone on the team dev needs to do. So you can always consider jumping onto a Windows VM for that task.
I need to run an SQL Server (Express) instance on my dev machine to work with a web application.
I recently started working with Docker and I'm wondering if there is some advantage using the Microsoft SQLServer Docker image instead of simply installing SQL Server on my machine.
I'm working on a Win10 machine.
Fast installation
Better to say no installation needed if you already have Docker installed. Just provide 3 env vars (Server type, password and accept EULA) to docker run and you're ready.
Automatic installation/deploy
You can start SQL with just few commands, no need for user interactive process. Very useful for CI/CD pipeline.
Cloud-ready
Want to run you solution on VPS? Or GCP/AKS/AWS? You are just one step away from kubernetes - your containers can be run anywhere.
Cheap
Windows-based virtual servers are more expensive than Linux. Testing your solution could be done on Linux runners and save you money.
Testing against different servers/version
Following #DanGuzman 's comment, you can test your solution with different version on SQL server with just changing the tag of an image or SQL Server type in environment var.
Isolation
Easily create separate bridge networks with SQL server, control access. Can start several instances on one PC at once easily with just separating networks by Docker means.
Resetting
Testing requires that you can reset all changes and start all tests from scratch (from same starting point). With containers and their volumes you achieve that with one command.
Transparent configuration
You provide Dockerfile and docker-compose.yml where all steps are explicitly written clear. No need to provide additional readme's on how to setup your server.
Cross-platform
Developers can use different operating systems when working on big project (our case). Docker configuration will run on any without changes. Maybe you designes use MacOS and also want to run solution locally? Easy with Docker.
It is not clear to me how I should use the new features of SSIS in SQL Server 2012/2014 in an enterprise environment. Specifically, I am referring to the project deployment model, project parameters, environments, etc. We use a three-tier environment workflow; developing in development, testing and staging in QA, and production in production. The developers only have access to the development environment. The DBA’s migrate code to the other environments. All source is kept in TFS.
What is the intended workflow using these new features? If a developer develops the project/package, does the developer deploy the project to the SSISDB or does the developer stop after checking in the source? Where does the DBA come into the picture? Which environment contains SSISDB? How does the project/package get deployed to the other environments?
There seems to be many “how-to’s” published on the Internet, but I am struggling to find one that deals with the business workflow best practices. Can anyone suggest a link to an article on this subject?
Thanks.
What is the intended workflow using these new features?
It is up to the enterprise to determine how they will use them.
If a developer develops the project/package, does the developer deploy the project to the SSISDB or does the developer stop after checking in the source?
Where does the DBA come into the picture? Which environment contains SSISDB? How does the project/package get deployed to the other environments?
It really does depend. I advocate that developers have sysadmin rights in the development tier of servers. If they break it, they fix it (or if they've really pooched it, we re-image the server). In that scenario, they develop the implementation process and use deployments to Development to simulate the actions the DBAs will take when deploying to all the other pre-production and production environments. This generally satisfies your favorite regulatory standard (SOX/SAS70/HIPPA/CPI/etc) as those creating the work are not the same ones that install it.
What is the deliverable unit of work for SSIS packages using the project deployment model? It is an .ispac file. That is a self contained zip file with a manifest, project level parameters, project level connection managers and the SSIS packages.
How you generate that is up to you. Maybe you check the ispac in and that is what is deployed to your environments. Maybe the DBAs open the solution from source control and build their own ispac. Maybe you have Continuous Integration, CI, running and you click a button and some automated process generates and deploys the ispac.
That's 1/3 of the equation. From the SSISDB side, you likely want to create an Environment and populate it with variable values. Things like Connection Strings and file paths and user names & passwords. When you start creating those things, CLICK THE CREATE SCRIPT TO NEW WINDOW button! Otherwise, you're going to have to re-enter all that data when you lift to a new environment. I would expect your developers to check those scripts into source control. For passwords, blank out the value and make notes in your deployment checklist that they need to fix that before mashing F5.
You also need SQL Scripts to create the structure (folder) within the SSISDB for the project to be deployed into. Once deployed, you'll want to apply the Environment values, created in the preceding step, to the newly deployed project. Save those out as well.
I would have each environment contain an SSISDB. I don't want a missed configuration allowing a process in the production tier to reach across to the development tier and pull data. I've seen that, it's not pretty. When code is deployed to the QA/Stage tier, we find out quickly whether we missed a connection string somewhere because the dev servers reject the connection from QA. This means our SQL Instances don't all run under the same server account. Each tier gets their own account: domain\SQLServer_DEV, domain\SQLServer_QA, domain\SQLServer_PROD Do what you can to prevent yourself from having a bad day. If you go with a single/shared SSISDB across all your tiers, it can work, but you're going to have to invest a lot more energy ensuring that packages always run with the correct configuration environment applied lest bad things happen.
My team develop a web application using ASP.NET. The application is very much based on database (We use SQL Server). Most features require database development, in addition to server and client side code. We use GIT as our source code management system.
In order to check in (and later deploy) the database changes, we create SQL scripts and check them in. Our installer knows to run them and this is how we deploy these changes. Using scripts is very uncomfortable to merge changes (for example, if two developers added a column to the same table).
So my question is what other method or tool can you suggest? I know Visual Studio has a database project which may be useful, I still haven't learned about it, I wonder if there are other options out there before I start learning about it.
Thanks!
I think, you have to add in worlflow and use Liquibase from the first steps of database development (check Liquibase Quick-Start, where changelog started from creating initial structures).
From developers POV adding Liquibase means appearing of additional XML-file(s) in source-tree, when database-schema have to be changed in some changeset
First full disclosure that I work for Red Gate who make this product...
You might be interested in taking a look at SQL Source Control. It's a plugin for SSMS that connects your development database to your existing version control system, git in your case.
When a developer makes a change to a dev database (either in a dedicated local copy, or in a shared dev database) then this change is detected and can then be committed to the repository. Other developers can pick up this change, and you can propagate it up to other environments.
When it comes to deployment you can then use the SQL Compare tool to deploy from a specific revision in your repository that you check out.
It's really handy in cases like your example with two developers making a change to the same table. Either the 2nd developer can pick up the change from the version control system before they commit their change. Or you can use the branching/merging features of git to track these in separate branches and deploy them as separate changes. There's scope to couple this into CI systems too.
Some specifics on running SQL Source Control with git:
http://datachomp.com/archives/git-and-red-gate-sql-source-control/
And a link to a more general set-up guide
http://www.troyhunt.com/2010/07/rocking-your-sql-source-control-world.html
My main problem is where does database go?
The project will be on SVN and is developed using asp.net mvc repository pattern. Where do I put the sql server database (mdf file)? If I put it in app_data, then my other team mates can check out the source and database and run it with the database being deployed in the vs instance.
The problem with this method are:
I cannot use SQL Management Studio with this database.
Most web hosts require me to deploy the database using their UI or SQL Management studio. Putting it in App Data will make no sense.
Connection String has to be edited each time I'm moving from testing locally to testing on the web host.
If I create the database using SQL Management studio, my problems are:
How do I keep this consistent with the source control (team mates have to re-script the db if the schema changes).
Connection string again. (I'd like to automatically use the string when on production server).
Is there a solution to all my problems above? Maybe some form of patterns of tools that I am missing?
Basically your two points are correct - unless you're working off a central database everyone will have to update their database when changes are made by someone else. If you're working off a central database you can also get into the issues where a database change is made (ie: a column dropped), and the corresponding source code isn't checked in. Then you're all dead in the water until the source code is checked in, or the database is rolled back. Using a central database also means developers have no control over when databsae schema changes are pushed to them.
We have the database installed on each developer's machine (especially good since we target different DBs, each developer has one of the supported databases giving us really good cross platform testing as we go).
Then there is the central 'development' database which the 'development' environment points to. It is build by continuous integration each checkin, and upon successful build/test it publishes to development.
Changes that developers make to the database schema on their local machine need to be checked into source control. They are database upgrade scripts that make the required changes to the database from version X to version Y. The database is versioned. When a customer upgrades, these database scripts are run on their database to bring it up from their current version to the required version they're installing.
These dbpatch files are stored in the following structure:
./dbpatches
./23
./common
./CONV-2345.dbpatch
./pgsql
./CONV-2323.dbpatch
./oracle
./CONV-2323.dbpatch
./mssql
./CONV-2323.dbpatch
In the above tree, version 23 has one common dbpatch that is run on any database (is ANSI SQL), and a specific dbpatch for the three databases that require vendor specific SQL.
We have a database update script that developers can run which runs any dbpatch that hasn't been run on their development machine yet (irrespective of version - since multiple dbpatches may be committed to source control during a single version's development).
Connection strings are maintained in NHibernate.config, however if present, NHibernate.User.config is used instead, however NHibernate.User.config is ignored from source control. Each developer has their own NHibernate.User.config, which points to their local database and sets the appropriate dialects etc.
When being pushed to development we have a NAnt script which does variable substitution in the config templates for us. This same script is used when going to staging as well as when doing packages for release. The NAnt script populates a templates config file with variable values from the environment's settings file.
Use management studio or Visual Studios server explorer. App_Data isn't used much "in the real world".
This is always a problem. Use a tool like SqlCompare from Redgate or the built in Database Compare tools of Visual Studio 2010.
Use Web.Config transformations to automatically update the connection string.
I'm not an expert by any means but here's what my partner and I did for our most recent ASP.NET MVC project:
Connection strings were always the same since we were both running SQL Server Express on our development machines, as were our staging and production servers. You can just use a dot instead of the computer name (eg. ".\SQLEXPRESS" or ".\SQL_Named_Instance").
Alternatively you could also use web.config transformations for deploying to different machines.
As far as the database itself, we just created a "Database Updates" folder in the SVN repository and added new SQL scripts when updates needed to be made. I always thought it was a good idea to have an organized collection of database change scripts anyway.
A common solution to this type of problem is to have the database versioning handled in code rather than storing the database itself in version control. The code is typically executed on app_start but could be triggered in other ways (build/deploy process). Then developers can run their own local databases or use a shared development database. The common term for this is called database migrations (migrating from one version to the next). Here is a stackoverflow question for .net tools/libraries to make this easier: https://stackoverflow.com/questions/8033/database-migration-library-for-net
This is the only way I would handle this on projects with multiple developers. I've used this successfully with teams of over 50 developers and it's worked great.
The Red Gate solution would be to use SQL Source Control, which integrates into SSMS. Its maintains a sql scripts folder structure in source control, which you can keep in the same folder/ respository that you keep your app code in.
http://www.red-gate.com/products/SQL_Source_Control/