in my Company we use MSSQL Server 2019 Enterprise. We are hosting our Business Intelligence Database in it. There we have three layers: Source, Staging and Dw. For some of the tables transfered/synced between the layers we use CDC. Let's give me an example:
There is a table "dbo.MyData" in SOURCE
"SOURCE.dbo.MyData" is being updated from let's say an CRM-System using the UpdateTimestamp of the Source-Data and the High Watermark-Method
In SOURCE there is CDC enabled on the whole table "dbo.MyData"
When Updating the Table "dbo.MyData" in STAGING it uses the CDC Instance (Net Changes) of "SOURCe.dbo.MyData" as CDC-Source and then propagates the changes to "STAGING.dbo.MyData"
This workflow is automated (also in SSIS)
This usualy works flawlessly. But now I had an issue I don't understand.
A new line was added to "SOURCE.dbo.MyData"
When updating "STAGING.dbo.MyData" the change/new line wasn't recognized
When running the update of "STAGING.dbo.MyData" later the change/new line was recognized
My only suggestion why this could happen is that the cdc-changelog wasn't yet built when the sync to "STAGING.dbo.MyData" happened. Is this possible? I thought the log was being built the moment the server was asked for it and if it wasn't finished yet the answer to the question would just take longer.
If it is possible: Is there a way to check if a log-rebuild is currently in progress and wait for it? Any other options to work around this issue? Or may there be a completely other issue I'm not seeing and can check for?
Greetings!
Related
I have a web application that has an SQL database.
For clarity I'm using Asp.Net 4.0/c#/SQL Server 2008 Web edition.
I recently puclished the site, which was my first, by creating a deployment package for the database.
Now a couple of months down the line, I need to update the database structure. The web application now has data that has been entered via the web, so i'll need to update the structure, then copy data across.
As this is the first time I've done it, I'm unsure of the process I should follow - is there a standard practice for this kind of update?
Also, since some of the tables use incremental ID's I need to ensure they remain the same in the newly updated database.
Any tips, links, advice appreciated.
Important Guidelines:
I assume you have not changed structure entirely (means keys column are same though solution is around for that too)
Steps are as follows:
Take export of the database
Add or remove the columns or whatever changes you want
Import the database back
Check the log for rows/tables (if some) were not updated successfully
Make SQL queries for them and run them to sync
Here are some general steps for this:
Take backup of your online database and restore it locally
Modify local database to suite your needs
Use third party comparison and synchronization tool to publish changes to your production database
There are many of these available and you can use them in trial mode to get the job done if you’re on a tight budget. You can try tools from Red Gate, ApexSQL, Idera, Dev Art and others…
At the moment we manually push changes from our DEV SQL environment to the TEST and production (using Schema compare in Visual studio, plus some script we create while making changes to the DEV), but this is very time consuming and error prone.
We were wondering if there was a better way of doing this and how would we need to implement this.
I've read about maybe using versioning (how would this work?), or maybe using RED GATES' SQL Source control (but can this be used to push changes to the TEST, or is it only used to keep track of local changes?)
We want a reliable way to update our TEST & Production servers so that data won't be corrupted/lost... We use SQL Server 2008 R2 and Visual Studio 2012.
We are starting a new project, so it's time for a change! Thank you for your time!
One simple way to do this would be to have simple version table in the db with one row and one column which stores the version number.
Now everytime you are pushing changes to dev, create the incremental sql script, Have a master script which based on the current version of the db, will call the necessary incremental sql scripts to upgrade the schema to the latest version.
Be careful of dropping columns, changing column types, or reducing columns sizes e.g. varchar(100) to varchar(10) in your incremental scripts, as that could result in data loss if not planned properly.
You incremental scripts should be idempotent, that they could be run over and over, just in case to handle the case when db crashes during upgrade.
Although there are many benefits in using SQL Source Control (and I'd love for you to give it a go, as I'm the product manager!), its purpose is limited to versioning and not managing and deploying to your various environments. The correct Red Gate tool for this would be Deployment Manager.
http://www.red-gate.com/delivery/deployment-manager/
There is a blog maintained by the Deployment Manager project team here, which should give you an idea of where the tool is headed:
http://thefutureofdeployment.com/
Does Schema Compare in VS have CLI? If so you can probably automate it to run several times during the day. If not you can try using some other 3rd party tools that support CLI such as ApexSQL Diff for schema and ApexSQL Data Diff for synchronizing data.
I have been doing some research on the correct procedure to follow when working with both a development and live production database. The best article i have found was this one: Strategies for Database Development and Deployment, but i can not accept the idea that i have to maintain a Word document, manually, of every change i make to the DB. That seems ridiculous to me...
I use SQL Server Management Studio to manage my SQL databases both in dev and in prod. Is there a way to deploy latest changes to production WITHOUT destroying tables and data. Can someone please point me to a good procedural article on how this is done in SSMS.
Thanks
It is irresponsible to make changes to a database design without creating change scripts that are put into source control.
However, if you are already in this curcumstance, I suggest buying red_gate's SQLCompare. It will look at the two databases and script the differnces. You still can't run this willy-nilly though - sometimes you have made changes to the dev database that are not yet part of the curent version being pushed to prod and SQLCompare has no way to know this. It is far better to create the scripts as you go (Using alter table when the table currently exists so as not to disrupt existing data) and keep them in source control with the rest of code that you will be pushing at the same time.
The only right way to do it in prodiction - with or without Management Studio - prepare, check, test and run the scripts manually.
WITH FRESH BACKUP!
A common strategy is to keep an ordered set of change scripts, e.g. prefixed with date or database version, which can easily be tested on the development database by starting with a fresh backup from production. The change scripts can often be generated from SQL Server Management Studio when making changes, or could be crafted manually in case of more complex changes.
Another suggestion would be to version control the database definitions (tables, procedures etc). This can be easily achieved by using SQL Server Management Studio to generate create scripts for all objects after each update. This way you can easily compare changes made over time, or between different environments.
I am using github for maintaining versions and code synchronization.
We are team of two and we are located at different places.
How can we make sure that our databases are synchronized.
Update:--
I am rails developer. But these days i m working on drupal projects (where database is the center of variations). So i want to make sure that team must have a synchronized database. Also the values in various tables.
I need something which keep our data values synchronized.
Centralized database is a good solution. But things get disturbed when someone works offline
if you use visual studio then you can script your database tables, views, stored procedures and functions as .sql files from a database solution and then check those into version control as well - its what i currently do at my workplace
In you dont use visual studio then you can still script your sql as .sql files [but with more work] and then version control them as necessary
Have a look at Red Gate SQL Source Control - http://www.red-gate.com/products/SQL_Source_Control/
To be honest I've never used it, but their other software is fantastic. And if all you want to do is keep the DB schema in sync (rather than full source control) then I have used their SQL Compare product very succesfully in the past.
(ps. I don't work for them!)
You can use Sql Source Control together with Sql Data Compare to source control both: schema and data. Here is an article from redgate: Source controlling data.
These are some of the possibilities.
Using the same database. Set-up a central database where everybody can connect to. This way you are sure everybody uses the same database all the time.
After every change, export the database and commit it to the VCS. This option requires discipline and manual labor.
Use some kind of other definition of the schema. For example, Doctrine for php has the ability to build the database from a yaml definition which can be stored in the vcs. This can be easier automated then point 2.
Use some other software/script which updates the database.
I feel your pain. I had terrible trouble getting SQL Server to play nice with SVN. In the end I opted for a shared database solution. Every day I run an extensive script to backup all our schema definitions (specifically stored procedures) for version control into text files. Due to the limited number of changes this works well.
I now use this technique for our major project and personal projects too. The only negative is that it relies on being connected all the time. The other answers suggest that full database versioning is very time consuming and I tend to agree. For "live" upgrades we use the Red Gate tools, they do both schema and data compare and it works very well.
http://www.red-gate.com/products/SQL_Data_Compare/. We were using this tool for keeping databases in sync in our company. Later we had some specific demands so we had to write our own code for synchronization. Depends how complex is you database and how much changes is happening. It is much simpler if you have time when no one is working and you can lock database for syncronization.
Check out OffScale DataGrove.
This product tracks changes to the entire DB - schema and data. You can tag versions in any point in time, and return to older states of the DB with a simple command. It also allows you to create virtual, separate, copies of the same database so each team member can have his own separate DB. All the virtual copies are tracked into the same repository so it's super-easy to revert your DB to someone else's version (you simply check-out their version, just like you do with your source control). This means all your DBs can always be synchronized.
Regarding a centralized DB - just like you don't want to work on the same source code, you don't want to be working on the same DB. It means you'll constantly break each other's code and builds each time someone changes something in the DB.
I suggest that you go with a separate DB for each developer, and sync them using DataGrove.
Disclaimer - I work at OffScale :-)
Try Wizardby. This is my personal project, but I've used it in my several previous jobs with great deal of success.
Basically, it's a tool which lets you specify all changes to your database schema in a database-independent manner and then apply these changes to all your databases.
Building and maintaining a database that is then deplyed/developed further by many devs is something that goes on in software development all the time. We create a build script, and maintain further update scripts that get applied as the database grows over time. There are many ways to manage this, from manual updates to console apps/build scripts that help automate these processes.
Has anyone who has built/managed these processes moved over to a Source Control solution for database schema management? If so, what have they found the best solution to be? Are there any pitfalls that should be avoided?
Red Gate seems to be a big player in the MSSQL world and their DB source control looks very interesting:
http://www.red-gate.com/products/solutions_for_sql/database_version_control.htm
Although it does not look like it replaces the (default) data* management process, so it only replaces half the change management process from my pov.
(when I'm talking about data, I mean lookup values and that sort of thing, data that needs to be deployed by default or in a DR scenario)
We work in a .Net/MSSQL environment, but I'm sure the premise is the same across all languages.
Similar Questions
One or more of these existing questions might be helpful:
The best way to manage database changes
MySQL database change tracking
SQL Server database change workflow best practices
Verify database changes (version-control)
Transferring changes from a dev DB to a production DB
tracking changes made in database structure
Or a search for Database Change
I look after a data warehouse developed in-house by the bank where I work. This requires constant updating, and we have a team of 2-4 devs working on it.
We are fortunate because there is only the one instance of our "product", so we do not have to cater for deploying to multiple instances which may be at different versions.
We keep a creation script file for each object (table, view, index, stored procedure, trigger) in the database.
We avoid the use of ALTER TABLE whenever possible, preferring to rename a table, create the new one and migrate the data over. This means that we don't have to look through a history of ALTER scripts - we can always see the up to date version of every table by looking at its create script. The migration is performed by a separate migration script - this can be partly auto-generated.
Each time we do a release, we have a script which runs the create scripts / migration scripts in the appropriate order.
FYI: We use Visual SourceSafe (yuck!) for source code control.
I've been looking for a SQL Server source control tool - and came across a lot of premium versions that do the job - using SQL Server Management Studio as a plugin.
LiquiBase is a free one but i never quite got it working for my needs.
There is another free product out there though that works stand along from SSMS and scripts out objects and data to flat file.
These objects can then be pumped into a new SQL Server instance which will then re-create the database objects.
See gitSQL
Maybe you're asking for LiquiBase?