We have developed and delivered a database-driven application about 1 and half a year ago. During this time, they have backed up database, re-installed software and restored database for a few times. Also they have sent us their database afew times to perform some update to the db structure. They have used the built-in backup and restore capability of the software. The software is using SMO (SQl Server Management Object) to perform backup/restore operations.
They now claim that some of their data has been lost during recent backup and restores.
Is such a claim possible or is just their data entry fault?
I have checked their db manually and data they have been added was not there.
Was there any report on SMO bug?
Backup and restore are critical facilities in SQL Server. They always, always, always backup the database consistently (point-in-time). You cannot backup or restore in a way that looses a part of the data or introduces other inconsistencies. SQL Server always prevents you from doing that with an error message.
(The myth that data loss might be possible comes from other RDBMSs like MySQL where backups were a challenge a few years ago).
The problem is elsewhere: In the DML executed on the database.
Related
I have an Azure VM - Windows (Windows Server 2008 R2 Datacenter). It has Microsoft SQL Server 2008 R2 running on it (version v10.50.6549).
The Azure VM has backups running according to a policy - and I can see in the backups blade for the VM that they are running nightly.
If I have an issue with the SQL Server, and need to roll back to a prior version of the database, will the File Recovery option from the VM backup be adequate?
Or should I also be running SQL Server backups via a maintenance plan on the server on the VM?
If I have an issue with the SQL Server, and need to roll back to a
prior version of the database, will the File Recovery option from the
VM backup be adequate?
Maybe. VM Backups don't always give you consistent SQL backups. They usually work, but not always. If you have everything setup just right and get consistent VM backups, it might be ok-- but you are running a fairly old OS on that VM, so I'd be nervous. Very nervous. If the data is really important to you, then you should backup the data, not just the VM. Sometimes you want to restore just the data to another VM to investigate, not the entire server. I also hope you have more than just "last night's VM backup" at any given time. Sometimes bad things happen on Friday and you don't notice until Monday.
Or should I also be running SQL backups via a maintenance plan on the
server on the VM?
Yes, you should be running SQL backups if you your data is important. If your data is really important (you don't want to lose half a day of it), you should be doing full backups periodically (e.g. nightly) transaction log backups many times per hour, and keeping a few weeks worth of backups in rotation. If your data is super-important (you don't want to lose more than a few seconds), you should be mirroring it over to another database server in near-time (asynchronously). If it is critical (you don't want to lose any data), then you want to mirror to another server in real-time (synchronously).
Of course, if you are already running in Azure and don't have a DBA, managing a database is a lot easier, safer, more available, and generally cheaper if you use Azure SQL rather than trying to manage your own instance SQL Server in a VM-- oh yeah, and backups are handled for you, with millisecond point-in-time recovery for up to 45 days-- and they handle the mirroring for you to. If you want to mirror to another region across the country, you do have to pay extra for that, though.
It's a long story, but I work for the research department of a school system, and we have beef with our IT department. A while back the IT dept. issued me a server with SQL installed on it and basically told me that I was on my own - they will not help with maintenance of the server or the behind-the-scenes functioning of the SQL database. For the most part this suits me fine, but I am not a DB Admin and I've never been able to solve setting up database backups and transaction logging.
The issue I'm trying to solve for now is transaction logging. Each night I use SSIS to pull in a large amount of records from our master database into my database. I've chunked this process into 18 individual steps, and step 9 & 15 are to do transaction log backups. Unfortunately my process still crashes regularly due to the transaction log filling up -- sometimes even on Step 10 just after a backup. I was going to disable logging during my process but have been reading and see that you cannot turn off logging. I'm hoping there is a simple solution that I just don't know about...
I'm running:
Microsoft SQL Server 2012 (SP3-GDR) (KB3194721) - 11.0.6248.0 (X64)
Sep 23 2016 15:49:43
Copyright (c) Microsoft Corporation Enterprise Edition (64-bit)
on Windows NT 6.2 <X64> (Build 9200: ) (Hypervisor).
Honestly I don't even know what type of transaction logging I'm doing or how to check and find out.
Any help would be greatly appreciated
First, when you say,
Each night I use SSIS to pull in a large amount of records from our master database into my database
I presume this does not literally mean the system master database. If it does, there should not be user objects in there and it should not be processing or storing any user data there. If you are, migrate this to a user database.
Second, the short answer to the logging problem is this:
The default recovery model for a database is Full. As noted in the comments, this means that no logs are overwritten until a back up occurs. This model works for point in time recovery and makes sense for a transactional system. So there are a few options:
If this is a reporting system, it usually makes sense to put this into simple recovery model and do nightly backups. Especially if the data is only changing once per day.
If you do require up to the minute point in time recovery, log backups should be performed every 15 min. I'm guessing that the management and retention of these backups will make no sense for you and you should use option 1.
When processing data in an SSIS dataflow, each buffer is committed one at a time. If you have not fiddled with the defaults, this means you are committing 10k rows at a maximum. So everything is already getting nicely chunked up. The problem then, is not that the batch sizes are too great, it is that you are in the wrong recovery model or that you are not backing up your logs often enough.
We recently moved from a simple DB recovery model (with daily full database dumps) on our SQL Server 2000 Standard database to full recovery -- combined with weekly full database backups, daily incremental, and transaction dumps every 10 minutes.
Our previous reporting DB instance (SQL Server 2005) was built from the daily backups which no longer exist. I can re-build the reporting database by loading the weekly dump, leaving it in recovery mode, and then restore the incremental backups. Unfortunately, this is not easily scriptable and doing this by hand sucks.
Taking additional full backups of the 2000 production database will ruin the incrementals (which are desirable for a number of reasons).
Is there a better way to do this? We can't do log shipping since we're only SQL Server 2000 Standard (eventually we'll upgrade to 2K5).
Depending on how up-to-date your data needs to be, snapshot replication seems like the best fit for you. It's not that difficult to set up and I believe that it's fairly common in scenarios like yours.
I really need some help here.
I'm the owner of a SQL Server Database application that lost three days data! I can't understand how or why.
So here is the set-up.
SQL Server 2005 32bit standard edition database on Windows 2000 server. (Database B)
Database is in simple recovery mode
The database is connected as a subscriber to another database(SQL Server 2005 64bit enterprise edition on Win2k3 enterprise) using SQL Server continuous Merge Replication. (Database A)
DatabaseB was rebooted on night X as part of scheduled reboot. When the database came back up it was used as normal for a couple of days and data was created into it perfectly fine.
But then yesterday Day X + 4 it lost a lot of data.
Database B is on a server with another instance of SQL Server and they both started to run out of memory(conflicting with each other).
Here is the sequence of events from the event log when I think this happened.
AppDomain 2 (DatabaseB.dbo[runtime].1) is marked for unload due to memory pressure.
AppDomain 2 (DatabaseB.dbo[runtime].1) unloaded.
BACKUP LOG WITH TRUNCATE_ONLY or WITH NO_LOG is deprecated.
The simple recovery model should be used to automatically truncate the transaction log. (on DatabaseB)
AppDomain 3 (DatabaseB.dbo[runtime].2) created.
I know the data is missing because of my audit logs and that a user had taken a screen shot of some of the data before it was deleted.
So here is my dilema...how could this have happened?
How can several days data go missing from DatabaseB?? (it subsequently is missing from the publication db also!)
Did the truncate with the Appdomain down cause the data to be flushed from the log?
Any and all theories considered. If anyone needs more data I can add it.
Help!
This isn't the answer you want to hear, but in a nutshell, SQL Server doesn't "lose" data. Someone deleted it. If you had the database in full recovery mode, you could use a product like Quest LiteSpeed to read the logs and identify exactly how it was deleted, but in simple mode...sorry, sir, but you're out of luck.
Merge replication is implemented with triggers, so it doesn't need full recovery. Is it possible that someone disabled all triggers in the db? its easy to do DISABLE TRIGGER [database] This would at least account for the subscriber losing data.
Those appdomain lines in the log don't mean that much, its the SQL CLR telling you its unloading assemblies to free up some memory. & then reloading them later on.
Truncating the log removes inactive parts that have been committed to disk, having the recovery model set to simple means there's no point in truncating the log, as the message suggests.
None of this explains why data went missing on both the servers though. There has to be something else that caused this.
How did you verify that for the 4 days when everything was 'created perfectly fine' that it actually was? do you have backups from these days? can you see records with time stamps from those days?
Is it possible there's a ghost in the machine that did a restore without telling you?
Anyone using MozyPro to backup SQL Server databases?
I'm concerned about the way it does the backup. It just copies data files the way they are. Not using the backup database command.
Is it safe?
MozyPro uses the Volume Shadow Service (VSS) to create backups for SQL Server. SQL Server 2005 has been engineered so that VSS backups are consistent. So this is definitely a valid way to back up SQL Server databases.
Here is a white paper on how the SQL Server 2005 SQL Writer works with VSS.
Microsoft® SQL Server™ 2005 provides
support for creating snapshots from
SQL Server data using Volume Shadow
Copy Service (VSS). This is
accomplished by providing a VSS
compliant writer (the SQL writer) so
that a third-party backup application
can use the VSS framework to back up
database files. This paper describes
the SQL writer component and its role
in the VSS snapshot creation and
restore process for SQL Server
databases. It also captures details on
how to configure and use the SQL
writer to work with backup
applications in the context of the VSS
framework.
Here is the MozyPro manual (PDF), which describes how to restore SQL Server backups that were made using VSS.
That being said, if you don't trust this method, there is nothing stopping you from setting up a backup job and just having Mozy backup your *.bak files.
Judging by the hell I am currently going through with Mozy.. NO NO NO!
The backups work, in theory, just not the restore part. Mozy's extreme incremental backup system results in restores that can take weeks. Apparently. I'm still waiting despite talking their top level tech support, over 10 days have passed.
https://github.com/candera/hobocopy
WHY DOES HOBCOPY USE THE VOLUME SHADOW SERVICE?
Because HoboCopy copies from a VSS snapshot, it is able copy even
files that are in locked by some other program. Further, certain
programs (such as SQL Server 2005) are VSS-aware, and will write their
state to disk in a consistent state before the snapshot is taken,
allowing a sort of "live backup". Files locked by VSS-unaware programs
will still be copied in a "crash consistent" state (i.e. whatever
happens to be on the disk). This is generally a lot better than not
being able to copy the file at all.