I really need some help here.
I'm the owner of a SQL Server Database application that lost three days data! I can't understand how or why.
So here is the set-up.
SQL Server 2005 32bit standard edition database on Windows 2000 server. (Database B)
Database is in simple recovery mode
The database is connected as a subscriber to another database(SQL Server 2005 64bit enterprise edition on Win2k3 enterprise) using SQL Server continuous Merge Replication. (Database A)
DatabaseB was rebooted on night X as part of scheduled reboot. When the database came back up it was used as normal for a couple of days and data was created into it perfectly fine.
But then yesterday Day X + 4 it lost a lot of data.
Database B is on a server with another instance of SQL Server and they both started to run out of memory(conflicting with each other).
Here is the sequence of events from the event log when I think this happened.
AppDomain 2 (DatabaseB.dbo[runtime].1) is marked for unload due to memory pressure.
AppDomain 2 (DatabaseB.dbo[runtime].1) unloaded.
BACKUP LOG WITH TRUNCATE_ONLY or WITH NO_LOG is deprecated.
The simple recovery model should be used to automatically truncate the transaction log. (on DatabaseB)
AppDomain 3 (DatabaseB.dbo[runtime].2) created.
I know the data is missing because of my audit logs and that a user had taken a screen shot of some of the data before it was deleted.
So here is my dilema...how could this have happened?
How can several days data go missing from DatabaseB?? (it subsequently is missing from the publication db also!)
Did the truncate with the Appdomain down cause the data to be flushed from the log?
Any and all theories considered. If anyone needs more data I can add it.
Help!
This isn't the answer you want to hear, but in a nutshell, SQL Server doesn't "lose" data. Someone deleted it. If you had the database in full recovery mode, you could use a product like Quest LiteSpeed to read the logs and identify exactly how it was deleted, but in simple mode...sorry, sir, but you're out of luck.
Merge replication is implemented with triggers, so it doesn't need full recovery. Is it possible that someone disabled all triggers in the db? its easy to do DISABLE TRIGGER [database] This would at least account for the subscriber losing data.
Those appdomain lines in the log don't mean that much, its the SQL CLR telling you its unloading assemblies to free up some memory. & then reloading them later on.
Truncating the log removes inactive parts that have been committed to disk, having the recovery model set to simple means there's no point in truncating the log, as the message suggests.
None of this explains why data went missing on both the servers though. There has to be something else that caused this.
How did you verify that for the 4 days when everything was 'created perfectly fine' that it actually was? do you have backups from these days? can you see records with time stamps from those days?
Is it possible there's a ghost in the machine that did a restore without telling you?
Related
It's a long story, but I work for the research department of a school system, and we have beef with our IT department. A while back the IT dept. issued me a server with SQL installed on it and basically told me that I was on my own - they will not help with maintenance of the server or the behind-the-scenes functioning of the SQL database. For the most part this suits me fine, but I am not a DB Admin and I've never been able to solve setting up database backups and transaction logging.
The issue I'm trying to solve for now is transaction logging. Each night I use SSIS to pull in a large amount of records from our master database into my database. I've chunked this process into 18 individual steps, and step 9 & 15 are to do transaction log backups. Unfortunately my process still crashes regularly due to the transaction log filling up -- sometimes even on Step 10 just after a backup. I was going to disable logging during my process but have been reading and see that you cannot turn off logging. I'm hoping there is a simple solution that I just don't know about...
I'm running:
Microsoft SQL Server 2012 (SP3-GDR) (KB3194721) - 11.0.6248.0 (X64)
Sep 23 2016 15:49:43
Copyright (c) Microsoft Corporation Enterprise Edition (64-bit)
on Windows NT 6.2 <X64> (Build 9200: ) (Hypervisor).
Honestly I don't even know what type of transaction logging I'm doing or how to check and find out.
Any help would be greatly appreciated
First, when you say,
Each night I use SSIS to pull in a large amount of records from our master database into my database
I presume this does not literally mean the system master database. If it does, there should not be user objects in there and it should not be processing or storing any user data there. If you are, migrate this to a user database.
Second, the short answer to the logging problem is this:
The default recovery model for a database is Full. As noted in the comments, this means that no logs are overwritten until a back up occurs. This model works for point in time recovery and makes sense for a transactional system. So there are a few options:
If this is a reporting system, it usually makes sense to put this into simple recovery model and do nightly backups. Especially if the data is only changing once per day.
If you do require up to the minute point in time recovery, log backups should be performed every 15 min. I'm guessing that the management and retention of these backups will make no sense for you and you should use option 1.
When processing data in an SSIS dataflow, each buffer is committed one at a time. If you have not fiddled with the defaults, this means you are committing 10k rows at a maximum. So everything is already getting nicely chunked up. The problem then, is not that the batch sizes are too great, it is that you are in the wrong recovery model or that you are not backing up your logs often enough.
I've got SQL2012 running on 2 different servers with public, static IP addresses. I want to implement replication in a way that will keep both servers in sync at all times, regardless of which server is actually receiving the data. I've been reading about the subscriber/publisher model but I'm not exactly sure which should be which. A few facts about our setup:
I'm trying to achieve failover. If server A goes down, I need server B to be operational and have all latest data, or as close as possible. And vice versa. When the server comes back online, I need the replication to get caught up quickly and start working again. I need failures to be graceful, in other words I can't have server A get weird just because server B went offline.
I don't need realtime replication, but close would be nice. If server A was 10 seconds behind server B with data updates, nobody would care. But if it were an hour behind, that would be bad. Fast DB performance is more important that realtime replication, but again, close would be nice.
My database is just shy of 900Mb, and grows by 3Mb per day.
I am looking for advice on the best way to set this up given my setup and needs. Much appreciated.
Since one server will be Primary and the other Failover, use Log Shipping. It will keep two databases the same for all transactions completed on Primary server upto the failure moment. All transactions that have not completed at the moment of failure, will not appear on Failover server, so they should resubmitted by the application and hit Failover server.
Also there should be a Recovery procedure, to ensure than Primary server is up to date.
Useful articles:
Database Mirroring and Log Shipping.
Configure Log Shipping
We have developed and delivered a database-driven application about 1 and half a year ago. During this time, they have backed up database, re-installed software and restored database for a few times. Also they have sent us their database afew times to perform some update to the db structure. They have used the built-in backup and restore capability of the software. The software is using SMO (SQl Server Management Object) to perform backup/restore operations.
They now claim that some of their data has been lost during recent backup and restores.
Is such a claim possible or is just their data entry fault?
I have checked their db manually and data they have been added was not there.
Was there any report on SMO bug?
Backup and restore are critical facilities in SQL Server. They always, always, always backup the database consistently (point-in-time). You cannot backup or restore in a way that looses a part of the data or introduces other inconsistencies. SQL Server always prevents you from doing that with an error message.
(The myth that data loss might be possible comes from other RDBMSs like MySQL where backups were a challenge a few years ago).
The problem is elsewhere: In the DML executed on the database.
I just migrated a SQL Server 2008 database while the sales staff went to lunch. I did a full backup and then copied the backup to the new server where I restored it to the new SQL Server installation.
The staff got back from lunch and told me that a few of the recent deals they put in right before lunch were no longer showing up in the system. Upon investigation it seems that the full backup was not including some of the very recent data that was added to the database before the backup was performed.
I suspect there is something I'm not accounting for that relates to the transaction log, but not being a MSSQL specialist, I'm not quite sure what I'm doing wrong. Could someone point me in the right direction?
Did you back up just the database, or with transaction logs as well? My guess is that you didn't backup transaction logs, which will contain recent transactions not yet flushed down to the mdb file.
The data may have been entered in an open transaction. Since the backup has to adhere to the ACID model, if a transaction was started before the backup started and was not committed by the time the backup has completed, the transaction will be rolled back upon restoring the database.
This is why you never do these things during a work day! Next time do this over the weekend and close the database off to users by putting it in single user mode. They could have added the data after your backup or during it if you didn't lock them out first.
I have the following scenario:
Our system is running a SQL Server Express 2005 database locally (on each users desktop, if you will). The system is storing a lot of production data from a machine. There are high demands on the safety of the data, and doing a backup each night, or even each hour is not enough. We need a backup strategy that will ensure almost instantaneous/continuous backup of the database.
Is there anyone out there that has successfully implemented a system similar to this, and/or has got some ideas of how to accomplish it? The only thing I can think of right now is to have mirrored drives (raid) to hold the data, but that would be complicated and expensive.
I would appreciate any and all thoughts on this, since it is a real issue for me and my company. Thanks in advance!
Update:
I was not clear enough in my description of the scenario. The system is storing data in a vehicle that has no connection to anything. A centralized database is therefor not possible. Neither can we use a standard/enterprise version of SQL Server, since it would be to expensive (each vehicle would need a license). Thanks for your input!
Switch your database into "Full" recovery mode. Do full backup every night and do delta backup after major user action. The delta backups can be done to the flash memory or different hard-drive, and all data can be synchronized with server when online.
Another simple way is to trace all user changes and important data in a text file that stored on a separate drive. If SQL database crashes the user or other operator can repeat steps to restore data.
One way I've seen this done is by using DoubleTake.
I will assume that a central database on a server is not feasible because your systems are running standalone and are not connected to anything. So this is what I would do
Set up RAID on the computer. This insures you against simple disk failure.
Any SQLSever database can be recovered to the point of the last commited transaction if you have a full database backup and a set of transaction logs available. Basically you simply restore the last full backup then apply the transaction logs going forward. See these links.
http://www.enterpriseitplanet.com/storage/features/article.php/11318_3776361_3
https://web.archive.org/web/1/http://blogs.techrepublic%2ecom%2ecom/datacenter/?p=132
So what you need to do is set up a periodic full backup of both the database and transaction logs, and more regular transaction log backups (and ensure that your transaction log can never run out of space).
In the event of failure you restore the last full backup, then apply the transaction logs going forward.
Myself, if these are critical systems, I would be inclined to add an additional drive to the system and make sure that the backups are copied over to that. This is because as good as raid is it does sometimes have issues - raid controllers fail, disks get wiped accidentally in parallel, disk failures go unnoticed so your just running on one disk etc. If you ensure backups are copied to a separate disk then you can always recover to the last transaction log backup. You should also ensure tape backups of course, but they are generally a last resort in the event of trouble.
If for some reason you cannot set up raid then you should still install a second disk, but place the database file on one drive and the transaction log on the other and copy backups to both disks. In the event of failure of the C drive, or some other software issue crashing the database you can still recover to the last commited transaction. Failure on the D drive limits you to the last transaction log backup (Oracle used to allow you to mirror the transaction log from the database, which again would completely cover you, but I don't think this facility exists in SQL Server)
If you are looking for a scheduler for SQL Server Express (which doesn't come with one) then I've been using SQLScheduler quite happily without problems, and it's free.
The most obvious answer would be to ditch SQL Server Express running locally and use a single source for your data (such as a standard SQL server install on a central storage location). Unless your system requires individual back ups of every single person's own individual instance of SQL Server Express.
If your requirements are so stringent as to call for instantaneous backups on every operation, you should definitely think about a different method of storage than local instances of SQL Server Express.
Wouldn't it be easier to just use one centralized SQL Server and back that up every hour or so? If you truly need instantaneous backup, your company (which seems not to want to spend money by installing Express on each machine) will need to spring for two servers and two SQL Server Enterprise licenses to implement Mirroring.
Raid isn't that expensive, but it is also not the best option. If you really want high availability data you should upgrade to sql server standard on a remote server where each user connects to and use transaction based replication to an sql server (express) instance on another machine. Raid doesn't always protect you from dataloss. If the data is that important for you then the costs should not be that much of an issue.
Update in response to the question update.
If you can't use remote servers then there a couple of options:
You write a trigger which initiates a backup script on each insert or update and stores it on a seperate harddrive.
You use raid. But beware that if the raid controller fails that you still got a problem.
RAID is not expensive. Use RAID to protect against hard drive failure. You also need monitoring though. No point in having this if you let both drives fail.
Also, implement hourly incremental backups, then daily incremental backups and finally weekly full backups.
You need all of these strategies working together because they protect against different things. RAID does not protect against human or coding errors destroying data. Hourly and weekly backups don't protect against hard drive failure.