How to debug SQL Server 2014 Managed Backup to Azure? - sql-server

I'm hoping to configure and use SQL Server 2014's Managed Backup to Azure feature. I've tested a manual backup to URL T-SQL query, and it worked. But, the backups created by the managed backup to Azure feature are failing to finish.
The managed backup process leaves a bunch of 0 byte backup files on Azure storage, and they're all locked (active leases) because of the failed backup process. I'm trying to figure out what the issue is. My SQL Server 2014 edition is the Web edition, which apparently does not support compression. The SQL Server Management Studio dialog for configuring the managed backup to Azure feature does not provide any options for compression, though. So, perhaps it's trying to compress the backup file and failing? I don't know how to see what's actually being tried by SQL Server while it does its automated backup to Azure, though. Anybody have any ideas there?
I'm seeing the following errors in the event log ...
Date 10/1/2014 3:10:39 PM
Log SQL Server (Current - 10/1/2014 2:42:00 PM)
Source Backup
Message
BACKUP failed to complete the command BACKUP LOG CaseLaw. Check the backup application log for detailed messages.
Date 10/1/2014 3:12:01 PM
Log SQL Server (Current - 10/1/2014 2:42:00 PM)
Source spid71
Message
Error: 18210, Severity: 16, State: 1.
Date 10/1/2014 3:12:01 PM
Log SQL Server (Current - 10/1/2014 2:42:00 PM)
Source spid71
Message
BackupVirtualDeviceFile::DetermineFileSize: SetPosition(0,EOF) failure on backup device 'https://caselaw.blob.core.windows.net/sqldb01-mssqlserver/CaseLaw_d68e6c91feba4f7f891a66cb86fcce8d_20141001151038-05.log'. Operating system error Backup to URL received an exception from the remote endpoint. Exception Message: The remote server returned an error: (409) Conflict..
I can't read the full 409 error message, so I've got no idea what the actual 409 error is. According to MSDN, there are multiple possible 409 responses.
I have been struggling with this issue for several days. I tried a manual scheduled task that runs my T-SQL query, which works, but I can't have it automatically clean itself up after X days, using that method. Any ideas? Thanks!

This is indeed related to COMPRESSION being set on the backup command that gets called by the Managed Backup process.
Enabling debug messages pointed me to the stored procedure getting called: [msdb].[smart_admin].[sp_do_backup]. If you look inside this stored procedure you will see "COMPRESSION" hard coded into the various backup commands that it issues.
A find and replace for "COMPRESSION" to "NO_COMPRESSION" should fix your issue.
Note this only need apply to SQL 2014 Web Edition, as other versions support compression, and this shouldn't even come up.

Related

Error when trying to load database into backup server

Im trying to load a database dump into my Sybase backup server.
Running sybase ASE-16_0 on both my primary and backup machine.
Import is done in isql cli via
load database DB from ./dumps/data_dump
Error message is the following:
Backup Server session id is: 22. Use this value when executing the
'sp_volchanged' system stored procedure after fulfilling any volume change
request from the Backup Server.
Backup Server: 4.141.2.40: [11] The 'open' call failed for database/archive
device while working on stripe device
'/opt/dumps/data_dump' with error number
13 (Permission denied). Refer to your operating system documentation for further
details.
Also found this SAP Knowledge Base article, but it's hidden behind a paywall:
https://userapps.support.sap.com/sap/support/knowledge/en/3140989
Setting up a new Server fixed the issue for me. Im still working on a possible answer for this problem.

MSSQL error: "Script level upgrade for database 'master' failed ... upgrade step 'msdb110_upgrade.sql' encountered error 200, state 7, severity 25."

All of a sudden one day (on my DEV PC) my Microsoft SQL Server 2012 instance (installed as instance name "SQL2012") would not start (all my other installed instances did). Trying to start it manually under Services failed. I don't recall making any recent changes prior to this. The cause of the failure was a mystery.
On inspecting Event Viewer, under System it showed a rather amusing error message [emphasis mine]:
The SQL Server (SQL2012) service terminated with the following service-specific error:
WARNING: You have until SQL Server (SQL2012) to logoff. If you have not logged off at this time, your session will be disconnected, and any open files or devices you have open may lose data.
checking under Application Event Log, I found these 2 error messages (preceded by a number of MSSQL$SQL2012 informational messages):
Script level upgrade for database 'master' failed because upgrade step 'msdb110_upgrade.sql' encountered error 200, state 7, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the 'master' database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
followed by:
Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
Fearing having lost my system databases (and not having a backup of them to restore - who makes backups of their system dbs anyway??) and needing to access the instance, and attached databases - I was willing to try anything. Even the possible restore of the system databases: Restoring the SQL Server Master Database Even Without a Backup - but that looked quite complex.
Fortunately, I was eventually able to start the instance (thank you to this answer: https://stackoverflow.com/a/59676743/4993856 which I trusted, because Pinal Dave also mentions that particular switch in: SQL SERVER – Script level upgrade for database ‘master’ failed because upgrade step msdb110_upgrade.sql encountered error 926, state 1, severity 25) if I ran:
net start mssqlserver$SQL2012 /T902
This pointed to some issue with the upgrade script... (Remember SQL is installed with instance name: SQL2012, hence the mssqlserver$SQL2012 used above for the named instance).
After some more searching I discovered this post: Installing service pack / cumulative update on SQL Server 2016 / 2017 breaks database engine (not exactly the same SQL version as mine) which pointed to the following possible Region Settings setting (Control Panel [when viewed by 'icons'] > All Control Panel Items > Region > Administrative > "Change system locale..."):
"Beta: Use Unicode UTF-8 for worldwide language support" in Region Settings
THAT WAS IT!!! After de-selecting that option (and possibly restarting my computer), the MSSQL Server 2012 Instance started up without any issue, and I was able to access all my previously attached databases.
I assume the pending upgrade scripts ran successfully. Thinking back about it now, it is possible that I agreed to installing a SQL Update, and never bothered to test access to the instance afterwards.
I also don't recall exactly why I chose to enable that specific setting under Region Settings, possibly due to some Linux compatibility, but it looks like it has become defaulted 'on' in recent Windows builds.
I got the same problem SQL2017 after update Windows Patch Hotfix3391(KB5001228)
after restart server MSSQL Fail to start and event viewer shown error below
Script level upgrade for database 'master' failed because upgrade step 'msdb110_upgrade.sql' encountered error 200, state 7, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the 'master' database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
Solution
Fix by remove Beta:Use Unicode UTF-8 for Worldwide lang.. in the Region Settings
Then it require restart server. After restart MSSQL can start as normal.
The problem is the msdb_110.sql update script, the script is a bit of a mess, with mixed tabs and spaces (wtf?).
It tries to run a couple of procedures that fail, on startup of sql-server. They fail when the code-page is 65001 (usually because the BETA utf-8 code page option has been selected) and so SQL server fails to start.
This appears to happen any time a SQL Server update is installed. I only experience this error with SQL Server 2017, not 2019
Why?
Don't know? The script is a mess.
Solution
Deselect the use utd-8 code page option
Restart the machine
Start sql server and let it run the script
(optional) reselect the use utd-8 code page option
Restart machine again and sql server
(optinal but recommended) uninstall windows, install a unix and run postgres

SQL Server Agent - BCryptDecrypt failed

I'm using SQL Server 2016 and in the SQL Server Agent event log, I have thousands of error's with the following message:
Date 26/12/2019 3:47:06 PM
Log SQL Server Agent (Current - 16/12/2019 4:28:00 PM)
Message
[442] BCryptDecrypt failed (-1073741762).
I can't find anything that relates to this error and there doesn't seem to be anything wrong that I can see except for the message - does anyone know what it is and how I could potentially fix it?
Thank you
I found the issue.
We use to use managed backups to Azure blob for our DB's - we canned this method for our on prem servers and went another path - however when we swapped over we didn't disable managed backups on the server and we deleted our storage account within Azure. This in return was throwing these weird errors up.
You can't disable managed backup's on your DB's if your storage account in Azure is gone - i had to recreate the storage account with the same naming convention and recreate the blob - update the SAS key in your credentials - then you can disable managed backups.
Hope this helps others.

Failed to restore Team Foundation server databases

Getting errors restoring TFS databases
I've scheduled the backups of our team foundation server with the Team Foundation Server Administration Console. It takes every night a full backup, and it looks like it really does.
Now disaster occured, our tfs crashed, and i'm setting up a new windows server with tfs 2012. Luckely i've got backups... But.....
Restoring the backups seems not so easy.
When i restore, I get the following error messages:
TF401001: The restore operation for database ... failed. Please refer to the logs for more information
I get this for every database (ReportServer, ReportServerTempDB, Tfs_warehouse, tfs_defaultcollection, TFS_configuration).
Looking at the log file:
[Info #09:07:33.420] RESTORE DATABASE [Tfs_Configuration] FROM DISK = N'\\172.16.46.234\Backup_TeamFS.crashed\Tfs_Configuration_9858508765097608601L.trn' WITH RECOVERY, NOUNLOAD, STATS = 10, STOPATMARK = N'TfsMarkTfpt'
[Error #09:07:33.514] TF401001: The restore operation for database Tfs_Configuration failed. Please refer to the logs for more information
[Error #09:07:33.514]
Exception Message: The database "Tfs_Configuration" does not exist. RESTORE can only create a database when restoring either a full backup or a file backup of the primary file.
RESTORE DATABASE is terminating abnormally. (type SqlException)
SQL Exception Class: 16
SQL Exception Number: 3118
SQL Exception Procedure:
SQL Exception Line Number: 1
SQL Exception Server: TEAMFS
SQL Exception State: 1
SQL Error(s):
SQL Error[1]: System.Data.SqlClient.SqlError: RESTORE DATABASE is terminating abnormally.
Class: 16
Number: 3013
Server: TEAMFS
Source: .Net SqlClient Data Provider
State: 1
Procedure:
Line Number: 1
Exception Data Dictionary:
HelpLink.ProdName = Microsoft SQL Server
HelpLink.EvtSrc = MSSQLServer
HelpLink.EvtID = 3118
HelpLink.BaseHelpUrl = http://go.microsoft.com/fwlink
HelpLink.LinkId = 20476
Exception Stack Trace: at Microsoft.TeamFoundation.Admin.SqlDatabaseBackupRestoreComponent.Execute(String sqlStatement)
at Microsoft.TeamFoundation.Admin.SqlDatabaseRestore.Restore(String connectionString)
at Microsoft.TeamFoundation.Admin.RestoreFactory.RestoreLogs(ActivityContext context, String serverName, String databaseName, String path, List`1 backupChain, Boolean useMarkAtEnd, Int32 minPercent)
at Microsoft.TeamFoundation.Admin.RestoreDatabases.Run(ActivityContext context)
But, i'm fully sure i do have a full backup. Looking at log files from the backup:
[9/1/2013 10:24:44 PM] [Info] starting full backups
[9/1/2013 10:24:44 PM] [Info] Taking Full Backup server:TEAMFS database:Tfs_Configuration filename:\\172.16.46.234\Backup_TeamFS\Tfs_Configuration_9858508747697505276F.bak
[9/1/2013 10:24:44 PM] [Info] BACKUP DATABASE [Tfs_Configuration] TO DISK = N'\\172.16.46.234\Backup_TeamFS\Tfs_Configuration_9858508747697505276F.bak' WITH NOFORMAT, NOINIT, NOSKIP, REWIND, NOUNLOAD, STATS = 10
[9/1/2013 10:32:17 PM] [Info] Backup Complete for database Tfs_Configuration
[9/1/2013 10:32:17 PM] [Info] Finished Full Backup server:TEAMFS database:Tfs_Configuration filename:\\172.16.46.234\Backup_TeamFS\Tfs_Configuration_9858508747697505276F.bak
update
I've only scheduled the backups to make a full backup, not incrementals. Maybe something to do with that?
we have managed to do this so I shall share my experience..
So you have got backups..
Also, you have got a fresh new OS and installation files for TFS (TFS 2013 in our case)
Double click installation file.
Once installation completes, ideally MSDN tells you to close the windows and then in the TFS CONSOLSE you have to go to scheduled backups to restore which FAILS so below method will help you resolve the problem:
a. Restore the config and team collections databases via the SQL Management console
b. Open the TFS management console
c. Click on Application Tier and Configure this feature
d. Run CMD as administrator and run following command:
TFSConfig Accounts /add /AccountType:ApplicationTier /Account:"NT Authority\Network Service" SQLInstance:ServerName /DatabaseName:TFS_Configuration
e. Then just follow the next button in the tool from step c.
Cheers
Here is how I just walk around this issue. Use your SQL management tool to restore the backup file. They are just like regular Database backup files, so you just treat them like ones.
After you finish, you can see your collections in your TFS console.
You might bump into some errors during the process.
The one I got was missing one collection. You need to restore everything before going to TFS console.
BTW, after you finish, reboot your TFS service or the whole server, otherwise you might bump into another error like can't find the object reference.
Check out this blog to find some help.
BTW, I assume the reason you did this because you tried to restore TFS to a different server.
You need to reinstall the application tier only from TFS installation. It took me forever to figure it out.

transactional replication error due to OS error 3

I am for the first time trying to setup transactional replication. This is from an sql 2000 server sp3a to an sql 2005 server which I believe should work.
I did a quick test on my local machine (sql 2005) using it as both the publisher and subscriber and had no trouble setting it up. I repeated a similar process for the real servers using enterprise manager for the 2000 publisher parts and management studio to setup the PULL subscriber. This all seemed to work and the publisher logs seem to be indicating it was preparing the initial data however I am not getting anything coming over as of yet. I checked the logs and am getting an os error 3. I have included the two log sections I think are important below.
2009-07-21 21:37:42.043 The process could not read file 'D:\Program Files\Microsoft SQL Server\MSSQL\ReplData\unc\DOMINO_qlsdat_DOMINO qlsdat to PONGOSQL\20090721164816\enbhostname_1.sch' due to OS error 3.
Message
The replication agent encountered an error and is set to restart within the job step retry interval.
See the previous job step history message or Replication Monitor for more information.
Looks to me like I need to give share permission to the replication data, should I have setup the replication data to go to a share using unc path.
OS Error 3 is no a permission problem, is a path correctness problem: Error code: (Win32) 0x3 (3) - The system cannot find the path specified.. A permission problem would be error 5: Error code: (Win32) 0x5 (5) - Access is denied..
The path D:\Program Files\Microsoft SQL Server\MSSQL\ReplData\unc\DOMINO_qlsdat_DOMINO qlsdat to PONGOSQL\20090721164816\enbhostname_1.sch is incorrect on the server that executes it. Usually one has to use UNC paths in replication, I can't know for sure if that is the problem, but it likely is.

Resources