So I followed the AWS documentation to perform a native RDS backup using MS-SQL Server. My goal is to be able to download the .bak file.
The config seems to be correct, and I was able to execute the backup stored procedure:
And I created the the option group and have the S3 bucket linked to it.
But when I went to the S3 bucket, the .bak file is not there, even the stored procedure is performed successfully.
Try using the rds_task_status stored procedure to see if any errors occurred during native backup - exec msdb.dbo.rds_task_status #db_name='aa144bgo6mn8srl'. This will produce a table of sync statuses.
Do you see a lifecycle of Completed when you run this query?
Related
System Error: Cannot bulk load because the file "XYZ.txt" could not be opened. Operating system error code 1311(There are currently no logon servers available to service the logon request.)
I have a stored procedure in SQL Server 2008 R2 which is using Bulk Insert command to load data from txt files into SQL Server tables. These files are on a shared folder on a drive located on different domain. I have full access to the drive. I tried copying files to different directory on that drive, moving files, deleting files and everything works.
When I execute the stored procedure from a ssms session from local computer it works like a champ. It is able to open the files on shared drive, read it and load the data into the SQL Server tables without any issue. When I call the stored procedure from a SQL Server Agent job, it throws this error.
SQL Server Agent is using the account which is very powerful with lot more permissions than mine. But the job fails.
To find a workaround I created a SSIS package which calls the stored procedure from "Execute SQL Task". It uses Windows authentication to connect to the database. I tried executing the package and it ran successfully. It is able to upload the data from txt to table.
So, then I created a SQL Server user with my account details, and then used that credentials to create a proxy with ssis sub system. I then scheduled the job to execute the step with newly created proxy to see if it can upload the data. But it failed with the same error.
I am confused what am I doing wrong..? I even added myself to bulkdmin role and ran the job again with no success.
I'll appreciate if someone can help.
Thanks.
Just out of curiosity I tried replacing Bulk Insert command with BCP. For some reason BCP worked. It is able to Open the files on network drive and read through it to insert the data in sql server tables. I can even call the same stored proc from sql agent job and it works perfectly fine. I didn't need to use SSIS package to solve this.
I use Sql server Native client 11 for database programing. I use this command to create a full backup
BACKUP DATABASE [AminDB] TO DISK = 'E:\AminExe\Hogug02\BKUP\PYBKUPyy-13970515-1211-1.BAK' WITH INIT, COMPRESSION, NOSKIP;
When I execute this command in SSMS, I get this result:
Processed 175632 pages for database 'AminDB', file 'AminDB_dat' on file 1.
Processed 2 pages for database 'AminDB', file 'AminDB_log' on file 1.
BACKUP DATABASE successfully processed 175634 pages in 6.029 seconds (227.588 MB/sec).
When I execute this command using SqlExecDirect, I get SUCCESS_WITH_INFO and SQLGetDiagRec returns this result:
Processed 175632 pages for database 'AminDB', file 'AminDB_dat' on file 1.
and backup is not created.
for many other databases, executing backup database command always returns SQL_SUCCESS_WITH_INFO, but the backup file randomly is created or not.
What should I do for my Application create successful backup?
thanks in advance
mr.abzadeh
I called SQLMoreResults after BACKUP DATABASE statement repeatedly until it returned SQL_NO_DATA. This completely solved the problem.
I am testing to see if a SQL Server server based program can also work on AWS Cloud Server with 2016 SQL Server on the Amazon server. In order for me to test it, I need to restore 2 databases.
The first one eventually restored fine once i figured it out...restoring the database from my S3 "bucket" BAK file.
So then I tried to restore the 2nd database, using the same restore stored proceudre, and get this message:
[2017-12-28 02:44:22.320] The file 'D:\rdsdbdata\DATA\smsystemdata.mdf' cannot be overwritten. It is being used by database 'amwsys'.
[2017-12-28 02:44:22.320] File 'sm_system_data' cannot be restored to 'D:\rdsdbdata\DATA\smsystemdata.mdf'. Use WITH MOVE to identify a valid location for the file.
I can't find where to use the WITH MOVE because it won't let me restore it interactively through the Management Studio restore menu; instead I have to give it a stored procedure command:
exec msdb.dbo.rds_restore_database
#restore_db_name='sample99',
#s3_arn_to_restore_from='arn:aws:s3:::lighthouse-chicago/sample999.bak';
And each time it tells me it can't restore it because it's going to overwrite the first database's files.
Much thanks
bill
I think you are stuck in RDS's restriction.
I had the similar problem as you. Multiple restore from one DB instance is impossible at RDS.
Here is RDS's restriction you may encounter.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/SQLServer.Procedural.Importing.html
You can't restore a backup file to the same DB instance that was used
to create the backup file. Instead, restore the backup file to a new
DB instance. Renaming the database is not a workaround for this
limitation.
You can't restore the same backup file to a DB instance multiple
times. That is, you can't restore a backup file to a DB instance that
already contains the database that you are restoring. Renaming the
database is not a workaround for this limitation.
If you are in this case, you can't use .BAK file. To avoid it, you should create DB instance with DML and import table data.
When I extract a data-tier application from a Microsoft Azure SQL database that has a Master Key, I was unable to import it into SQL server on my local PC.
You will find others had this issue here: SSMS 2016 Error Importing Azure SQL v12 bacpac: master keys without password not supported
However the steps provided as the answer did not work on my installation.
Steps are
1. Disable auditing on the server (or database)
2. Drop the database master key with DROP MASTER KEY command.
Microsoft Tech Support verified this solution did not work on my installation of SQL Server and after actually taking remote control of my PC and trouble shooting, they were unable to determine why this was occurring.
I needed to find a way to remove the Master Key from the bacpac file. I have a Powershell script to remove the Master Key from the BACPAC file but it requires extracting, renaming files and running scripts from Windows Powershell to get the db imported.
Does anyone have a program or set of scripts which would automate the process of removing the Master Key and importing a SQL DB from Azure with a single command?
I am new to this forum. Please do not be harsh with this post. I am trying to do the best I can to help others to save the many hours I spent coming up with this.
I have cobbled together a T-SQL script which calls a Windows Powershell script (also cobbled from multiple sources) to extract a data-tier application (database) from Microsoft Azure SQL database and import it into a database on my local SQL Server by running ONE command. Over the months I found some of the code that is in my scripts from other blogs etc. I am not able to provide the credit due to those folks as I didn't keep track of where I got the info. If you are reading this and you see your code, please take credit. I apologize for not being able to give you the credit for your work.
There may be configuration settings on your PC and your local SQL server that need adjustments as this entire solution requires pretty much full access to your computer. If you run into trouble with compatibility, let me know and I will do the best I can to let you know how my system is configured in case it will help you.
I am using Windows 10 Pro and Microsoft SQL Server Developer (64-bit) v12.0.5207.0
I have placed the two files that do all the work on GitHub here: https://github.com/Wingloader/Auto-Azure-BACPAC-Download.git
GetNewBacpac-forGitHub.sql
GetAzureDB-forGitHub.ps1
WARNING: The Powershell script file will store your SQL sa password and your Azure SQL login in clear text!
If you don't want to do this, don't use this solution.
My computer is owned and controlled solely by me so I am able to open up the security in my system and I am willing to assume the responsibility of safeguarding it.
The basic steps of my solution are are accomplished as follows: (steps 1 and 2 are optional as I like to keep a version of the DB I am working with as of the point in time I pull down a clean production copy of my Azure DB)
Back up the current DB as MyLocalDB.bak.
Restore that backup from step 1 to a new DB with the previous day stamped at the end of the DB name (e.g., MyLocalDB20171231)
Delete the original MyLocalDB database (needed so we can recreate the DB with the original name later on)
Pull down the production database from Azure and create a new database with the name MyLocalDB.
The original DB is deleted in step 3 so that the restored DB can use the original name (important when you have data connections referring to that DB name)
In Step 4, the work of extracting the data-tier application DB from Azure is initiated by this line in the T-SQL:
EXEC MASTER..xp_cmdshell '%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe -File C:\Git\GetUpdatedAzureDB\GetAzureDB.ps1"'
The Powershell script does the following:
The target for the extract is a file named today.bacpac (hardcoded). The first thing to do is delete that file if it already exists.
Extract the DB from Azure into the today.bacpac file.
Note: my DB on Azure has a Master Key for encryption. This will need to be removed from the files prior to importing the bacpac file into your local DB or it will fail (this may not be required in SQL 2017 according to my previous conversations with MS Support). If you do not use a Master Key, you can either strip out the code that does this step or just leave it alone. It won't remove anything if it isn't there. It would just add a little overhead to the program.
Open the today.bacpac file (zip file) and remove the MasterKey node from the Origin.xml file.
Modify the Model.xml file to updates the SHA hash length. This is required in order for the file not to appear to have been tampered with when SQL opens the bacpac file.
Re-zips the files back into a new file today-patched.bacpac
Runs this line of code (from Powershell) to import the bacpac file into SQL Server
&C:\Program Files (x86)\Microsoft SQL Server\140\DAC\bin\SqlPackage.exe" /Action:Import /SourceFile:"C:\Git\GetUpdatedAzureDB\today-patched.bacpac" /TargetConnectionString:"Data Source=MyLocalSQLServer;User ID=sa; Password=MySAPassword; Initial Catalog=MyLocalDB; Integrated Security=false;"
After editing the two files to provide updated paths, usernames and passwords, run the SQL script. You do not need to edit the scripts again. You can run the SQL script again without modification and it will create a new copy of your Azure DB.
Done!
I have a small (few hundred MB) SQL Server database running on RDS. I've spent several hours trying to get a copy of it onto my local SQL Server 2014 instance. All of the following fail. Any ideas what might work?
Task -> Backup fails because it doesn't give my admin account permission to backup to a local drive.
Copy Database fails during create package with While trying to find a folder on SQL an OLE DB error was encountered with error code 0x80040E4D
From SSMS, while connected to the RDS server, running BACKUP DATABASE. This fails with message BACKUP DATABASE permission denied in database 'MyDB'. Even after running EXEC sp_addrolemember 'db_backupoperator' for the connected user.
General scripts generates a 700MB .sql file. Running that with sqlcmd -i fails at some point after producing plausible .mdf and .ldf files that can't be mounted on the local server (probably because the sqlcmd failed to complete and unlock them).
AWS has finally provided a reasonably easy means of doing this: It requires an S3 bucket.
After creating a bucket called rds-bak I ran the following stored procedure in the RDS instance:
exec msdb.dbo.rds_backup_database
#source_db_name='MyDatabase',
#s3_arn_to_backup_to='arn:aws:s3:::rds-bak/MyDatabase.bak',
#overwrite_S3_backup_file=1;
The following stored procedure returns the status of the backup request:
exec msdb.dbo.rds_task_status #db_name='MyDatabase'
Once it finished I downloaded the .bak file from S3 and imported it into a local SQL Server instance using the SSMS Restore Database... wizard!
The SSIS Import Export Wizard can generate a package to duplicate a whole set of tables. (It's not the sort of Copy Database function that relies on files - it makes a package with data flow components for each table.)
It's somewhat brittle but can be made to work :-)
SSMS Generate Scripts feature can often fail with any large data set as the script for all the data is just to large/verbose. This method never scripts out the data.
Check this out: https://github.com/andrebonna/RDSDump
It is a C#.NET Console Application that search for the latest origin database Snapshot, restore it on a temporary RDS instance, generate a BACPAC file, upload it to S3 and delete the temporary RDS instance.
You can transform your RDS snapshot into a BACPAC file, that can be downloaded and imported onto your local SQL Server 2014 instance using the feature answered here (Azure SQL Database Bacpac Local Restore)
Redgate's SQL Compare and SQL Data Compare are invaluable for these types of things. They are not cheap (but worth every penny imo). But if this is a one-time thing, you could use the 14 day trial and see how it behaves for you.
http://www.red-gate.com/products/