I'm currently using MariaDB 10.3.17 on CentOS 8 and I'm trying to restore a backup of a specific database. I made 2 dummy databases named test and test_restore. I updated test_restore then created an incremental backup with --databases="mysql test_restore"
I ran:
# mariabackup --prepare --target-dir=<dir>/mariadb/backup/2020-02/18_10h_full/
# mariabackup --prepare --target-dir=<dir>/mariadb/backup/2020-02/18_10h_full/ --incremental-dir=<dir>/mariadb/backup/2020-02/18_10h10m_inc/
# mariabackup --copy-back --target-dir=<dir>/mariadb/backup/2020-02/18_10h_full/
After that, I lost all data on my test db but kept my updated test_restore db
I can do a full backup and restore with the incremental back up of ALL databases altogether but thats gonna take a long time.
I might be wrong, but MariaDB's blog article How to Restore a Single Database from MariaDB Backup seems very suitable here. I'm linking it, because it doesn't make sense to me to simply copy all of the instructions and hints.
Related
I need a backup from a database that makes a backup from the whole database but changes a few fields in table with random data (instead of the real data).
What is a good way to do that?
Thanks in advance ;-)
There is no silver bullet or one size fits all script for this issue, however the process itself is not a big deal, if I follow your question to the letter, you are looking for guidance on how to script the following 3 operations
Backup the database
Restore into a temporary location
Execute scripts to anonymize the data
Backup the anonymized data
See issue 1
This is a common development scenario, but also comes up when we want to otherwise test or demonstrate an application to prospective clients. Read over this tip for detailed information as well as a solution Automatically Create and Anonymize Downstream Databases from Azure
Scripting Backups
MSDN is a good source of scripts for common SQL Server tasks, there are many different flavours for scripting backups and many of them will be dependent on what resources your database is hosted on, this is a good start, but google or SO is your friend here: Create a Full Database Backup (SQL Server)
USE SQLTestDB;
GO
BACKUP DATABASE SQLTestDB
TO DISK = 'E:\Backups\SQLTestDB.Bak'
WITH FORMAT,
MEDIANAME = 'E_Backups',
NAME = 'Full Backup of SQLTestDB';
GO
Restoring backups
You can easily use Backup to clone an existing database and restore it into the same server as long as you use a different database name, and you restore to different logical files on disk
by default a restore operation would try to use the original filenames but that will not work when we restore side-by-side the original source database, because those files are still in use by the source database!
Have a read over Restore a Database to a New Location (SQL Server)
First you need to know the names of the files stored within the backup, then you can construct a query that will restore the database mapped to new files.
This answer to How do you backup and restore a database as a copy on the same server? should help a lot.
Putting that together we get:
BACKUP DATABASE SQLTestDB TO DISK = 'E:\Backups\SQLTestDB.Bak'
GO
-- use the filelistonly command to work out what the logical names
-- are to use in the MOVE commands. the logical name needs to
-- stay the same, the physical name can change
restore filelistonly from disk='E:\Backups\SQLTestDB.Bak'
-- --------------------------------------------------
--| LogicalName | PhysicalName |
-- --------------------------------------------------
--| SQLTestDB | C:\mssql\data\SQLTestDB.mdf |
--| SQLTestDB_log | C:\mssql\data\SQLTestDB_log.ldf |
-- -------------------------------------------------
restore database SQLTestDB_Temp from disk='E:\Backups\SQLTestDB.Bak'
with move 'SQLTestDB' to 'C:\mssql\data\SQLTestDB_Temp.mdf',
move 'SQLTestDB_log' to 'C:\mssql\data\SQLTestDB_log.ldf'
It is possible to put this script into a stored proc so you can reuse it, one issue is how to use the results from RESTORE FILELISTONLY, you'll find this answer will help if you want to go down that path: https://stackoverflow.com/a/4018782/1690217
Anonymizing Data
This is where things get specific, now that your database has been restored to a temporary location, you can pretty much do whatever you want to the data in a series of INSERT, UPDATE, or `DELETE' statements that you need, you could even modify the schema to remove particularly sensitive or audit or other logging tables that you may have that you don't need to distribute.
Do not leave audit tables in the database that you have anonymized, unless you plan on anonymizing the context within those logs as well! Depending on your circumstances, also consider nulling out all IMAGE and VARBINARY columns as their contents will be particularly hard to process sufficiently.
I wont go into the specifics, there are already healthy discussions on this topic on SO:
Anonymizing customer data for development or testing
Backup the Anonymized data
When you have finished scrubbing or anonymizing your database, simply complete your script with a call to backup the temporary DB that has been anonymized.
I removed some indexes on a very large table and realized I needed them. Instead of adding them back concurrently, which would take a very long time, I was wondering if I could just do restore using a database copy that was taken before the indexes were removed?
If by "database copy" you mean a copy of the Postgres DB directory at file level (with Postgres not running to get a consistent state), then yes, such a snapshot includes everything, indexes too. You could copy that back on file level, and then start Postgres - falling back to the previous state, of course.
If, OTOH, you mean a backup with the standard Postgres tools pg_dump or pg_dumpall, then no, indexes are not included physically. Just the instructions to build them. It would not make sense to include huge junks of functionally dependent values. Building them from restored data may be about as fast.
Either way, you could not add back an index from an older snapshot to a live DB anyway, after changes to the table have been made. That's a logically impossible. Then there is no alternative to rebuilding the index one way or another.
I'll answer for MySQL. You tagged your question with both mysql and postgresql so I don't know which one you really use.
If your backup was a physical backup made with a backup solution like Percona XtraBackup or MySQL Enterprise Backup, it will include the indexes, so restoring it will be quicker.
If your backup was a logical backup made with mysqldump or mydumper, then the backup includes only data. Restoring it will have to rebuild the indexes anyway. It will not save any time.
If you made the mistake of making a "backup" only by copying files out of the data directory, those are sort of like the physical backup, but unless you copied the files while the MySQL Server was shut down, the backup is probably not viable.
What is the fastest way to backup/restore Azure SQL database?
The background: We have the database with size ~40 GB and restoring it from the .bacbac file (~4GB of compressed data) in the native way by Azure SQL Database Import/Export Service takes up to 6-8 hours. Creating .bacpac is also very long and takes ~2 hours.
UPD:
UPD.
Creating the database (by the way transactional consistent) copy using CREATE DATABASE [DBBackup] AS COPY OF [DB] takes only 15 minutes with 40 GB database and the restore is simple database rename.
UPD. Dec, 2014. Let me share with you our experience about the fastest way of DB migration schema we ended up with.
First of all, the approach with data-tier application (.bacpac) turned out to be not viable for us after DB became slightly bigger and it also will not work for you if you have at least one non-clustered index with total size > 2 GB until you disable non-clustered indexes before export - it's due to Azure SQL transaction log limit.
We stick to Azure Migration Wizard that for data transfer just runs BCP for each table (parameters of BCP are configurable) and it's ~20% faster than approach with .bacpac.
Here are some pitfalls we encountered with the Migration Wizard:
We run into encoding troubles for non-Unicode strings. Make sure
that BCP import and export runs with same collation. It's -C ... configuration switch, you can find parameters with which BCP calling
in .config file for MW application.
Take into account that MW (at least the version that is actual at the moment of this writing) runs BCP with parameters that will leave the constraints in non-trusted state, so do not forget to check all non-trusted constraints after BCP import.
If your database is 40GB it's long past time to consider having a redundant Database server that's ready to go as soon as the main becomes faulty.
You should have a second server running alongside the main DB server that has no actual routines except to sync with the main server on an hourly/daily basis (depending on how often your data changes, and how long it takes to run this process). You can also consider creating backups from this database server, instead of the main one.
If your main DB server goes down - for whatever reason - you can change the host address in your application to the backup database, and spend the 8 hours debugging your other server, instead of twiddling your thumbs waiting for the Azure Portal to do its thing while your clients complain.
Your database shouldn't be taking 6-8 hours to restore from backup though. If you are including upload/download time in that estimate, then you should consider storing your data in the Azure datacenter, as well as locally.
For more info see this article on Business Continuity on MSDN:
http://msdn.microsoft.com/en-us/library/windowsazure/hh852669.aspx
You'll want to specifically look at the Database Copies section, but the article is worth reading in full if your DB is so large.
Azure now supports Point in time restore / Geo restore and GeoDR features. You can use the combination of these to have quick backup / restore. PiTR and Geo restore comes with no additional cost while you have to pay for
Geo replica
There are multiple ways to do backup, restore and copy jobs on Azure.
Point in time restore.
Azure Service takes full backups, multiple differential backups and t-log backups every 5 minutes.
Geo Restore
same as Point in time restore. Only difference is that it picks up a redundant copy from a different blob storage stored in a different region.
Geo-Replication
Same as SQL Availability Groups. 4 Replicas Async with read capabilities. Select a region to become a hot standby.
More on Microsoft Site here. Blog here.
Azure SQL Database already has these local replicas that Liam is referring to. You can find more details on these three local replicas here http://social.technet.microsoft.com/wiki/contents/articles/1695.inside-windows-azure-sql-database.aspx#High_Availability_with_SQL_Azure
Also, SQL Database recently introduced new service tiers that include new point-in-time-restore. Full details at http://msdn.microsoft.com/en-us/library/azure/hh852669.aspx
Key is to use right data management strategy as well that helps solve your objective. Wrong architecture and approach to put everything on cloud can prove disastrous... here's more to it to read - http://archdipesh.blogspot.com/2014/03/windows-azure-data-strategies-and.html
I have one database which contains the most recent data, and I want to replicate the database content into some other servers. Due to non-technical reasons, I can not directly use replicate function or sync function to sync to other SQL Server instances.
Now, I have two solutions, and I want to learn the pros and cons for each solution. Thanks!
Solution 1: detach the source database which contains the most recent data, then copy to the destination servers which need the most recent data, and attach database at the destination servers;
Solution 2: make a full backup of source server for the whole database, then copy data to destination servers and take a full recovery at the destination server side.
thanks in advance,
George
The Detach / Attach option is often quicker than performing a backup as it doesn't have to create a new file. Therefore, the time from Server A to Server B is almost purely the file copy time.
The Backup / Restore option allows you to perform a full backup, restore that, then perform a differential backup which means your down time can be reduced between the two.
If it's data replication you're after, does that mean you want the database functional in both locations? In that case, you probably want the backup / restore option as that will leave the current database fully functional.
EDIT: Just to clarify a few points. By downtime I mean that if you're migrating a database from one server to another, you generally will be stopping people using it whilst it's in transit. Therefore, from the "stop" point on Server A up to the "start" point on Server B this could be considered downtime. Otherwise, any actions performed on the database on server A during transit will not be replicated onto server B.
In regards to the "create a new file". If you detach a database you can copy the MDF file immediately. It's already there ready to be copied. However, if you perform a backup, you have to wait for the .BAK file to be created and then move it to it's new location for a restore. Again this all comes down to is this a snapshot copy or a migration.
Backing up and restoring makes much more sense, even if you might eek out a few extra minutes from a detach attach option instead. You have to take the original database offline (disconnect everyone) prior to a detach, and then the db is unavailable until you reattach. You also have to keep track of all of the files, whereas with a backup all of the files are grouped. And with the most recent versions of SQL Server the backups are compressed.
And just to correct something: DB backups and differential backups do not truncate the log, and do not break the log chain.
In addition, the COPY_ONLY functionality only matters for the differential base, not for the LOG. All log backups can be applied in sequence from any backup assuming there was no break in the log chain. There is a slight difference with the archive point, but I can't see where that matters.
Solution 2 would be my choice... Primarily becuase it won't create any downtime on the source database. The only disadvatage i can see is that depending on the database recovery model, the transaction log will be truncated meaning if you wanted to restore any data from the transaction log you'd be stuffed, you'd have to use your backup file.
EDIT: Found a nice link; http://sql-server-performance.com/Community/forums/p/5838/35573.aspx
We would like to be able to nightly make a copy/backup/snapshot of a production database so that we can import it in the dev environment.
We don't want to log ship to the dev environment because it needs to be something we can reset whenever we like to the last taken copy of the production database.
We need to be able to clear certain logging and/or otherwise useless or heavy tables that would just bloat the copy.
We prefer the attach/detach method as opposed to something like sql server publishing wizard because of how much faster an attach is than an import.
I should mention we only have SQL Server Standard, so some features won't be available.
What's the best way to do this?
MSDN
I'd say use those procedures inside a SQL Agent job (use master.xp_cmdshell to perform the copy).
You might want to put the big huge tables on their own partition and have this partition belong to a different file group. You would backup then backup and restore the main file group.
You might want to also consider doing incremental backups. Say, a full backup every weekend and an incremental every night. I haven't done file group backups, so I don't know if these work well together.
I'm guessing that you are already doing regular backups of your production database? If you aren't, stop reading this reply and go set it up right now.
I'd recommend that you write a script that automatically runs, say once a day, that:
Drops your current test database.
Restores your current production backup to your test environment.
You can write a simple script to do this and execute it using the isql.exe command line tool.