I'm looking for a way to perform VM backups for my oVirt 3.5 environment. I came across a couple of scripts like this one: http://www.clevernetsystems.com/virtual-machines-backup-script-rhev-3/ but they shutdown each VM before the backup. It also seems like they perform a clone which isn't ideal.
I'd like to avoid downtime altogether. Does anybody know of any scripts that will perform a snapshot of a live VM and then transfer the snapshot over to the export domain? It would be great too if the the snapshot was deleted after the transfer as well but right now I'll take a snapshot of a live VM and the transfer.
Thanks.
This script works nicely: https://github.com/wefixit-AT/oVirtBackup
In general, you can use the API to snapshot VM;clone snapshot to export domain; delete snapshot, and the export domain will contain a backup you can copy or use elsewhere. To restore, attache the export domain and import
Related
There is one postgres database in rds. We need to sync it to another postgres database in rds.
I have connected one database with pg admin but not sure how to synchronise the database with the other one.
Is this a recurring task or a one time task? If it's a one time thing, you can do a dump/backup and restore it to a new instance. If this is a recurring thing, then...
Why do you want to replicate the data? What is your actual goal? If it's for scaling, you should likely use the native RDS functionality for that. If it's for testing, then you will likely want to sanitize data as part of the copy.
As smac2020 said in their answer, AWS provides a migration service you can use for several use cases, but if this is a one-time thing, that is likely overkill. Just do a backup and restore that to another server.
You can also leverage change-data-capture to replicate data out, but that will likely take a lot to get working well and will only be worth it in certain circumstances. We leverage CDC to copy data to our warehouse, for example. Here is an example that streams data to Kinesis. Here is another example that streams to Kinesis using the migration service.
So I recently started using AWS and Elastic Beanstalk with RDS.
I wonder whats the best practices for creating database backups?
So far my setup is this.
Enable automatic backups.
bash script that creates manual snapshots everyday and removes manual snapshots older than 8 days.
bash script that creates a sql dump of the database and uploads it to S3.
Reason why I am creating the manual snapshots is if I were to delete the database by misstake, then I still have snapshots.
The bash scripts is on an EC2 instance launched with a IAM role which is allowed to execute these scripts.
Am I on the right track here?
I really appreciate answers, thanks.
A bit of context...
That automated backups are not saved after DB deletion is a very important technical gotcha. I've seen it catch devs on the team unawares, so thanks for bringing this up.
After the DB instance is deleted, RDS retains this final DB snapshot and all other manual DB snapshots indefinitely. However, all automated backups are deleted and cannot be recovered when you delete a DB instance. source
I suspect for most folks, the final snapshot is sufficient.
Onto the question at hand...
Yes. 110%. Absolutely.
I wouldn't create manual snapshots; rather, copy the automated ones.
Option 1: You already have the automated snapshots available. Why not just copy the automated snapshot (less unnecessary DB load; though, admittedly less of an issue if you're multi-AZ since you'll be snapshoting from the replica), which created a manual snapshot. I'd automate this using the aws sdk and a cron job.
Option 2: Requires manual adherence. Simply copy your automated snapshots (to create manual snapshots) before terminating a DB.
Unclear why you'd need the s3 dump if you have the snapshots.
For schema changes?: If you're doing it for schema changes, these should really be handled with migrations (we use knex.js, but pick your poison). If that's a bridge too far, remember that there's an option for schema-only dumps (pg_dump --schema-only). Much more manageable.
Getting at the data?: Your snapshots are already on s3, see faq. You can always load a snapshot, and sql dump it if you choose. I don't see an immediately obvious reason for purposely duplicating your data.
I've seen a few posts on this, but I just want to make sure I'm not missing something.
I'm seriously considering moving from Azure to App Harbor, but I'm a bit dismayed that there doesn't seem to be a way to maintain daily SQL Server database backups.
I understand that App Harbor maintains daily file system snapshots. This is great for recovering from a catastrophic failure, but doesn't do much to deal with recovering from user errors. For example, if I accidentally delete a chunk of rows, I may want to restore a database from a few days ago to help recover.
I know about these tools for transferring data to/from App Harbor:
- "Generate Scripts" tool in SQL Management Studio
- Bulk copy tool: https://github.com/appharbor/AppHarbor-SqlServerBulkCopy
Those are fine for doing a one-off backup or restore, but I'm looking to figure out some way to back up data automatically, and ideally save it off to AWS S3 storage. Is there a tool or service out there that could possibly do this?
Thank you!
I've created a simple console app that does a daily backup of tables in a SQL Server database. The output is then zipped and uploaded to Amazon S3 storage. This app can be deployed as an AppHarbor background worker. No local SQL server required!
See notes in the readme file for instructions and limitations. This does what we need for now, but I'm happy to let others work on the project if you'd like to extend it.
https://bitbucket.org/ManicBlowfish/ah-dbbackup
I would like to know how people tackle backup and restore of data in the cloud.
I plan to use AppEngine for business use and as far as I can tell there is no classic backup and restore functionality built into AppEngine. It is reasonable because there database structure is much different.
So the question is how to approach backup and restore in high replication AppEngine application?
Agree with the previous posts. The built-in datastore backup / restore is solid enough.
However, if your data is partioned by namespace, Google does not offer the possibility to backup / restore by namespace which is a big limitation. It means that if you have a multi-tenant application, you cannot backup / restore data for one namespace.
Would you really need backup / restore by namespace, you will have to extend the backup / restore from Google which is open source (see http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/datastore_admin/backup_handler.py).
I am currently on the path to perform this modification to the open source of Google but did not find the time to do it yet.
Hope this helps !
Backup/Restore, Copy and Delete on Google App Engine is still experimental, but it is there. I would suggest you to built a prototype and try to Backup/Restore a few times before deciding to build the whole thing. The data is pretty much secure, but if you want to protect the datastore from some abuse/attack then it's necessary to have that resolved. If you are afraid of losing data, then the chances are pretty low for that actually to happen, but still, you never know!
GAE backup works pretty well for us: few days ago I backed up entities of about 800MB in size. No problems there. It also does restore - just save data to a file in blobstore or Cloud Store and you can restore it anytime. There is a limitation: no automatic/programmable backup - it's all manual.
I have live and dev versions of an ASP.NET website, with corresponding live and dev versions of the SQL Server back end. Each is on its own server (so 4 total) on the same local network.
Every so often, we've been manually replacing the dev DB with the live DB, using backup & restore, so that the dev DB is a reasonably recent snapshot of the live DB. What's the best way to automate that process?
I'd be interested in either having it run on a schedule, or making it a pushbutton process where we still control the timing.
(For what it's worth, we use CruiseControl.net; so an option that would let us use the CC.net dashboard to launch this process would be nice.)
1- you can use replication (transactional or snapshot) for synchronizing two databases.
2- write an application that get backup from db and restore it. (use SMO)
3- write an application that delete all data from dev db and copy all data from live db to it. you can use bcp (SQLBulkCopy) for doing this work.
I think scheduling or automating a restore would be relatively simple with the RESTORE command in T-SQL. Schedule your backups to go to a designated location and set whatever schedule / script that refreshes the dev database to look in the same location.
Why not set up a Snapshot replication between your live and dev database? This is the way we handle keep our dev database in synch with our production database.
http://msdn.microsoft.com/en-us/library/ms151198.aspx