Our EPOS system copies data by compressing the database into a zip file, and manually copying to each till, using shared directories.
Each branched is liked to the main location, using VPN which can be problematic, but is required for the file sharing to work correctly.
Since our database system currently does not support replication, is there another solution for copying data or should we migrate our software to another database?
Replication is the "right" way to go, so if migrating to another database is an option (is it really?), that's the best route.
You might consider a utility that queries all the tables for raw data (in CSV?), sending that to files. Then at least you don't have to take the database down to do the backup.
Related
Background:
I have a medium-sized database (900GB) that needs to be copied onto another server (driven via code, not scheduled). Currently we take a backup (to .bak), move it to a staging server, and restore it to the target server. The target server does not have enough space to hold the backup file, and the restored instance simultaneously, thus the staging server. These transfers (backup to staging, restore from staging) happen over SMB2. The staging server needs to go away due to business requirements, however. It is worth mentioning the target server will be taken offline (and used offline) after the transfer, so I'm not sure the mirroring or replication options are valid.
I have identified two options -- one is to backup the database to the primary server, and open up firewall rules/smb to serve the backup file to the target server over SMB. ("RESTORE FROM \x.x.x.x\blah\db.bak"). Security isn't a fan, though.
The ideal solution (and one that could easily be implemented in every other database I've worked with), is to quiesce the database and transfer the datafiles (in the case of ms-sql, mdf and ldf files). However, upon research I see there is no such functionality available out of the box. I know I can take the database offline to copy the mdf/ldf safely, but that's not an acceptable solution (database must remain online).
I have read LOTS of posts and Microsoft documentation regarding VSS / shadow copy, but I have also read lots of conflicting information about the reliability of using VSS/sqlwriter to copy the mdf/ldf file to the target server, and simply re-attaching the database.
I am looking for documentation or advice (or even backup software that can be programmatically driven via an API) to accomplish this goal of transferring the database without requiring a secondary holding place. Currently I'm researching how to drive this copying process with Powershell, using VSS(vssadmin/vshadow from sdk), but I'm not confident in what I'm reading, and it's not even clear to me if VSS/sqlwriter is a supportable method to copying online LDF/MDF files. Any advice is appreciated.
Thanks,
In my project (similar to mediafire and rapidshare), clients can upload files to the server. I am using DB2 database and IBM WAS web server and JSP as server side scripting. I am creating my own encryption algorithm, as it is the main aim of the project.
I need suggestion whether files themselves should be stored in the database or if only the location of the files should be stored. Which approach is best?
There are Pros and Cons for storing BLOBs in the database.
Advantages
DBMS support for BLOBs is very good nowadays
JDBC driver support for BLOBs is very good
access to the "documents" can happen inside a transaction. No need to worry about manual cleanup or "housekeeping". If the row is deleted, so is the BLOB data
Don't have to worry about filesystem limits. Filesystems are typically not very good at storing million of files in a single directory. You will have to distribute your files across several directories.
Everything is backed up together. If you take a database backup you have everything, no need to worry about an additional filesystem backup (but see below)
Easily accessible through SQL (no FTP or other tools necessary). That access is already there and under control.
Same access controls as for the rest of the data. No need to set up OS user groups to limit access to the BLOB files.
Disadvantages
Not accessible from the OS directly (problem if you need to manipulate the files using commandline tools)
Cannot be served by e.g. a webserver directly (that could be performance problem)
Database backup (and restore) is more complicated (because of size). Incremental backups are usually more efficient in the filesystem
DBMS cache considerations
Not suited for high-write scenarios
You need to judge for yourself which advantage and which disadvantage is more important for you.
I don't share the wide-spread assumption that storing BLOBs in a database is always a bad idea. It depends - as with many other decisions.
It's general knowledge that storing files in the database -especially big ones- it's generally a bad idea. There are brilliant explanations in these questions:
Storing a file in a database as opposed to the file system?
Storing Images in DB - Yea or Nay?
And I'd like to highlight some points myself:
Storing files in your DBMS will make your data very big, and big databases are a maintaining hell (specially backups)
Portability becomes an issue, as every DBMS vendor makes its own implementation of BLOB files
There's a performance lost related to SELECT sentences to BLOB fields, compared to disk access
Well my Opinion would be to store the relevant information like path, name, description, etc... in the database and keep the file evtl. encrypted on the filesystem, it would be cheaper to scale your system adding a webserver than adding a database one as webspace is cheap comparing with databases, all you will need then is to add an IP column to your database or server name so you can address teh new webserver.
We are working on generating a database that will be distributed to several third parties. We will also re-generate this database on an on-going basis, redistribute it, and those third parties will need to overwrite their existing database copy with the new version.
I'm trying to decide whether I want to detach the database temporarily, make a copy of the .mdf, and send that copy out, or whether I should just do a full backup of the database, and send the .bak out.
The primary difference I can see is that to distribute the .mdf, you must detach the database temporarily, so that you can copy it.
What are the other pros/cons of each format?
Are there security implications with distributing one over the other?
Is it easier to initially import one format over the other?
Thank you.
Neither. The proper way to distribute database changes is via upgrade scripts, otherwise those third parties using the database will loose the actual data contained in the database.
For the case when the data is never changed by the third parties (ie. the database is read only at those sites) then distribution by backup file is feasible. MDF is completely out of the question, first and foremost because MDF is not the entire database: at least the LDF is required in addition to recreate a coherent database. Simply attaching the MDF w/o a corresponding LDF will result in most cases, in a corrupt database. In addition to being incorrect, MDF distribution is inefficient (BAK files are smaller than the corresponding MDF because they do not contain unallocated pages) and also MDF manipulation requires placing the database offline during the file copy.
what's the best way to dump a large(terabytes) db? are there other faster/efficient way besides mysqldump? this is intended to be zipped, unzipped, and then reimported into another mysql db on another server.
If it's possible for you to stop the database server, the best way is probably for you to:
Stop the database
Do a file copy of the files (including appropriate transaction logs, etc) to a new file system.
Restart the database.
Then move the copied files to the new server and bring up the database on top of the files. It's a bit complicated to do this, but it's by far the fastest way.
I used to be a DBA for a terabyte+ database in MySQL and this is one of the ways we'd do nightly backups of the database. mysqldump would've never worked for data that large. We'd stop the database each night and file copy the underlying files.
Since your intent seems to be having two copies of the DB, why not set up replication to do this?
That will ensure that both copies of the DB remain in an identical state (in terms of data anyway).
And, if you want a snapshot to be exported, you can:
wait for a quiet time.
disable replication.
back up the slave copy.
re-enable replication.
Yesterday I asked the question on how I should save my files.
After some research I've desided to go with storing the files "in" the database.
I've checked the difference between storing the files using filestream and storing the files in the database itself.
Each has it's advantages and disadvantages. To help me with my research this site helped me out a lot:
http://www.codeproject.com/KB/database/SqlFileStream.aspx
So basically it says that saving the files using filestream is better if the files are bigger than 1mb.
But I've discovered another problem with filestreaming. If you delete a record in the database the file still exists on the filesystem.
Therefore I need you guys opinion.
What to use? Filestream or saving the files in the database using VARBINARY?
Grtz,
M.
The data on the filesystem will (should) be removed soon after you delete the data from the database, though this is done in a separate system background thread, so it may remain on the filesystem until basically the garbage collector runs again. However, all results accessed through any of the filestream APIs (i.e. tsql or streaming) will be guaranteed to not access anything that has been removed, whether or not the filesystem data still resides on disk (ACID is ensured with filestream).
Use Filestreams.
Databases are designed to store relational data, file systems are designed to store files.
I heard often from Microsoft employees that storing large blobs in an SQL Server 2000/2005 (cant remember) is not a good idea.
Also consider Backup: A Databasefile is ONE file (EDIT: If you configure it so). If you're using Filestreams you can backup individual files.