Split large SQL backup - sql-server

Question

I have a large SQL database (~1TB) that I'm trying to backup.
I can back it up fine but we want to store it offsite on Amazon S3, where the maximum object size is 5GB.
I thought I could split it by using multiple files, but it seems the maximum is 64 so I'm still ending up with 16GB chunks which are too big for S3.
Is there any other way to do it?

Answer 1

The maximum blob size for S3 is 5TB, not 5GB. 5GB is only the largest object that can be uploaded with a single HTTP PUT.
All cloud providers follow the same pattern: instead of uploading one huge file and storing it as a single blob, they break it apart into blocks that they replicate across many disks. When you ask for data, the provider retrieves it from all these blocks. To the client though, the blob appears as a single object.
Uploading a large file requires blocks too. Instead of uploading a large file with a single upload operation (HTTP PUT) all providers require that you upload individual blocks and finally notify the provider that these blocks constitute one object. This way, you can re-upload only a single failed block in case of failure, the provider can commit each block while you send the next, they don't have to track and lock a large blob (on a large disk) waiting for you to finish uploading etc.
In your case, you'll have to use an uploader that understands cloud storage and uses multiple blocks, perhaps something like Cyberduck, or S3 specific command-line tools. Or write a utility that uses Amazon's SDK to upload the backup file in parts.
Amazon's documentation site offers examples for multipart uploads at Uploading Objects Using Multipart Upload API. The high-level examples demonstrate various ways to upload a large file. All calls though use multi-part uploads, eg the simplest call :
var client= new AmazonS3Client(Amazon.RegionEndpoint.USEast1);
var fileTransferUtility = new TransferUtility(client);
fileTransferUtility.Upload(filePath, existingBucketName);
will upload the file using multiple parts and use the file's path as its key. The most advanced example allows you to specify the part size, a different key, redundancy options etc:
var fileTransferUtilityRequest = new TransferUtilityUploadRequest
{
BucketName = existingBucketName,
FilePath = filePath,
StorageClass = S3StorageClass.ReducedRedundancy,
PartSize = 6291456, // 6 MB.
Key = keyName,
CannedACL = S3CannedACL.PublicRead
};
fileTransferUtilityRequest.Metadata.Add("param1", "Value1");
fileTransferUtilityRequest.Metadata.Add("param2", "Value2");
fileTransferUtility.Upload(fileTransferUtilityRequest);