Run appengine backup from task queue - google-app-engine

Question

In the 1.8.4 release of Google App Engine it states:
A Datastore Admin fix in this release improves security by ensuring that scheduled backups can now only be started by a cron or task queue task. Administrators can still start a backup by going to the Datastore Admin in the Admin Console.
The way to run scheduled backups with cron is documented, but how can we initiate backups from a task queue task?
Are there any other ways to programmatically run backup tasks?

Answer 1

You can create a task queue task with method GET and URL "/_ah/datastore_admin/backup.create" with your parameters specified as arguments to the URL and target the task to run on the 'ah-builtin-python-bundle' version. Example:
url = '/_ah/datastore_admin/backup.create?filesystem=blobstore&name=MyBackup&kind=Kind1&kind=Kind2'
taskqueue.add(
url=url,
target='ah-builtin-python-bundle',
method='GET',
)
I have cron jobs that trigger my own handlers that then look up a config and create a task queue backup based on that config. This lets me change backup settings without having to update cron jobs and lets me have a lot more control.
The options you can specify in the URL are the same as the documentation describes for CRON job backups so you can specify namespace and gs-bucket-name as well.
I believe to do this in Java you have to create a queue with a target in the queue definition and add your tasks to that queue.

Answer 2

I did this by combining Bryce' solution with the code from googles scheduled backup documentation. This way, I'm still using cron.yaml but I have the flexibility for differences in each environment (e.g. don't run a backup in the dev/stage branch based on config in the datastore, don't specify types in the URL that haven't made it out of dev yet).
I also was able to generate the &kind=xxx pairs using this:
from google.appengine.ext.ndb import metadata
backup_types = "".join(["&kind=" + kind for kind in metadata.get_kinds() if not kind.startswith("_")])
The steps were pretty simple in retrospect.
Setup:
a) Enable your default cloud storage bucket
b) Enable datastore admin
Steps:
Add a cron job to kick off the backup (cron.yaml):
cron:
- description: daily backup
url: /taskqueue-ds-backup/
schedule: every 24 hours from 21:00 to 21:59
Add a queue to process the tasks (queue.yaml):
- name: ds-backup-queue
rate: 1/s
retry_parameters:
task_retry_limit: 1
Add a route to the task queue handler:
routes = [...,
RedirectRoute('/taskqueue-ds-backup/',
tasks.BackupDataTaskHandler,
name='ds-backup-queue', strict_slash=True), ...]
Add the handler to process the enqueued items:
from google.appengine.api import app_identity
from google.appengine.api import taskqueue
from google.appengine.ext.ndb import metadata
import config
class BackupDataTaskHandler(webapp2.RequestHandler):
def get(self):
enable_ds_backup = bool(config.get_config_setting("enable_datastore_backup", default_value="False"))
if not enable_ds_backup:
logging.debug("skipping backup. backup is not enabled in config")
return
backup_types = "".join(["&kind=" + kind for kind in metadata.get_kinds() if not kind.startswith("_")])
file_name_prefix = app_identity.get_application_id().replace(" ", "_") + "_"
bucket_name = app_identity.get_default_gcs_bucket_name()
backup_url = "/_ah/datastore_admin/backup.create?name={0}&filesystem=gs&gs_bucket_name={1}{2}".format(file_name_prefix, bucket_name, backup_types)
logging.debug("backup_url: " + backup_url)
# run the backup as a service in production. note this is only possible if you're signed up for the managed backups beta.
if app_identity.get_application_id() == "production-app-id":
backup_url += "&run_as_a_service=T"
taskqueue.add(
url=backup_url,
target='ah-builtin-python-bundle',
method='GET',
)
logging.debug("BackupDataTaskHandler completed.")