I got a Google AppEngine Standard app running in region1 and want to deploy that same app as a backup region2 in case region1 is down. I'm looking for a way to make that failover happens seemlessly for my users (both human users on browsers and other third party services which call back to my app's service).
Currently I have my custom domain name (both naked name and www name) mapped to the app on region #1 (done in [Google Cloud Console][App Engine][Settings][Customer Domains]).
In the event region1 is down, I would like to go in that setting area of app1(region1), remove those maps and then go that setting area of app2(region2) add those maps, so that after that point, requests to myappdomainname.com and www.myappdomainname.com will go to app2 on region2
Question: is that plan feasible? In particular, if region1 is down, can I still be able to access app1's setting area to remove those maps, so that I can add the maps to app2?
Down time while switching these for about an hour is okay for my app, as long as my users can continue to use the same URL they been using when region1 was still running.
Google App Engine is a regional service, meaning that it cannot span to more than a region. However, it's replicated through all zones of the region to reduce any potential downtime.
The kind of implementation you want for GAE is opposite of the actual purpose of it. One of GAE's principal features is that you don't have to configure and manage the instances running in the background yourself.
The preferred way of getting this to work on Google Cloud Platform would be using Compute Engine. GCE gives you the option to create the instances in any region you want and configure a Load Balancer to serve the traffic and scale your instances as you want. Here is some documentation about serving applications using GCE:
Running a GO app on Compute Engine (part of a GCP quickstart)
Building Scalable and Resilient Web Applications on Google Cloud Platform
Designing Robust Systems
Also, here's a Google Groups post about this issue that goes a little bit more in detail.
Related
Being new to GCP, I have a question about which architecture to use in a particular case.
Suppose I have a Django website running on the App engine (flexible environment?). Users upload images to the website. I would like to first use Google Vision API to perform some label detection on the images and then feed the labels and images to a VM with GPU attached (all running on Google cloud), for additional computationally costly job on the images. After the job is completed by the VM, the resulting images are then available for the user to download or sent to the user email.
Because of the relatively large time spent on the VM+GPU side, and because the website will be accessed by users globally, I would like to reduce the overall latency time and pick the most efficient architecture for the job.
My first thought was to:
upload images to Google Cloud Storage;
use GC functions to perform some quick transformations and then call Google Vision API;
pull the resulting labels and transformed images to the VM and make computations on the VM side;
upload finalized images to Google Cloud Storage.
Now, that's a lot of bouncing back and forth between a storage bucket and APP engine plus VM on either side. I was wondering if there is a 1) quicker and 2) more efficient resources-wise way to achieve the same goal.
If your website is accessed globally, your App Engine choice is the wrong one: App Engine can be deployed in only one region, not globally.
For the frontend, I recommend to use Cloud Run instead (or VM, but I don't like VM) and to put a HTTPS load balancer in front of. Like that, the physical latency is reduced.
And, the files must be also store in the closest region, so in Cloud Storage in different region.
And finally, to duplicate the VM/GPU infrastructure in each region (it could be costly, but it's the best way to reduce latency.
Your process is the right one. I recommend you to expose an API on your VM to notify it when a file is ready. You can use the PubSub notification on Cloud Storage to sink the event in PubSub, and then create a push subscription to invoke your VM directly (instead of a cloud functions).
Like that, you remove a component and you perform all your processing on the VM side.
In my app (Google App Engine Standard Python 2.7) I have some flags in global variables that are initialized (read values from memcache/Datastore) when the instance start (at the first request). That variables values doesn't change often, only once a month or in case of emergencies (i.e. when google app engine Taskqueue or Memcache service are not working well, that happened not more than twice a year as reported in GC Status but affected seriously my app and my customers: https://status.cloud.google.com/incident/appengine/15024 https://status.cloud.google.com/incident/appengine/17003).
I don't want to store these flags in memcache nor Datastore for efficiency and costs.
I'm looking for a way to send a message to all instances (see my previous post GAE send requests to all active instances ):
As stated in https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed
Note: Targeting an instance is not supported in services that are configured for auto scaling or basic scaling. The instance ID must be an integer in the range from 0, up to the total number of instances running. Regardless of your scaling type or instance class, it is not possible to send a request to a specific instance without targeting a service or version within that instance.
but another solution could be:
1) Send a shutdown message/command to all instances of my app or a service
2) Send a restart message/command to all instances of my app or service
I use only automatic scaling, so I'cant send a request targeted to a specific instance (I can get the list of active instances using GAE admin API).
it's there any way to do this programmatically in Python GAE? Manually in the GCP console it's easy when having a few instances, but for 50+ instances it's a pain...
One possible solution (actually more of a workaround), inspired by your comment on the related post, is to obtain a restart of all instances by re-deployment of the same version of the app code.
Automated deployments are also possible using the Google App Engine Admin API, see Deploying Your Apps with the Admin API:
To deploy a version of your app with the Admin API:
Upload your app's resources to Google Cloud Storage.
Create a configuration file that defines your deployment.
Create and send the HTTP request for deploying your app.
It should be noted that (re)deploying an app version which handles 100% of the traffic can cause errors and traffic loss due to:
overwriting the app files actually being in use (see note in Deploying an app)
not giving GAE enough time to spin up sufficient instances fast enough to handle high income traffic rates (more details here)
Using different app versions for the deployments and gradually migrating traffic to the newly deployed apps can completely eliminate such loss. This might not be relevant in your particular case, since the old app version is already impaired.
Automating traffic migration is also possible, see Migrating and Splitting Traffic with the Admin API.
It's possible to use the Google Cloud API to stop all the instances. They would then be automatically scaled back up to the required level. My first attempt at this would be a process where:
The config item was changed
The current list of instances was enumerated from the API
The instances were shutdown over a time period that allows new instances to be spun up and replace them, and how time sensitive the config change is. Perhaps close on instance per 60s.
In terms of using the API you can use the gcloud tool (https://cloud.google.com/sdk/gcloud/reference/app/instances/):
gcloud app instances list
Then delete the instances with:
gcloud app instances delete instanceid --service=s1 --version=v1
There is also a REST API (https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1/apps.services.versions.instances/list):
GET https://appengine.googleapis.com/v1/{parent=apps/*/services/*/versions/*}/instances
DELETE https://appengine.googleapis.com/v1/{name=apps/*/services/*/versions/*/instances/*}
Is it possible for my GAE to be balanced accross regions? Like US and Europe.
https://cloud.google.com/about/locations/
Reading the link above it says that it scales automatically within or accross multi-regional locations.
What does that mean? Do I have to enable automatic scaling across regions? If so, how do I do that?
And secondly, if it does handle automatic scaling across regions, the choice one makes for App engine location when creating a new project, is that irrelevant for a Google App Engine instance?
See this from Google Groups: https://groups.google.com/forum/m/#!topic/google-appengine/svMhwuFC5zk
The short answer today is no it cannot be accomplished in App Engine as is.
But, if you feel like hacking, it seems the above thread is suggesting setting up a load balancer to serve Compute Engine and then have these instances route to your "different" projects with App Engine deployed in your various regions (i.e. one deployment in a region per project). Seems like a mess, but possible.
There may be light at the end of the tunnel (eventually): https://issuetracker.google.com/issues/65039982
You cannot change the region. Your app will be served from the region you chose when creating the app. Anyone can use the app, but users closer to the selected region will have lower latency.
While the App Engine service itself operates in multiple regions, an App Engine app is served from a single region. The app will, however, scale across multiple zones within that region.
See App Engine Locations for more.
You can use a potential solution using HTTPS LB with Serverless NEG, Serverless NEG connects internal services directly without GCE. See https://stackoverflow.com/a/62660953/4185100
Yon asked this question on the mailing list, where it was answered. To reiterate what folks said there: App Engine is automatically scaled across multiple regions, but those regions are defined by Google. You'd have to set up another project to handle traffic in the EU, for example.
I'm running an app on a VM instance (instance-1) and would like myproject.appspot.com requests to be served by instance-1.
I read https://cloud.google.com/appengine/docs/java/modules/routing but it wasn't clear. Is there a way to say "send all traffic to my one instance"?
If I go to my (ephemeral) external IP address for that instance, I can see the server. But, that won't work for an oAuth2 domain (no IP addresses allowed), so I need it to go through the named domain.
I'd be ok if I could use something constant like instance-1-dot-myproject.appspot.com but would prefer the base myproject.appspot.com to say "any instances? great! use that."
I think you want to use Managed VMs. They give you the flexibility of Google Compute Engine but work more like the PaaS that is Google App Engine.
You don't create the Google Compute Engine VM instances yourself, however, Managed VMs will spin them up on demand, using the Docker image you provide as the container of the code, data, etc.
Note that as of 29 Sep 2015, per the docs:
Beta
This is a Beta release of Managed VMs. This feature is not covered by any SLA or deprecation policy and may be subject to backward-incompatible changes.
Is that somehow possible to access one datastore? Or access one app from different domains.
App Engine recently add support for a feature called modules (Docs: go, python, java)
App Engine Modules (or just "Modules" hereafter) is a feature that lets developers factor large applications into logical components that can share stateful services and communicate in a secure fashion. An app that handles customer requests might include separate modules to handle other tasks:
API requests from mobile devices
Internal, admin-like requests
Backend processing such as billing pipelines and data analysis
When you create a new module, App Engine will create a url that corresponds to the module name. If you only have one module then the name will be default. e.g.
http://default.myapp.appspot.com
http://mobile‑frontend.myapp.appspot.com
http://my-module.myapp.appspot.com
Using Domain masking, you could then direct from:
www.myapp.com => http://default.myapp.appspot.com
www.myapp-mobile.com => http://mobile‑frontend.myapp.appspot.com
www.example.com => http://my-module.myapp.appspot.com
Every app has its own datastore and memcache (shared among all versions of that app).
It seems not possible to share datastores between applications right now (unless you provide some web service for that), but that would be a nice feature to have, so maybe you should file a feature request with Google vote for it.
As for domains, you can associate your app with domains managed by Google Apps. Multiple domains for the same application should be no problem (except for SSL certificates).
Every version of an app is backed by the same datastore. If you want to limit access for individual requests, you'll need to add a field to your model to enforce that restriction. There are low level hooks in the datastore API for this sort of thing, if you want to go that far.
And yes, you can add a single App Engine app to multiple domains - even in multiple Apps accounts.
Kyle's solution would work, but App Engine was never designed to be used in this way. So if you architect your app(s) to rely on this kind of setup and Google clamps down for whatever reason then you'd be screwed.
You can have multitenancy using the Namespace Java API