We have a service running on Google App Engine.
If that service does not receive a traffic for some time then all instances are killed and the next call takes a few additional seconds to start the application.
We are thinking about specifying a min_instances option in app.yaml to always keep at least one instance alive.
We deploy new versions of that service quite frequently and keeping old versions for some time. Those old versions are not serving traffic and kept just in case.
What we would like to do is to always keep at least one instance of default service version alive and leave all other non-default versions with default behavior – we want them to be scaled automatically to 0 instances if they do not receive any traffic.
I didn't find such option in the documentation (https://cloud.google.com/appengine/docs/standard/python3/config/appref#scaling_elements) and didn't come to any workarounds.
I am thinking about creating a cron job (https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml) which will periodically "ping" only default version of my application periodically thus making it always asleep. But I am not sure if it is good solution.
Are there any better solutions to such case?
Thanks!
min_idle_instances config option seems to solve my problem.
Note following in the documentation: "This setting only applies to the version that receives most of the traffic" which is almost exactly my case:
automatic_scaling:
min_idle_instances: 1
Related
On our development environment, we have been charged about USD100 every month for an instance we didn't know existed (and of course we are not using), and we cant find in the entire AppEngine or Console Engine.
Also, the usage report shows no activity for the whole month, but we are still getting the charges.
The instance is: Flex Instance Core Hours Sao Paulo
I found similar posts in stackoverflow, so, here are the questions:
- is this some bad strategy from Google???
- where can I see this instance to stop it or delete it?
- where can I see who started this instance and when?
Of course, I called google support and no answer received.
Many thanks!
Google Cloud Platform Support here! I found your ticket and see that you were provided an answer there already. In addition to what Dan described in his answer, if your app has currently the "Serving" status it will still run with the corresponding instances regardless of any requests incoming or not. As long as the version is serving it will continue to bill for hours that you are using. Also, if you are using automatic scaling with a minimum number of instances:
that specified number of instances run as resident instances while any
additional instances are dynamic
(Instance scaling description in GCP docs)
You can use basic or manual scaling if this is not what you're interested in.
Check the App Engine Versions pages for all your projects, you should find at least one with Flexible environment. The Deployed column should indicate who deployed it and when.
Based on that information you can decide to keep or delete the respective version(s). Simply stopping the instance may not be sufficient, depending on the scaling configuration for that service version GAE may automatically start one or more new instances.
You should also check the App Engine Instances pages for your projects and cross-reference that with the versions info to make sure no undesired instances are accidentally left behind (at least in the standard environment they are normally stopped when the respective versions are deleted, not entirely certain the same is true for the flex environment)
The running flexible environment instances are billed by the hour, even if they receive no requests, which could explain why you're seeing charges without any activity.
Apparently, the source of this instance was a firebase setting we made to make some test, and it automatically creates this instance. I shut off the billing account for this space, and instantly I received an email from firebase saying it detected changes that make some functions unavailable.
I am having trouble with using Push Queues on Google App Engine's Flexible Environment (formally named, their Managed VM Environment). I am receiving numerous 404 Instance Unavailable (see picture below).
After a bit of sleuthing, I believe these errors may be because I am adding a task to a task queue, then deploying a new version of the Flexible VM instance. The taskqueue that I previously pushed is locked to the older instance, and can no longer run. Is this how taskqueues work with Flexible VM? If so, how does one use push taskqueues with the Flexible VM?
I was 90% done migrating to flexible env when I came across this same problem. After extensive research, I concluded there are three options:
REST API (experimental)
Use the beta REST API for task queues (this, as all other google APIs from flexible env, is external, so you need to deal with auth appropriately).
REST API reference: https://cloud.google.com/appengine/docs/python/taskqueue/rest/
Note, this is external and experimental. Find e.g. a java sdk without any meaningful documentation here: https://developers.google.com/api-client-library/java/apis/ (current version: https://developers.google.com/api-client-library/java/apis/taskqueue/v1beta2)
Compat runtime
Build your own flexible environment, based off a -compat runtime. This offers the old appengine api in a container suitable for the flexible env:
https://cloud.google.com/appengine/docs/flexible/custom-runtimes/build (look for images with "YES" in the last column)
e.g.: https://cloud.google.com/appengine/docs/flexible/java/dev-jetty9-and-apis
https://cloud.google.com/appengine/docs/flexible/java/migrating-an-existing-app
Note: I spent two weeks in blistered frustration pleading every God almighty help me get this to work, following container rabbit holes into the depths of Lucifer's soul and across unexplored dimensions. I eventually had to give in. I just can't get this to work to a satisfying degree.
Proxy service
Kind of a hacky alternative, but it gets the job done: create a very thin standard environment wrapper service which proxies tasks onto / off your queue. Pass them to your own app however you want. ¯\_(ツ)_/¯
Downside is you are now spinning up extra instances and burning extra minutes.
I ended up with a variation of this, where I'm using a proxy service in standard env, but just ported my eventual task handler to AWS Lambda (so it's completely off GAE). It's a different disaster, but a more manageable one.
Good luck!
I'm the tech lead and manager for this product.
There are two distinct answers to your question.
For the first, it seems like you have a version routing issue -- as you say, tasks cannot run against a VM because you launched a new version. By default, tasks are assigned to run on the version from which they were enqueued to avoid version mismatches. You should be able to override the version by re-configuring the target in your queue.yaml (or queue.xml). Documentation for that can be found here. You might also need to look at your
From a broader perspective, building a migration path away from standard/MVM-only support for task queues is currently our highest priority.
The replacement is Cloud Tasks, which exposes the same interface but can be used fully independently from App Engine. It exists in the same universe as AppEngine Task Queues, so you will be able to add tasks to existing queues (both push and pull). It is currently available in closed alpha. You can sign up to join the alpha here.
We strongly recommend against writing new code against the REST API. It is unsupported and the cloud tasks alpha is already substantially more feature complete.
I'm upvoting hraban's answer (he did wrestle with the devil after all) but providing an additional answer here.
Keep in mind that the Flexible Environment (Managed VMs) is still just a compute engine instance... with Google doing a good job of extracting features from AppEngine to make them reachable in a transparent manner. TaskQueues didn't quite make it. Keep a sharp eye on the cloud library--that's the mechanism by which the DataStore becomes usable (for Java go to http://googlecloudplatform.github.io/google-cloud-java/0.3.0/index.html). If you go to that link you can also select other languages. I have it on official word that TaskQueues are still on the roadmap (but no ETA).
As of now you can't use the REST api to enqueue onto PUSH queues. Now the way that I decided to tackle this problem was to use the REST API and create a PULL queue to put tasks in. Then I poll that queue inside an AppEngine service (i.e. module) and put it into a PUSH task queue. Why do I go to all that trouble? Because I need scheduled execution... which is a feature that TaskQueues alone can give you on AppEngine. So I package my task in an envelope and then unpack and re-push it into a task queue. Sounds crazy? It's been working reliably for me. Don't be scared off by the fact that the REST api is "alpha".
I will say if you're starting something new take a good look at the Pub/Sub API.
I have GAE application with 3 versions. Only default version has active instance, other two versions do not have active instances. Usually after deploy of any version, instance is created. But now that is not the case. Instances of non default versions wont to start after deploy, and I need those versions up.
I saw this question:
Google App Engine instances not starting
but there is no concrete answer.
Any idea how to start version instance?
The version will start after you send it the first request (unless you use Managed VM - https://cloud.google.com/appengine/docs/managed-vms).
It's unclear what you are trying to accomplish by having multiple versions running at the same time, having a little more info would really help providing a specific solution.
As #Vlad stated App Engine won't start (or keep) any app until it gets some traffic. you can start them by hitting ..appspot.com
If this "versions" are actually doing different things you should be implementing them as Modules .This way you can set up different rules for each module and setup the interactions between them.
I want to make sure that when I update my GAE site that it does not have any bad side effects for users who are currently using my site.
Does Google automatically make this a safe process, e.g. by:
complete all pending requests
temporarily delay new requests
update code
process new requests
Or does Google just terminate whatever is running and update the server code?
If the latter, any recommendations for doing a safe update? Any other precautions for tasks in queues?
I think that for some (usually quite short) time, your two versions are serving side-by-side. So some requests are on the new version, but some are still finishing up on the old version. Be sure to consider that when thinking about how your data structures will update.
Occasionally, I have received errors from a small number of users while deploying.
I think the safest way to do it is to deploy to a new 'version' that isn't the active one, then to change your default 'version' in the admin console. This ensures the quickest, cleanest, changeover.
I have a couple of questions about the App Engine Map Reduce API. First of all there's a mapreduce package in the SDK, and there's a separate mapreduce bundle here:
https://developers.google.com/appengine/downloads
Which one should I be using? Should I be using the bundle, or is the documentation out of date and I should actually use the SDK version?
Second I'd like to be able to run mapreduce's on a non-default version to make sure that the requests from the mapreduce don't interfere with user requests.
What's the best way to do this? Can I start the pipeline with a task queue, and set the target version of that queue to be my non-default version?
We recommend using the open source version of Map Reduce for GAE at http://code.google.com/p/appengine-mapreduce/
The stale bundle link in the docs is a bug. That'll get cleaned up soon.
A few of our SDKs have bits of MapReduce (for historic reasons), but the open source version is the way to go for now.
As for using a separate version, this is kind of "it depends". If you're thinking of interference in terms of competition for the processor, that's not likely to be a noticeable issue. Depending on queue processing rates you've set up, more instances of your app will be spun up to handle mapping tasks as needed. I'd try some experiments first. Make sure you have a problem before you invest time and effort solving it.
mapreduce can be start on a not default version. And after it starts, it will continue run on that version automatically.
In my case I just deploy the code on a non default version and trigger the mapreduce with version_id.app_id.appspot.com/path_to_start_a_job.
cron job can also trigger the mapreduce on non default version without problem.