I have been trying to get an AppEngine project to simply stop running. After trying deleting versions, deleting instances, and even uploading straight up empty main.py and worker.py files the project is still using about 3 hours of instance hours per hour. I don't understand how this is physically possibly. Where are some places I can start looking, since where I've been looking before doesn't seem to have any relevance whatsoever.
One possible approach would be to disable (or even shutdown) your application. From Google App Engine FAQ:
How can I disable one of my existing applications?
Disabling your application stops all serving requests, but your data and state are retained. You are still be billed for applicable
charges, such as Compute Engine instances. To release all the
resources used within the project, shut down your project.
To disable your application:
In the GCP Console, go to the App Engine Settings page.
Click Disable application and follow the instructions.
Disabling your app takes effect immediately. Confirm that your application has been disabled by visiting the URL of your app, such as
http://[YOUR_PROJECT_ID].appspot.com/. Your application should
return an HTTP 404 Not Found error.
Note that shutting the project down will be automatically followed by deletion in 30 days, so don't do that if you still want to re-enable the project at some point.
You can set the max instances in app.yaml:
instance_class: B2
basic_scaling:
max_instances: 1
Not sure if you can set that to 0, but at least limit it to 1.
Also
1) Make sure you don't have some backend instance (which uses its own app.yaml) running.
2) make sure you don't have any cron jobs running, or tasks stuck in the taskqueue
3) Try:
health_check:
enable_health_check: False
4) Shut down your instances.
Related
I have a python app engine that handles api results and it's stateful. However it seems that after a few hours of inactivity (no requests), the server shuts off, resetting all states, and when a new request is made, it's listening again.
But the states are reset. I want the server to actively remain unchanged 24/7 and not reset/restart as I want to maintain states.
I have configured as per documentation but it's still restarting, I am not sure what's wrong
Here is my app.yaml:
runtime: python37
entrypoint: python main.py
manual_scaling:
instances: 1
In App Engine the general recomendation is to create stateless applications as mentioned on the documentation
Your app should be "stateless" so that nothing is stored on the instance.
As an alternative for the application not to get restarted you can deploy it on Compute Engine, As that service is a Virtual Machine you can have total control of the states.
It appears that AppEngine standard has a warmup feature to warm up an app after a deployment but I don't see the same feature available for Flex. The readiness & liveness probes also don't work for this since setting the path setting to a custom path inside the application doesn't seem to make the probes actually hit the internal endpoint.
Is there some solution I'm missing other than doing things like manually hitting the endpoints myself after the deployment which won't be very reliable since the calls don't necessarily always round robin to each instance?
In App Engine Standard, warmup requests essentially load your app's code into a new instance before any live requests reach that instance. This can happen in the following situations:
When you redeploy a version of your app.
When new instances are created due to the load from requests
exceeding the capacity of the current set of running instances.
When maintenance and repairs of the underlying infrastructure or
physical hardware occur
In App Engine Flexible, you can achieve the same result by using the initial_delay_sec setting for liveness checks in your app.yaml file. If you set up its value to give enough time for your code to initialize, the first request coming to that instance will be processed quickly by your already-initialized code.
I've deployed a Node.js application on Google App Engine (standard environment).
I've noticed that after 10 minutes of inactivity, the app is undeployed and the number of instances goes to 0.
So the first request I make takes 4-5 seconds to reply.
This is my app.yaml
runtime: nodejs10
service: backend
automatic_scaling:
min_instances: 1
I added also min_idle_instances, but the issue seems not solved:
You can use ‘min_idle_instances’ instead of ‘min_instances’.
When using ‘min_instances’, you define how many instances you would like spun up when your app receives traffic.
When you use ‘min_idle_instances’, you define how many instances you want to keep alive. These instances are kept idle and running in the background in order to receive traffic.
Do note that it may increase your monthly invoice as those instances are live, whether or not they are receiving traffic.
I switched from standard to flexible environment, and it seems really better
In my app (Google App Engine Standard Python 2.7) I have some flags in global variables that are initialized (read values from memcache/Datastore) when the instance start (at the first request). That variables values doesn't change often, only once a month or in case of emergencies (i.e. when google app engine Taskqueue or Memcache service are not working well, that happened not more than twice a year as reported in GC Status but affected seriously my app and my customers: https://status.cloud.google.com/incident/appengine/15024 https://status.cloud.google.com/incident/appengine/17003).
I don't want to store these flags in memcache nor Datastore for efficiency and costs.
I'm looking for a way to send a message to all instances (see my previous post GAE send requests to all active instances ):
As stated in https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed
Note: Targeting an instance is not supported in services that are configured for auto scaling or basic scaling. The instance ID must be an integer in the range from 0, up to the total number of instances running. Regardless of your scaling type or instance class, it is not possible to send a request to a specific instance without targeting a service or version within that instance.
but another solution could be:
1) Send a shutdown message/command to all instances of my app or a service
2) Send a restart message/command to all instances of my app or service
I use only automatic scaling, so I'cant send a request targeted to a specific instance (I can get the list of active instances using GAE admin API).
it's there any way to do this programmatically in Python GAE? Manually in the GCP console it's easy when having a few instances, but for 50+ instances it's a pain...
One possible solution (actually more of a workaround), inspired by your comment on the related post, is to obtain a restart of all instances by re-deployment of the same version of the app code.
Automated deployments are also possible using the Google App Engine Admin API, see Deploying Your Apps with the Admin API:
To deploy a version of your app with the Admin API:
Upload your app's resources to Google Cloud Storage.
Create a configuration file that defines your deployment.
Create and send the HTTP request for deploying your app.
It should be noted that (re)deploying an app version which handles 100% of the traffic can cause errors and traffic loss due to:
overwriting the app files actually being in use (see note in Deploying an app)
not giving GAE enough time to spin up sufficient instances fast enough to handle high income traffic rates (more details here)
Using different app versions for the deployments and gradually migrating traffic to the newly deployed apps can completely eliminate such loss. This might not be relevant in your particular case, since the old app version is already impaired.
Automating traffic migration is also possible, see Migrating and Splitting Traffic with the Admin API.
It's possible to use the Google Cloud API to stop all the instances. They would then be automatically scaled back up to the required level. My first attempt at this would be a process where:
The config item was changed
The current list of instances was enumerated from the API
The instances were shutdown over a time period that allows new instances to be spun up and replace them, and how time sensitive the config change is. Perhaps close on instance per 60s.
In terms of using the API you can use the gcloud tool (https://cloud.google.com/sdk/gcloud/reference/app/instances/):
gcloud app instances list
Then delete the instances with:
gcloud app instances delete instanceid --service=s1 --version=v1
There is also a REST API (https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1/apps.services.versions.instances/list):
GET https://appengine.googleapis.com/v1/{parent=apps/*/services/*/versions/*}/instances
DELETE https://appengine.googleapis.com/v1/{name=apps/*/services/*/versions/*/instances/*}
I'm just going through the node.js tutorials with a free trial account, and i'm stuck on the second one where you add a db. I add the mongodb deployment, shows up as a VM instances, fine. And my first deploy worked, but now that i'm trying to edit stuff, my deploy's keep failing.
The error i get is that I've exceeded my CPU quota. Watching the list of VM Instances under Compute Engine, i see it keeps spawning up instances, even though the app isn't being used. Guessing it just spins up 8 instances by default?
But then i guess the build system needs its own VM's, but the CPU capacity is used up, so none available to do subsequent builds?! I feel like i'm missing something...
Also, i see i can explicitly start VM's myself, so what process is creating them form me? And can i turn it off? or set a cap on number of instances it spawns?
Can i tell my project to only use 4
Also, the deploy takes forever, is that normal? Following the tutorials, so far I've only seen this command to deploy:
gcloud preview app deploy app.yaml --set-default
Is there another command that does an incremental deploy or something?
By using gcloud preview app deploy you're actually using Managed VMs which is an App Engine runtime which in turn runs Docker containers on Google Compute Engine, which it creates on its own. In other words, you're not using Google Compute Engine directly.
To get rid of extra VMs, you need to delete old app versions: navigate to Compute > App Engine > Versions and delete the versions you don't want.
See also this answer for more details and suggestions.