I have a deployed app engine service with a flexible environment which I'm trying to update, by running from the terminal:
gcloud app deploy
I am met with this error message:
ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: The following quotas were exceeded: BACKEND_SERVICES (quota: 5, used: 5 + needed: 1), IN_USE_ADDRESSES (quota: 8, used: 8 + needed: 1).
I have only one other service running with a flexible environment.
In the past, I fixed this problem by stopping, then deleting older versions of my service (or of the other one running on the flexible environment).
This problem was caused now because I deleted a version without stopping it first. I have a feeling these versions still occupy my "quota", though I can't see them anywhere (not in the web GUI, nor in the terminal).
Please help!
This issue is related to this thread wherein there are options that can be followed to stop the error.
It is possible that the backend load balancer or the VM instances of your App Engine Flex still exist even if you have removed the whole service. You can verify it through the following:
Load Balancer (Cloud Console -> Network Services -> Load balancing -> Backends) https://console.cloud.google.com/net-services/loadbalancing/backends/list
VM Instances (Cloud Console -> Compute Engine -> VM Instances)
https://console.cloud.google.com/compute/instances
If there is no existing backend, you can file a support ticket to verify if:
Your project quotas are still maxed out.
The service backends are still active only that they do not reflect in Cloud Console.
Related
I have an existing Laravel site which I would like to host on Google app Engine. I've created the app.yaml file and made changes in the composer.json file. When I run the command gcloud app deploy , I get the operation timed out Error. I've updated the config using gcloud config set app/cloud_build_timeout 1000 but still no luck.
It is possible that your In-use IP addresses quota in the region of your App Engine Flexible application has reached its limit. You can check your in-use addresses by clicking here and will be able to increase this by clicking the 'Edit Quotas' button in the Cloud Console.
To find out whether this is the issue causing the error, you can do so by going to the “Activity” tab of your project home page. You may have warnings about quota limits and VMs failing to be created. As App Engine by default leave previous versions up and running that may taking up the ip addresses. You can delete the previous versions and/or request an increase of your IP address quota limit. It is also advised to use the up to date gcloud tools and SDK which may resolve the issue.
If you follow the steps above but are still having trouble, I suggest that you create a new private issue in the Public Issue Tracker and provide the contents of your app.yaml file, Error logs and Project ID. Google Cloud Support team will look further into the matter.
Is it possible to implement a custom health-check for instances in App Engine Standard? I tried to configure liveness-check and readiness-check, but these seem to be supported only in flex environment.
The reason I would like to do this is that recently we ran into following issue several times: because of a bug, one of the instances stopped responding successfully to requests, always returning HTTP status 500. This instance has been then kept alive and accessible for hours, leading to intermittent failures for our customers.
There is a very similar question (or even the same) in Google Groups, you can check it in this link.
I quote the response:
You cannot implement custom health checks in Google App Engine (GAE) Standard Environment. Our public documentation shows that adding legacy and updated health checks you are referring to is possible in Custom Runtime apps (i.e. GAE Flex apps) by configuring the app.yaml file [ 1]. This is an advantage of switching to GAE Flex.
For App Engine Standard, which doesn't afford you that flexibility, hardware and software failures that cause early termination or frequent restarts can occur without prior warning. This sometimes manifests as a non-responsive instance, returning HTTP status 500. We advised that you construct your GAE Standard apps to be able to handle this [2]. And you can reference this documentation for configuring your app.yaml file for your GAE app's app.yaml.
[ 1] https://cloud.google.com/appengine/docs/flexible/custom-runtimes/configuring-your-app-with-app-yaml
[2] https://cloud.google.com/appengine/docs/standard/go/how-instances-are-managed#instance_uptime
[3] https://cloud.google.com/appengine/docs/standard/python/config/appref
In my app (Google App Engine Standard Python 2.7) I have some flags in global variables that are initialized (read values from memcache/Datastore) when the instance start (at the first request). That variables values doesn't change often, only once a month or in case of emergencies (i.e. when google app engine Taskqueue or Memcache service are not working well, that happened not more than twice a year as reported in GC Status but affected seriously my app and my customers: https://status.cloud.google.com/incident/appengine/15024 https://status.cloud.google.com/incident/appengine/17003).
I don't want to store these flags in memcache nor Datastore for efficiency and costs.
I'm looking for a way to send a message to all instances (see my previous post GAE send requests to all active instances ):
As stated in https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed
Note: Targeting an instance is not supported in services that are configured for auto scaling or basic scaling. The instance ID must be an integer in the range from 0, up to the total number of instances running. Regardless of your scaling type or instance class, it is not possible to send a request to a specific instance without targeting a service or version within that instance.
but another solution could be:
1) Send a shutdown message/command to all instances of my app or a service
2) Send a restart message/command to all instances of my app or service
I use only automatic scaling, so I'cant send a request targeted to a specific instance (I can get the list of active instances using GAE admin API).
it's there any way to do this programmatically in Python GAE? Manually in the GCP console it's easy when having a few instances, but for 50+ instances it's a pain...
One possible solution (actually more of a workaround), inspired by your comment on the related post, is to obtain a restart of all instances by re-deployment of the same version of the app code.
Automated deployments are also possible using the Google App Engine Admin API, see Deploying Your Apps with the Admin API:
To deploy a version of your app with the Admin API:
Upload your app's resources to Google Cloud Storage.
Create a configuration file that defines your deployment.
Create and send the HTTP request for deploying your app.
It should be noted that (re)deploying an app version which handles 100% of the traffic can cause errors and traffic loss due to:
overwriting the app files actually being in use (see note in Deploying an app)
not giving GAE enough time to spin up sufficient instances fast enough to handle high income traffic rates (more details here)
Using different app versions for the deployments and gradually migrating traffic to the newly deployed apps can completely eliminate such loss. This might not be relevant in your particular case, since the old app version is already impaired.
Automating traffic migration is also possible, see Migrating and Splitting Traffic with the Admin API.
It's possible to use the Google Cloud API to stop all the instances. They would then be automatically scaled back up to the required level. My first attempt at this would be a process where:
The config item was changed
The current list of instances was enumerated from the API
The instances were shutdown over a time period that allows new instances to be spun up and replace them, and how time sensitive the config change is. Perhaps close on instance per 60s.
In terms of using the API you can use the gcloud tool (https://cloud.google.com/sdk/gcloud/reference/app/instances/):
gcloud app instances list
Then delete the instances with:
gcloud app instances delete instanceid --service=s1 --version=v1
There is also a REST API (https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1/apps.services.versions.instances/list):
GET https://appengine.googleapis.com/v1/{parent=apps/*/services/*/versions/*}/instances
DELETE https://appengine.googleapis.com/v1/{name=apps/*/services/*/versions/*/instances/*}
I searched the web for a while, but can't seem to find the right answer.
I have created a VM Instance on Google Compute Engine and I am running Jenkins on it. This instance checks a code repository I have and when a change occurs, I want to run the following command
gcloud --project=test preview app deploy -q app.yaml --version=dev
When I want to trigger a build, or it triggers it by himself, I get the following error:
Beginning deployment...
ERROR: Error Response: [403] Request had insufficient authentication scopes.
ERROR: (gcloud.preview.app.deploy) Could not retrieve the default Google Cloud Storage bucket for [test].
Please try again or use the [bucket] argument.
The VM instance does have acces to the storage as you can see in the following image, so I don't understand why this error pops up.
Go to the App Engine developer console https://appengine.google.com/ and in
Application Settings > Cloud integration > and turn ON the Google Cloud Storage
so this will set up the default bucket for you. This something that the new Google Developer Console will do automatically.If the project is created on the old console you have to do it manually.
I'm just going through the node.js tutorials with a free trial account, and i'm stuck on the second one where you add a db. I add the mongodb deployment, shows up as a VM instances, fine. And my first deploy worked, but now that i'm trying to edit stuff, my deploy's keep failing.
The error i get is that I've exceeded my CPU quota. Watching the list of VM Instances under Compute Engine, i see it keeps spawning up instances, even though the app isn't being used. Guessing it just spins up 8 instances by default?
But then i guess the build system needs its own VM's, but the CPU capacity is used up, so none available to do subsequent builds?! I feel like i'm missing something...
Also, i see i can explicitly start VM's myself, so what process is creating them form me? And can i turn it off? or set a cap on number of instances it spawns?
Can i tell my project to only use 4
Also, the deploy takes forever, is that normal? Following the tutorials, so far I've only seen this command to deploy:
gcloud preview app deploy app.yaml --set-default
Is there another command that does an incremental deploy or something?
By using gcloud preview app deploy you're actually using Managed VMs which is an App Engine runtime which in turn runs Docker containers on Google Compute Engine, which it creates on its own. In other words, you're not using Google Compute Engine directly.
To get rid of extra VMs, you need to delete old app versions: navigate to Compute > App Engine > Versions and delete the versions you don't want.
See also this answer for more details and suggestions.