deploy to GAE flexible environment stuck forever? - google-app-engine

I try to deploy a simple nodejs app to GAE flexible environment.
Followed the official guide, using this command:
gcloud app deploy --verbosity=debug
I tried a lot of times.
The logs give me these forever:
DEBUG: Operation [apps/just-aloe-212502/operations/b1e812f6-299c-438e-b335-e35aa343242a] not complete. Waiting to retry.
Updating service [flex-env-get-started] (this may take several minutes)...⠛DEBUG: Operation [apps/just-aloe-212502/operations/b1e812f6-299c-438e-b335-e35aa343242a] not complete. Waiting to retry.
Updating service [flex-env-get-started] (this may take several minutes)...⠛DEBUG: Operation [apps/just-aloe-212502/operations/b1e812f6-299c-438e-b335-e35aa343242a] not complete. Waiting to retry.
Updating service [flex-env-get-started] (this may take several minutes)...⠹DEBUG: Operation [apps/just-aloe-212502/operations/b1e812f6-299c-438e-b335-e35aa343242a] not complete. Waiting to retry.
Updating service [flex-env-get-started] (this may take several minutes)...⠼DEBUG: Operation [apps/just-aloe-212502/operations/b1e812f6-299c-438e-b335-e35aa343242a] not complete. Waiting to retry.
What happened?
I can run my simple nodejs hello-world app successfully in local. And, the GAE standard environment works fine.

I should note that App Engine Flexible environment is based on Google Compute Engine, so it takes time to configure the infrastructure when you deploy your app.
The first deployment of a new version of an App Engine Flexible application takes some time due to setting up of internal infrastructure however subsequent deployments should be relatively fast since it only modifies some GCP resources and then waits on the health checks.
Deployment requires docker image building (which you can skip if you already have a pre-built image uploaded to gcr.io). Using a pre-build (to gcr.io) docker image will skip docker build step and would optimize the deployment time.

Related

App Engine Deployment with Cloud Build failed with an "operation is already in progress" error

I try to deploy a new version of app engine service with google cloud build with following steps:
deploy maintenance dispatch.yml to route all requests to maintenance page
Upgrade database
deploy new version
deploy dispatch.yml to route requests back to default service
The first three steps are working, but step 4 results in the following error:
an operation is already in progress
Screenshot of error message
The running GAE process is the one which is stopping the previous version.
So how can I find the running process and wait until it is stopped before
I deploy the dispatch.yml?
Per the documentation, to migrate traffic from one version to another, you should use the set-traffic command. So I think your step 4 should be replaced by the set-traffic command
I could solve the problem by myself with the following statement:
gcloud app operations wait $(gcloud app operations list --format="value(id)" --pending --limit=1) || true
This would wait for a running operation. In my case I had to add this line twice because there where to running operations to wait for.

How to deploy a new version of a Google App Engine production server, without stopping old versions?

I'm running a Google App Engine production server, using basic_scaling as the scale type. Whenever I update the code and deploy it - using gcloud app deploy - the old version of the code is shutdown.
According to the documentation, that's expected:
The shutdown process might be triggered by a variety of planned and unplanned events, such as:
You manually stop an instance.
You deploy an updated version to the service.
...
I understand that it's easier for most developers that way. But in my case, I'd like to keep the old versions running until the idle_timeout limit is reached. Does anyone know if there's a way to avoid the automatic shutdown and let the old versions to shutdown by themselves?
Per Google's documentation, when you deploy your code, the default flag of --stop-previous-version is used. This forces the previous version to be stopped. If you do not want that, you should explicitly use the --no-stop-previous-version in your deploy command (we also have this as a feature on our App, a GUI for GAE; you check or uncheck a checkbox).
Unfortunately, Google does not provide a way for the service to automatically shut down later. You'll have to manually shut it down and start the other version later.

How can I warm up an AppEngine Flex app after a deployment?

It appears that AppEngine standard has a warmup feature to warm up an app after a deployment but I don't see the same feature available for Flex. The readiness & liveness probes also don't work for this since setting the path setting to a custom path inside the application doesn't seem to make the probes actually hit the internal endpoint.
Is there some solution I'm missing other than doing things like manually hitting the endpoints myself after the deployment which won't be very reliable since the calls don't necessarily always round robin to each instance?
In App Engine Standard, warmup requests essentially load your app's code into a new instance before any live requests reach that instance. This can happen in the following situations:
When you redeploy a version of your app.
When new instances are created due to the load from requests
exceeding the capacity of the current set of running instances.
When maintenance and repairs of the underlying infrastructure or
physical hardware occur
In App Engine Flexible, you can achieve the same result by using the initial_delay_sec setting for liveness checks in your app.yaml file. If you set up its value to give enough time for your code to initialize, the first request coming to that instance will be processed quickly by your already-initialized code.

How to run a non-web-server long running Nodejs process in Google App Engine?

I created a Nodejs program that is intended to be executed as a long-running process unattended. The program works on localhost when running npm start. I need to be able to 'tail -f' its stdout on demand. There is no port it listens to. Does Google App Engine support this use case? If so, how to configure it?
I have deployed the app with following app.yaml file but it doesn't seem to work. At least the stdout of the instance is empty/unexpected. I believe the reason is App Engine is designed for web server and this process failed to pass health check of some sort.
runtime: nodejs
vm: true
btw, on Heroku this use case is clearly supported because it has two types of dyno: web and worker. Obviously worker dyno is the choice.

Google Compute Engine keeps spawning instances, then deploy says no CPU's available

I'm just going through the node.js tutorials with a free trial account, and i'm stuck on the second one where you add a db. I add the mongodb deployment, shows up as a VM instances, fine. And my first deploy worked, but now that i'm trying to edit stuff, my deploy's keep failing.
The error i get is that I've exceeded my CPU quota. Watching the list of VM Instances under Compute Engine, i see it keeps spawning up instances, even though the app isn't being used. Guessing it just spins up 8 instances by default?
But then i guess the build system needs its own VM's, but the CPU capacity is used up, so none available to do subsequent builds?! I feel like i'm missing something...
Also, i see i can explicitly start VM's myself, so what process is creating them form me? And can i turn it off? or set a cap on number of instances it spawns?
Can i tell my project to only use 4
Also, the deploy takes forever, is that normal? Following the tutorials, so far I've only seen this command to deploy:
gcloud preview app deploy app.yaml --set-default
Is there another command that does an incremental deploy or something?
By using gcloud preview app deploy you're actually using Managed VMs which is an App Engine runtime which in turn runs Docker containers on Google Compute Engine, which it creates on its own. In other words, you're not using Google Compute Engine directly.
To get rid of extra VMs, you need to delete old app versions: navigate to Compute > App Engine > Versions and delete the versions you don't want.
See also this answer for more details and suggestions.

Resources