We are on App Engine Standard running Python 3. Always used the following command in App Engine to migrate traffic after tests are passed.
gcloud app versions migrate $VERSION --service=“default”
Today around 11 am EET it stopped working. It waits for very long time and than exits with error. Permissions or anything else we can spot are changed.
After some investigation we found out that the following works:
gcloud app services set-traffic default --splits $VERSION=1
But we are afraid it drops all current connections.
We also tried (same as above but with --migrate):
gcloud app services set-traffic default --splits $VERSION=1 --migrate
But it didn't work in the same way as the original command we used.
Any ideas what may have caused this and is it safe to use the "set-traffic" command without --migrate option?
Related
I'm running a Google App Engine production server, using basic_scaling as the scale type. Whenever I update the code and deploy it - using gcloud app deploy - the old version of the code is shutdown.
According to the documentation, that's expected:
The shutdown process might be triggered by a variety of planned and unplanned events, such as:
You manually stop an instance.
You deploy an updated version to the service.
...
I understand that it's easier for most developers that way. But in my case, I'd like to keep the old versions running until the idle_timeout limit is reached. Does anyone know if there's a way to avoid the automatic shutdown and let the old versions to shutdown by themselves?
Per Google's documentation, when you deploy your code, the default flag of --stop-previous-version is used. This forces the previous version to be stopped. If you do not want that, you should explicitly use the --no-stop-previous-version in your deploy command (we also have this as a feature on our App, a GUI for GAE; you check or uncheck a checkbox).
Unfortunately, Google does not provide a way for the service to automatically shut down later. You'll have to manually shut it down and start the other version later.
We've build few services on Google App Engine, we've ship them into production and everything worked as expected. But now I have little issue with one of my integration service which runs on flexible environment. I've deployed serveral times new version and maybe some of deploys wasn't successfully finshed or something like that. Now I see in Stack Driver logging some error logs from running instances of some old version (by version id in "resource" property). It made me few problems with with some version "missmatch" in integration tasks. I don't see this old versions in GC CLI SDK or in web dashboard. I've tried to delete whole service and redeploy it, but after first deploy, there are still new log entries in Stack Driver. I've changed DB credentials to prevent older versions to make mess in DB ¯_(ツ)_/¯. Production environment now runs OK, but there are still some old services spamming logging with errors about DB auth. I cant find this old versioned instances anywhere except this new log entries. Is it normal behaviour? Do you have some tips?
I have read the document which says that gcloud app versions migrate operation can migrate traffics into another version, but it couldn't. The error description seems to say something informative, but no help to me. On GCP control panel, version migration works fine. What's happening on this?
$ gcloud app versions migrate 20190122-120543-ebbfbc9 -s q
Migrating all traffic from version [q/20190129-164614-09e8288] to
[q/20190122-120543-ebbfbc9]
Do you want to continue (Y/n)?
ERROR: (gcloud.app.versions.migrate) Issues migrating all traffic of service(s): [q]
INVALID_ARGUMENT: Invalid request.
- '#type': type.googleapis.com/google.rpc.BadRequest
fieldViolations:
- description: Warmup requests must be enabled for all versions that will gain additional
traffic as a result of the traffic migration.
field: service.split.allocations[20190122-120543-ebbfbc9]
When you use gcloud app versions migrate it will try to do a gradual migration. Gradual traffic migration is only supported for the Standard environment and you have to enable warmup requests on the target version for this to work.
It’s working on GCP Console because there you’ll be prompted to do an immediate migration if the above conditions are not met.
If you want to use gcloud to migrate traffic to a GAE Standard version set to warmup disabled, or Flex environment, you can use the following command which will actually split the traffic immediately (not gradually) 100% to the target version:
gcloud app services set-traffic [MY_SERVICE] --splits [MY_VERSION]=1
You can find more information here
Hope it helps! :)
So, in the past, when I've used Google App Engine, I've utilized their wonderful version system to create an integration, staging, and production environment. When we're ready to go live, we set the production environment to be the default, and it starts receiving traffic at the standard AppSpot URL (myapp.appspot.com), if it's not the default you have to manually go to the version itself (dev.myapp.appspot.com) for your testing. This is an awesome system for a pre-prod deployment with a full test.
However, I just went to do an App Engine deployment today and got this warning:
WARNING: Soon, deployments will set the deployed version to receive all traffic by
default.
To keep the current behavior (where new deployments do not receive any traffic),
use the `--no-promote` flag or run the following command:
$ gcloud config set app/promote_by_default false
To adopt the new behavior early, use the `--promote` flag or run the following
command:
$ gcloud config set app/promote_by_default true
Either passing one of the new flags or setting one of these properties will
silence this message.
So, apparently, very soon Google will make it so that for every deploy of any version, they will just automatically promote that to be the default and be accessible at the standard AppSpot URL.
My question is, what's the benefit in this? I can't use this for A/B testing (since there's no B in the equation here). I can't use this as a pre-prod test, because it takes over my environment right away. Why would I want any version I push up to be the default? Am I missing something? Was I using versions wrong?
I'm just genuinely curious.
I'm just going through the node.js tutorials with a free trial account, and i'm stuck on the second one where you add a db. I add the mongodb deployment, shows up as a VM instances, fine. And my first deploy worked, but now that i'm trying to edit stuff, my deploy's keep failing.
The error i get is that I've exceeded my CPU quota. Watching the list of VM Instances under Compute Engine, i see it keeps spawning up instances, even though the app isn't being used. Guessing it just spins up 8 instances by default?
But then i guess the build system needs its own VM's, but the CPU capacity is used up, so none available to do subsequent builds?! I feel like i'm missing something...
Also, i see i can explicitly start VM's myself, so what process is creating them form me? And can i turn it off? or set a cap on number of instances it spawns?
Can i tell my project to only use 4
Also, the deploy takes forever, is that normal? Following the tutorials, so far I've only seen this command to deploy:
gcloud preview app deploy app.yaml --set-default
Is there another command that does an incremental deploy or something?
By using gcloud preview app deploy you're actually using Managed VMs which is an App Engine runtime which in turn runs Docker containers on Google Compute Engine, which it creates on its own. In other words, you're not using Google Compute Engine directly.
To get rid of extra VMs, you need to delete old app versions: navigate to Compute > App Engine > Versions and delete the versions you don't want.
See also this answer for more details and suggestions.