Over the past few weeks my gcloud app deploy has been failing regularly, today it fails even after attempting 10 times so far. According to the google cloud web interface, it is failing after build in the exporting phase:
7: exporter
/cnb/lifecycle/exporter asia.gcr.io/<project>/app-engine-tmp/app/ttl-2h:<uuid>
Is there anything we can do about this at all?
(The build itself takes just a few minutes, then this exporter phase runs until the 10m mark and is killed.)
Perhaps there is a way to make gcloud app deploy do the build in the asia region?
I deleted everything in the Google Cloud Console under the "Container Registry" menu option. Not sure why that helps, but builds now average 2 minutes instead of the historical 9-11 minutes for the past year or so.
(Although this technically doesn't answer why behind the question, this does solve the strange delays in building on appengine (standard). I am going to mark it as solved and leave this question here in case others encounter the error message above.)
Related
I deployed my own personal website to Google Cloud's App Engine last week, and since then it has built up around a $12 bill. But there would be very few if any people actually visiting my site, as I'm really the only one who knows about it.
It is a basic React app that uses React Router to display three different pages. I can't imagine the code that I used in the app is so complex that it's causing the bill to stack up, because the code is very basic.
The logging shows health, liveness and readiness checks being completed on the app constantly, is this something that would add to the cost of the app?
I'm relatively new to deploying apps on the cloud, is there anything I can do to reduce the cost of having this app deployed?
Is this Flex or Standard? A Flex instance never goes down to 0 (which means you're being billed) whereas a Standard instance goes to 0 when there's no traffic (which means no billing for that period).
Do you have multiple versions of your App running? When deploying, if you do not specify a version number, gcloud will generate a version number for you. This means that multiple deployments might lead to you having multiple versions and if you do not delete them or force traffic to be migrated to the newer versions, you might end up with multiple versions running and thus increasing your bill. You should check the number of versions you have by going to your console and delete any one you don't need.
Are you using manual scaling or what type of settings do you h
Check out the following tips (short twitter threads from our account) to see if the tips will help - tip 1, tip 2
I have around 10 microservices. Usually I use a script that loops through their folders (each of which contain an app.yaml for that service) and use gcloud app deploy to deploy it. However, is it possible to speed this up by using gcloud app deploy in say, 10 different terminals, one for each microservice? Will GAE processs the deploys in parallel or will this cause any issues?
Note: I am using App Engine flex, and from what I've been told, Standard environment deploys are a lot faster than deploys to Flexible environment. Regardless, switching to standard is not possible.
I tried deploying a couple of services on different tabs of the Cloud Shell Terminal and it works, so it appears to be possible but you should test that with all your microservices to see if this is not a bigger load than app engine could handle (which shoudn't be, but just in case).
However, it felt not worth the effort, since you have to lose time by opening up a new tab, waiting for it to be setup and typing the commands all over again. I did this manually, so if you are able to automate that opening of tabs into a script, it could be worth it.
Also, as you can see on this community post, another way to do a faster deploy is to deploy to another "version" of your app, then when all modules are deployed, do a very fast version switch.
NOTE: Not sure if you are doing this already, but an alternative is to stack the deployment of your services into a single command on Cloud Shell like this:
gcloud app deploy ~/my_app/app.yaml ~/my_app/another_service.yaml
Turns out it does work. I deployed 6 microservices in parallel, and it took < 10 mins for all of them to get deployed, compared to ~60 mins (average of 10 mins each for each service)!
For parallelizing it, I used Azure Devops pipelines, and since each job runs in parallel, all I had to do was create a job for each microservice, then run gcloud app deploy in each of the jobs to deploy them.
Trying out new flexible app engine runtime. In this case a custom Ruby on Rails runtime based on the google provided ruby runtime.
When firing of gcloud preview app deploy the whole process takes ~8 minutes, most of which is "updating service". Is this normal? And more importantly, how can I speed it up?
Regards,
Ward
Yes, that is totally normal. Most of the deployment steps happen away from your computer and are independent of your codebase size, so there's not a lot you can do to speed up the process.
Various steps that are involved in deploying an app on App Engine can be categorized as follows:
Gather info from app.yaml to understand overall deployment
Collect code and use the docker image specified in app.yaml to build a docker image with your code
Provision Compute Instances, networking/firewall rules, install docker related tools on instance, push docker image to instance and start it
Make sure all deployments were successful, start health-checks and if required, transfer/balance out the load.
The only process which takes most of time is the last part where it does all the necessary checks to make sure deployment was successful and start ingesting traffic. Depending upon your code size (uploading code to create container) and requirements for resources (provisioning custom resources), step 2 and 3 might take a bit more time.
If you do an analysis you will find that about 70% of time is consumed in last step, where we have least visibility into, yet the essential process which gives app-engine the ability to do all the heavy lifting.
Deploying to the same version got me from 6 minutes to 3 minutes in subsequent deploys.
Example:
$ gcloud app deploy app.yaml --version=test
Make sure you check what is in the zip it's uploading (it tells you the location of this on deploy), and make sure your yaml skip_files is set to include things like your .git directory if you have one, and node_modules
Note that the subsequent deploys should be much faster than 8 mins. It's usually 1 minute or less in my tests with Node.js on App Engine Flex.
As suggested above by #ludo you could use in the meantime Google App Engine Standard instead of Flex. Which, takes approximately ~30-50 seconds after the first deployment.
You can test GAE Standard by running this tutorial, which doesn't require a billing account:
https://codelabs.developers.google.com/codelabs/cloud-app-engine-springboot/index.html#0
And agreed. this doesn't address GAE Flex but gives some options to accelerate during development.
Just fire this command from root directory of app.yaml
From shell visit directory of app.yaml then run gcloud app deploy
It will be uploaded within few seconds.
I have encountered a Google App Engine problem for which I can not find an answer despite a lot of searching.
I can make a minor tweak to my code - say changing a constant or something, nothing that breaks the app code - and do a deployment using appcfg.py update.
After deployment, nothing works, and I find a "This app has no instances deployed" message in the developer console. Not a single instance is running. Eventually after about ten minutes I start to see instances appearing, the app working fine.
I have maximum idle instances set to 5, I've enabled warmup, and I am well within both quota and budget. My log files don't tell me anything of use.
So this one's got me stumped. I'm not anxious to see a ten minute outage every time I update my code. What is happening here, and how can I ensure that my instances start up as they should when I deploy new code?
Google app engine refuses to deploy my latest build, and looking at the releases list, I can see that another build has been 'deploying' for the better part of a week.
Google doesn't offer support anymore for this without paying for it, but this is stuff that just shouldn't happen.
Hope one of you google engineers out there can help me with this. The google project is caleld vxlpay.
Have you tried doing an appcfg rollback?
Please cancel the deployment if it gets stuck; just waiting for it to finish often leads to frustration and desk-flipping. There's a few ways to help you deploy the app.
1) Generally, you can simply redeploy after waiting a few minutes.
2) Redeploy with another deployment method (appcfg, Google App Launcher, Eclipse...)
3) Rollback then redeploy
If all 3 fails, there might be something wrong with your configuration and you would probably need to speak to the support engineers at Google.
I ran into to this just now.
I think my issue had something to do with having a browser open to the site I was trying to deploy to. Apparently that was locking up a process or something because, when I closed it, my deploy finished.
Silly, yes. I think it has something to do with GAE attempting to migrate traffic but not dealing with cases where there's browsers open... There's probably a feature that allows for deploying and controlling whether or not traffic is migrated.
I'll have to give that a try to see if closing the connection (browser) resolved it or if it was just a timing thing.
Nope... Just takes an absurdly long time.
Maybe it's due to file sizes?
Note: This only occurs when deploying a Flex environment rather a standard one.