Why would an AppEngine deploy fail with a build failure? - google-app-engine

Question

I started working on an app and suddenly it fails to deploy to appengine, with the following error message:
siim#pebble:~/projects/xyz$ gcloud app deploy
Services to deploy:
descriptor: [/home/siim/projects/xyz/app.yaml]
source: [/home/siim/projects/xyz]
target project: [xyz]
target service: [default]
target version: [20220313t182940]
target url: [https://xyz.uc.r.appspot.com]
target service account: [App Engine default service account]
Do you want to continue (Y/n)? y
Beginning deployment of service [default]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 2 files to Google Cloud Storage ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build 016c4604-6ea6-47df-8144-e7f1471c6df6 status: FAILURE
removing /layers/google.go.gomod/gopath: unlinkat /layers/google.go.gomod/gopath/pkg/mod/github.com/golang/protobuf#v1.3.1/regenerate.sh: permission denied
Full build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/016c4604-6ea6-47df-8144-e7f1471c6df6?project=576574664421
Clicking through to the build log, I see:
===> DETECTING
google.go.appengine_gomod 0.9.0
google.go.gomod 0.9.0
google.go.build 0.9.0
google.go.appengine 0.9.0
google.utils.label 0.0.2
===> ANALYZING
Previous image with name "us.gcr.io/xyz/app-engine-tmp/app/default/ttl-18h:69af35b6-6fc6-4167-88f3-da16f9cfc7e7" not found
Restoring metadata for "google.go.gomod:gopath" from cache
===> RESTORING
Restoring data for "google.go.gomod:gopath" from cache
===> BUILDING
=== App Engine Gomod (google.go.appengine_gomod#0.9.0) ===
--------------------------------------------------------------------------------
Running "cp --dereference -R . /layers/google.go.appengine_gomod/srv"
Done "cp --dereference -R . /layers/google.go.appengine_gomod/srv" (53.220262ms)
=== Go - Gomod (google.go.gomod#0.9.0) ===
DEBUG: go.mod SHA has changed: clearing GOPATH layer's cache
Failure: (ID: f51775d1) removing /layers/google.go.gomod/gopath: unlinkat /layers/google.go.gomod/gopath/pkg/mod/github.com/golang/protobuf#v1.3.1/regenerate.sh: permission denied
--------------------------------------------------------------------------------
Running "mv -f /builder/outputs/output-5577006791947779410 /builder/outputs/output"
Done "mv -f /builder/outputs/output-5577006791947779410 /builder/o..." (5.980458ms)
ERROR: failed to build: exit status 1
I'm completely mystified by this. AFAICT this is something AppEngine does internally so I don't think I have any way to debug this further? FWIW, my app is super simple ATM:
module xyz
go 1.17
require (
github.com/go-chi/chi/v5 v5.0.7
go.etcd.io/bbolt v1.3.6
google.golang.org/appengine/v2 v2.0.1
)
require (
github.com/golang/protobuf v1.3.1 // indirect
golang.org/x/sys v0.0.0-20200923182605-d9f96fdee20d // indirect
)
And:
runtime: go116
main: ./cmd/server

Answer 1

The solution was to go to container registry (maybe artifact registry soon) in the cloud console and delete all the containers there. The issue was quite likely just one of those images (maybe the build cache one) but my app is not released so I could just delete all of them.

Answer 2

I resolve it using --no-cache option, but its not perfect because cache is not enabled.
$ gcloud app deploy app.yaml --no-cache