I'm having an issue where an App Engine project will no longer build remotely (via gcloud app deploy)
This has started out of the blue, with no code changes at this end. Not sure if relevant, but it's a go 1.9 project deploying to the App Engine Flex environment.
I'm not sure how to test this in the same environment as the build, since the error is coming from Google's Container Registry
Here is the log from the Container Registry console
starting build "73f85b4d-7370-41bd-bbb2-bcf42fc38873"
FETCHSOURCE
Fetching storage object: gs://staging.[project].appspot.com/us.gcr.io/[project]/appengine/default.1ed3c690ead06f27aa651a30fab342611:latest#1531698266413753
Copying gs://staging.[project].appspot.com/us.gcr.io/[project]/appengine/default.1ed3c690ead49f731806f27aa630fab342611:latest#1531698266413753...
Operation completed over 1 objects/1.7 MiB.
BUILD
Starting Step #0
Step #0: Pulling image: gcr.io/gcp-runtimes/go1-builder#sha256:c62ac3fbec31ddec70601d6c5b44d07063bcff6a823bdcf5e0bbaa9d3799d1db
Step #0: sha256:c62ac3fbec31ddec70601d6c5b44d07063bcff6a823bdcf5e0bbaa9d3799d1db: Pulling from gcp-runtimes/go1-builder
Step #0: Digest: sha256:c62ac3fbec31ddec70601d6c5b44d07063bcff6a823bdcf5e0bbaa9d3799d1db
Step #0: Status: Downloaded newer image for gcr.io/gcp-runtimes/go1-builder#sha256:c62ac3fbec31ddec70601d6c5b44d07063bcff6a823bdcf5e0bbaa9d3799d1db
Step #0: exec: "gcc": executable file not found in $PATH Finished
Step #0 ERROR ERROR: build step 0 "gcr.io/gcp-runtimes/go1-builder#sha256:c62ac3fbec31ddec70601d6c5b44d07063bcff6a823bdcf5e0bbaa9d3799d1db" failed: exit status 2
It looks like you are using container gcr.io/gcp-runtimes/go1-builder as your build step. Looking at the source in GitHub, I see that there have been no updates since ~late June. I see in the Dockerfile that the base image in the FROM directive is gcr.io/google-appengine/debian9:latest, and a look at that image reveals no gcc installed. I see no step in the Dockerfile installing gcc, and looking at your build step image confirms that it isn't there:
~$ docker run --rm -t -i --entrypoint /bin/bash gcr.io/gcp-runtimes/go1-builder#sha256:c62ac3fbec31ddec70601d6c5b44d07063bcff6a823bdcf5e0bbaa9d3799d1db -- which gcc
Unable to find image 'gcr.io/gcp-runtimes/go1-builder#sha256:c62ac3fbec31ddec70601d6c5b44d07063bcff6a823bdcf5e0bbaa9d3799d1db' locally
sha256:c62ac3fbec31ddec70601d6c5b44d07063bcff6a823bdcf5e0bbaa9d3799d1db: Pulling from gcp-runtimes/go1-builder
e154cec6816f: Pull complete
<pulls elided>
Digest: sha256:c62ac3fbec31ddec70601d6c5b44d07063bcff6a823bdcf5e0bbaa9d3799d1db
Status: Downloaded newer image for gcr.io/gcp-runtimes/go1-builder#sha256:c62ac3fbec31ddec70601d6c5b44d07063bcff6a823bdcf5e0bbaa9d3799d1db
~$
Perhaps an earlier version of the base debian9 image had it installed, you could dig into history to look. But it looks like there is no recent change to the go1-builder image to remove gcc.
If you need gcc, you can always separate building your app from deploying it. Build with your own cloudbuild.yaml via gcloud container builds submit and then deploy the built container using gcloud app deploy --image-url=... With full control over the build, you can always based on the go-builder image and install additional tooling you need like gcc on top of that before using Docker to build your final app container.
Related
I have a Python 3 project which I am hosting on Google AppEngine Standard. Until a couple of days ago I was able to deploy normally (right since I did the initial setup in July 2019) until a couple of days ago. Now I get the following response:
starting build "abc"
FETCHSOURCE
BUILD
Starting Step #0 - "fetcher"
Step #0 - "fetcher": Already have image (with digest): gcr.io/cloud-builders/gcs-fetcher
Step #0 - "fetcher": Fetching manifest gs://staging.my-project.appspot.com/ae/xxx/manifest.json.
Step #0 - "fetcher": Processing 312 files.
Step #0 - "fetcher": Failed to fetch gs://staging.my-project.appspot.com/xxx, will no longer retry: fetching "gs://staging.my-project.appspot.com/xxx" with timeout 1h0m0s to temp file "/workspace/.download/staging.my-project.appspot.com-xxx": source-context.json SHA mismatch, got "xxx", want "yyy"
Step #0 - "fetcher": Failed to download at least one file. Cannot continue.
Finished Step #0 - "fetcher"
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/gcs-fetcher" failed: step exited with non-zero status: 1
Any idea why this would be happening and how to fix it?
P.S. I use the following command for deployment:
gcloud --project my-project app deploy app.yaml
After conversation with Google engineers (https://issuetracker.google.com/issues/154588981?pli=1) the following worked:
Remove the source-context.json file
Delete the bucket where the deployment files are, e.g. gs://staging.my-project.appspot.com
Deploy again
If you need the source-context.json file, you can follow these steps: https://www.google.com/url?q=https://cloud.google.com/debugger/docs/source-context&sa=D&usg=AFQjCNHMB7Dm_jISwG2AnpokQ7XN5GmLAw
This is what I did and it worked for me:
1.
Go to the link that in the error description:
2.
You'll see there some lines for staging fail, copy the name of the file in the first line:
3.
Go to the Google storage here and enter to a bucket with name that starts with "staging.***"
And inside that bucket I search for the string I copy in step 2:
4.
Deleted that file, and retry that steps for every line you'll see in the error details link (in my example there are 4 rows).
5.
Deploy again!
I had a similar problem after I changed a local file during a gcloud app deploy with a long upload (2200 files, because of a wrong .gcloudignore).
I fixed it by deleting the changed file in the cloud storage browser, and deploying again with gcloud app deploy.
For anyone using a NextJS project the solution may be to delete the .next folder and rebuild your project. It seems the some of the cache files from next can become out of sync.
From the root of your project:
rm -rf .next && next build
then redeploy as usual
I am following the instructions on https://cloud.google.com/appengine/docs/go/googlecloudstorageclient/download to begin migrating some code from the, now deprecated, Files API to the new Cloud Storage API without success.
The steps I'm following are ...
I'm running appengine v1.9.23 which is later than the required appengine v1.8.1.
My $GOPATH is set, so I skip step #1.
I proceed to step #2:
goapp get -u golang.org/x/oauth2
goapp get -u google.golang.org/cloud/storage
I am not developing on a managed VM, so I skip step #3.
Now when I run the application, I get:
go-app-builder: Failed parsing input: parser: bad import "syscall" in goapp/src/golang.org/x/net/internal/nettest/error_posix.go
What am I doing wrong?
Steps to reproduce
Download an install the Google Appengine runtime, version 1.9.23 from https://console.cloud.google.com/storage/browser/appengine-sdks/featured/ . Follow the installation instructions documented on https://cloud.google.com/appengine/downloads?hl=en
Create an appengine project directory:
% mkdir $HOME/myapp
Create a new app.yaml file as ~/myapp/app.yaml. Read the directions on the Google website for details: https://cloud.google.com/appengine/docs/go/config/appconfig
I use a version that does not have the static resources:
application: myapp
version: alpha-001
runtime: go
api_version: go1
handlers:
- url: /.*
script: _go_app
Create a location for the Go source files.
% mkdir $HOME/myapp/go
Set your GOPATH to the location of your sources
% export GOPATH=$HOME/myapp/go
Get the Go appengine example project: https://github.com/golang/example
% goapp get github.com/golang/example/appengine-hello
This command will download the example app to the first path entry in the GOPATH
Install the Google Cloud Storage client libraries as directed in https://cloud.google.com/appengine/docs/go/googlecloudstorageclient/download . Reference the steps at the top of this question for more details. Following the directions should result in you running 2 commands:
% go get -u golang.org/x/oauth2
% go get -u google.golang.org/cloud/storage
Attempt to run your go application
% goapp serve
You will see the following compilation error (no stack trace):
2015/12/23 10:37:07 go-app-builder: Failed parsing input: parser: bad import "syscall" in go/src/golang.org/x/net/ipv6/control_unix.go
This error is caused by either of two scenarios:
1) Implicitly importing syscall by importing another package that uses it, as referenced in this related question.
2) Having your package source files in your GOPATH located in a directory at or below the same level as your project's app.yaml (eg. app.yaml in ~/go, and packages sources in ~/go/gopath/src). If a package like x/net/internal/nettest exists in your GOPATH the syscall import will be parsed by goapp at compile time and throw the compilation error.
Avoiding these two scenarios should be sufficient to prevent any bad import "syscall" errors or related compilation errors.
Reproduced the initial steps above and got a similar error, even if not explicitly mentioning syscall. However, running “goapp serve” in the appengine-hello directory results in no error at all.
Adam’s explanation at point 2 applies here correctly: one needs to place the app.yaml file at the right level in the directory structure.
sirupsen/logrus references syscall.
They have an appengine tag specified, not to include syscall so it's usable in AppEngine, something like go build -tags appengine as per issue 310.
However I haven't yet succeeded including it in an AppEngine project so that this build param could be forwarded and specified somewhere so that it goes through. I'll come back to update if I manage.
My app.yaml
runtime: custom
vm: true
api_version: 1
health_check:
enable_health_check: False
Dockerfile
# Use the official go docker image built on debian.
FROM golang:1.5.1
# Grab the source code and add it to the workspace.
ADD . /go/
# Install revel and the revel CLI.
RUN go get github.com/revel/revel
RUN go get github.com/revel/cmd/revel
# Use the revel CLI to start up our application.
ENTRYPOINT revel run 4quorum-appengine dev 8080
# Open up the port where the app is running.
EXPOSE 8080
I was working through this article
http://jbeckwith.com/2015/05/08/docker-revel-appengine/
Preview
I am trying to preview it:
gcloud preview app run app.yaml --custom-entrypoint "revel run 4quorum-appengine dev 8080"
WARNING: The `app run` command is deprecated and will soon be removed.
Please use dev_appserver.py (in the same directory as the `gcloud` command) instead.
Module [default] found in file [/Users/802619/Projects/src/4quorum_root/app.yaml]
INFO: Looking for the Dockerfile in /Users/802619/Projects/src/4quorum_root
INFO: Using Dockerfile found in /Users/802619/Projects/src/4quorum_root
INFO 2015-11-06 18:03:44,226 application_configuration.py:399] No version specified. Generated version id: 20151106t180344
INFO 2015-11-06 18:03:44,226 devappserver2.py:763] Skipping SDK update check.
INFO 2015-11-06 18:03:44,266 api_server.py:205] Starting API server at: http://localhost:62780
INFO 2015-11-06 18:03:44,272 dispatcher.py:197] Starting module "default" running at: http://localhost:8080
INFO 2015-11-06 18:03:44,277 admin_server.py:116] Starting admin server at: http://localhost:8000
ERROR 2015-11-06 18:03:44,282 instance.py:280] [Errno 2] No such file or directory
The same thing if trying dev_appserver.py
Deploy
Deploy also doesn't work. Fails because of timeout.
gcloud preview app deploy ./app.yaml
WARNING: Soon, deployments will set the deployed version to receive all traffic by
default.
To keep the current behavior (where new deployments do not receive any traffic),
use the `--no-promote` flag or run the following command:
$ gcloud config set app/promote_by_default false
To adopt the new behavior early, use the `--promote` flag or run the following
command:
$ gcloud config set app/promote_by_default true
Either passing one of the new flags or setting one of these properties will
silence this message.
You are about to deploy the following modules:
- vaulted-gift-112113/default (from [/Users/802619/Projects/src/4quorum_root/app.yaml])
Deployed URL: [https://20151106t204027-dot-vaulted-gift- 112113.appspot.com]
(add --promote if you also want to make this module available from
[https://vaulted-gift-112113.appspot.com])
Beginning deployment...
Verifying that Managed VMs are enabled and ready.
Provisioning remote build service.
Copying certificates for secure access. You may be prompted to create an SSH keypair.
Building and pushing image for module [default]
Saving [.dockerignore] to [/Users/802619/Projects/src/4quorum_root].
----------------------------- DOCKER BUILD OUTPUT ------------------------------
Step 0 : FROM golang:1.5.1
---> f6271e8f3723
Step 1 : ADD . /go/
---> 94fafc5e8a30
Removing intermediate container cfbe197f6e93
Step 2 : RUN go get github.com/revel/revel
---> Running in d7ad8c923144
---> b65877cf3049
Removing intermediate container d7ad8c923144
Step 3 : RUN go get github.com/revel/cmd/revel
---> Running in 2a9b3320ce47
---> 428defd008f3
Removing intermediate container 2a9b3320ce47
Step 4 : ENTRYPOINT revel run 4quorum-appengine dev 8080
---> Running in 8b9e38ec69ec
---> 3749ee8a6636
Removing intermediate container 8b9e38ec69ec
Step 5 : EXPOSE 8080
---> Running in a0e6c66b56c8
---> dafff62b9643
Removing intermediate container a0e6c66b56c8
Successfully built dafff62b9643
--------------------------------------------------------------------------------
Copying files to Google Cloud Storage...
Synchronizing files to [gs://staging.vaulted-gift-112113.appspot.com/].
Updating module [default]...|Deleted [https://www.googleapis.com/compute/v1/projects/vaulted-gift- 112113/zones/us-central1-f/instances/gae-builder-vm-20151106t204027].
Updating module [default]...failed.
ERROR: (gcloud.preview.app.deploy) Error Response: [4] Timed out creating VMs.
About to drop this.
Moved to heroku. Google App Engine is not ready yet.
I've set up an App Engine project locally using Docker (on OSX), and have been running a server using the usual "gcloud preview app run app.yaml" command. From what I can tell, this keeps creating new images over and over again. After an hour or so of work I end up with something like 30 docker images, each taking 130MB.
Eventually I'm told I can no longer bind to localhost:8080. I tried killing all containers and images, but still cannot use localhost:8080 until I reboot.
Seems like I'm not using Docker/gcloud correctly. Anyone have an idea what I might be doing wrong? Is there another way I should be restarting App Engine instances other than hitting command C and running the "run" command again?
UPDATE: After looking closer, I noticed I'm getting this message when I run an app locally and a container is created: "http: Hijack is incompatible with use of CloseNotifier". I'm not familiar enough with Docker to understand what's going on here. All searches seem to point to Go, which I am not using.
UPDATE 2: Here is the trace:
Creating container...
INFO 2015-05-05 02:23:28,293 containers.py:560] Container 1564ce4344957114312d6d1dc696ffbb4176b40ace6dcff5e4239e13ee04a8f6 created.
Exception in thread Thread-2:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/Users/judeosborn/google-cloud-sdk/platform/google_appengine/google/appengine/tools/docker/containers.py", line 643, in _ListenToLogs
for line in log_lines:
File "/Users/judeosborn/google-cloud-sdk/./lib/docker/docker/client.py", line 225, in _multiplexed_response_stream_helper
socket = self._get_raw_response_socket(response)
File "/Users/judeosborn/google-cloud-sdk/./lib/docker/docker/client.py", line 167, in _get_raw_response_socket
self._raise_for_status(response)
File "/Users/judeosborn/google-cloud-sdk/./lib/docker/docker/client.py", line 119, in _raise_for_status
raise errors.APIError(e, response, explanation=explanation)
APIError: 500 Server Error: Internal Server Error ("http: Hijack is incompatible with use of CloseNotifier")
INFO 2015-05-05 02:23:28,606 module.py:1745] New instance for module "default" serving on:
http://localhost:8080
There's an ongoing issue with Docker 1.6.x [reference] that prevents gcloud to work well with Managed VMs (as you seem to be using). Easiest workaround until it gets fixed is to downgrade Docker in your development machine to version 1.5.0, which is the latest version known to work.
For Ubuntu, you can do something like:
$ curl -sSL https://get.docker.com/ubuntu | sed 's/lxc-docker/lxc-docker-1.5.0/' | sudo sh
For other Linux distros, you might have to modify that sed pattern, though.
On the other hand, if you're using Boot2Docker under Mac OS X, follow these steps:
Fully uninstall your previous Boot2Docker/Docker setup; there is a nice guide here
Reinstall Boot2Docker/Docker following instructions here. IMPORTANT: You MUST stop right after completing "Install Boot2Docker" step and before "Start the Boot2Docker Application". Once you get there, open up a terminal and execute the following commands:
$ mkdir ~/.boot2docker
$ echo 'ISOURL="https://github.com/boot2docker/boot2docker/releases/download/v1.5.0/boot2docker.iso"' > ~/.boot2docker/profile
At this point, you can continue with "Start the Boot2Docker Application" section and finish the installation. You should now have a valid Docker launchpad with which to start Managed VMs. It'd be nice to double check that you have the right versions installed by issuing:
$ boot2docker ssh docker version | egrep "(Client|Server) version"
The output should look like:
Client version: 1.5.0
Server version: 1.5.0
Now you can try again your original command:
$ gcloud preview app run app.yaml
Try running:
$ ps uax | egrep "gcloud|appserver"
If you see anything running, kill it... you may even need to kill -9 it.
Following the instruction in https://www.dartlang.org/server/google-cloud-platform/app-engine/run.html
doesn't work anymore.
During the docker build phase it can't find pubspec.yaml ( Using regular expression with wildcard * ).
I'm using boot2docker and docker 1.5 and the google/dart-runtime image.
Any solution to solve this problem in the deploy of a Dart application in Google Cloud?
INFO 2015-03-08 14:41:12,215 containers.py:280] Step onbuild-0 : ADD pubspec.* /app/
ERROR 2015-03-08 14:41:12,244 containers.py:283] pubspec.*: no such file or directory
INFO 2015-03-08 14:41:12,244 containers.py:292] --------------------------------------------------------
ERROR 2015-03-08 14:41:12,244 instance.py:280] Docker build aborted: pubspec.*: no such file or directory
Updating docker client to latest version done the work:
boot2docker stop
boot2docker download
boot2docker start