I'm trying to deploy my React app to Google Cloud App Engine using gcloud app deploy from my computer. This worked fine last weekend, and before.
I get the following output:
$ gcloud app deploy .
Services to deploy:
descriptor: [REDACTED]
source: [REDACTED]
target project: [REDACTED]
target service: [REDACTED]
target version: [REDACTED]
target url: [REDACTED]
Do you want to continue (Y/n)?
Beginning deployment of service [default]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 0 files to Google Cloud Storage ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build 53d56399-b9ac-4471-8ecb-e06e557523fa status: FAILURE.
Build error details: Build error details not available..
Check the build log for errors: https://console.cloud.google.com/gcr/builds/53d56399-b9ac-4471-8ecb-e06e557523fa?project=995586812495
Following the link in the build log takes me to a Cloud Build log in Google Cloud Console. Below is the last section of that log. There is way more log above this point, but it seems to have gone ok.
Step #1 - "builder": INFO rm_node_modules took 0 seconds
Step #1 - "builder": INFO starting: checking_cached_packages_json_layer
Step #1 - "builder": DEBUG Checking cache for cache_key 75e33f825bb6b991a429a251df57d5c82f04475bb87972004100213e6e245c4d
Step #1 - "builder": INFO checking_cached_packages_json_layer took 0 seconds
Step #1 - "builder": INFO build process for FTL image took 0 seconds
Step #1 - "builder": INFO full build took 0 seconds
Step #1 - "builder": Traceback (most recent call last):
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
Step #1 - "builder": "__main__", fname, loader, pkg_name)
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
Step #1 - "builder": exec code in run_globals
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__.py", line 65, in <module>
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__.py", line 54, in main
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/node/builder.py", line 81, in Build
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/node/layer_builder.py", line 60, in BuildLayer
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/common/cache.py", line 113, in Get
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/common/cache.py", line 137, in _getEntry
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/common/cache.py", line 152, in _getLocalEntry
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/common/cache.py", line 175, in getEntryFromCreds
Step #1 - "builder": File "/ftl-v0.9.0.par/containerregistry/client/v2_2/docker_image_.py", line 279, in exists
Step #1 - "builder": File "/ftl-v0.9.0.par/containerregistry/client/v2_2/docker_image_.py", line 293, in manifest
Step #1 - "builder": File "/ftl-v0.9.0.par/containerregistry/client/v2_2/docker_image_.py", line 250, in _content
Step #1 - "builder": File "/ftl-v0.9.0.par/containerregistry/client/v2_2/docker_http_.py", line 364, in Request
Step #1 - "builder": containerregistry.client.v2_2.docker_http_.V2DiagnosticException: response: {'status': '403', 'content-length': '288', 'x-xss-protection': '1; mode=block', 'transfer-encoding': 'chunked', 'server': 'Docker Registry', '-content-encoding': 'gzip', 'docker-distribution-api-version': 'registry/2.0', 'cache-control': 'private', 'date': 'Sun, 11 Nov 2018 12:13:02 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json'}
Step #1 - "builder": Permission denied for "75e33f825bb6b991a429a251df57d5c82f04475bb87972004100213e6e245c4d" from request "/v2/apparat-web/app-engine-tmp/build-cache/ttl-7d/node-cache/manifests/75e33f825bb6b991a429a251df57d5c82f04475bb87972004100213e6e245c4d". : None
Finished Step #1 - "builder"
ERROR
ERROR: build step 1 "gcr.io/gae-runtimes/nodejs8_app_builder:nodejs8_8_12_0_20180928_RC00" failed: exit status 1
As you can see, the error message is
Permission denied for "75e33f825bb6b991a429a251df57d5c82f04475bb87972004100213e6e245c4d" from request "/v2/apparat-web/app-engine-tmp/build-cache/ttl-7d/node-cache/manifests/75e33f825bb6b991a429a251df57d5c82f04475bb87972004100213e6e245c4d". : None
Finished Step #1 - "builder"
I don't understand what it's telling me is wrong. I am not able to map the 75e33-ID to a user account, and even if I am, what is the resource it's claiming is denying permissions and what can I do about it? It seems like google cloud internals to me and I have no clue how to proceed.
Any ideas would be much appreciated.
Things I've tried
gcloud auth revoke and gcloud auth login to ensure I'm logged in with the correct account for the current project
gcloud config list and verified the current project is correct.
When I run gcloud app deploy I've also verified that the project information is correct when it prompts for confirmation before performing the deploy
Verified my user account is (still) added with correct Roles in Google Cloud Console IAM
I had a very similar problem (identical, except that I was on python37 standard).
The logs had this in it:
Step #1 - "builder": Permission denied for "d7dab0ac46167635bcee11cda0f4dbd2963e1cfeaf021404fa295f99c37070e3" from request "/v2/<project-id>/app-engine-tmp/build-cache/ttl-7d/python-cache/manifests/d7dab0ac46167635bcee11cda0f4dbd2963e1cfeaf021404fa295f99c37070e3". : None
It's actually a GAE configuration problem. In my case it was that I deleted the database (moved it to a different service) and disabled billing.
Shockingly, I solved it by simply re-enabling billing for the project...
I had a similar error message, and turned out to be billing related.
If you are going from free credits to credit card-based billing, you need to manually go in and 'activate' that paid billing, even if your CC is already on file.
You've got the same error as me ... in your traces :
Step #1 - "builder": Traceback (most recent call last):
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
Step #1 - "builder": "__main__", fname, loader, pkg_name)
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
Like me, you have an older version (py27) which seems to break the full chain
I'm searchine too
Can't upload a new py37 version ;-(
Related
gcloud --project my-project-id app deploy app.yaml
fails on GAE Flex when I specify a engines.node version in package.json
but succeeds when I remove it from the package.json, which is contradictory to the https://cloud.google.com/appengine/docs/flexible/nodejs/runtime documentation.
I have specified
"engines": {
"node": "16.x"
}
in my package.json
The error I ultimately get is:
ERROR: gcloud crashed (IOError): [Errno 11] Resource temporarily unavailable
It seems this error is transient as on vary rare occasions in the past 24 hours, the app deploy succeeded.
Google folks, can you confirm the problem is on your side ?
I am deploying on GAE Flex in asia-southeast2-a (Jakarta)
Thanks
Below a more complete log trace
BUILD
Starting Step #0
Step #0: Pulling image: gcr.io/gcp-runtimes/nodejs/gen-dockerfile#sha256:770f37e7042652138c7dac203fc35ef0218002515ddd9f311db1c6c54d6816aa
Step #0: gcr.io/gcp-runtimes/nodejs/gen-dockerfile#sha256:770f37e7042652138c7dac203fc35ef0218002515ddd9f311db1c6c54d6816aa: Pulling from gcp-runtimes/nodejs/gen-dockerfile
Step #0: Digest: sha256:770f37e7042652138c7dac203fc35ef0218002515ddd9f311db1c6c54d6816aa
Step #0: Status: Downloaded newer image for gcr.io/gcp-runtimes/nodejs/gen-dockerfile#sha256:770f37e7042652138c7dac203fc35ef0218002515ddd9f311db1c6c54d6816aa
Step #0: gcr.io/gcp-runtimes/nodejs/gen-dockerfile#sha256:770f37e7042652138c7dac203fc35ef0218002515ddd9f311db1c6c54d6816aa
Step #0: Checking for Node.js.
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/kaniko-project/executor#sha256:f87c11770a4d3ed33436508d206c584812cd656e6ed08eda1cff5c1ee44f5870
Step #1: INFO[0000] Removing ignored files from build context: [node_modules .dockerignore Dockerfile npm-debug.log yarn-error.log .git .hg .svn app.yaml]
Step #1: INFO[0000] Downloading base image gcr.io/google-appengine/nodejs#sha256:721ce182842495a610589261688f1abc1801c915a7e24880fa15af9e9d725459
Step #1: INFO[0017] Taking snapshot of full filesystem...
Step #1: INFO[0026] Using files from context: [/workspace]
Step #1: INFO[0026] COPY . /app/
Step #1: INFO[0026] Taking snapshot of files...
Step #1: INFO[0026] RUN /usr/local/bin/install_node '16.x'
Step #1: INFO[0026] cmd: /bin/sh
Step #1: INFO[0026] args: [-c /usr/local/bin/install_node '16.x']
Step #1: % Total % Received % Xferd Average Speed Time Time Time Current
Step #1: Dload Upload Total Spent Left Speed
100 30.8M 100 30.8M 0 0 68.9M 0 --:--:-- --:--:-- --:--:-- 68.9M
Step #1: % Total % Received % Xferd Average Speed Time Time Time Current
Step #1: Dload Upload Total Spent Left Speed
100 4035 100 4035 0 0 35238 0 --:--:-- --:--:-- --:--:-- 35394
Step #1: node-v16.13.0-linux-x64.tar.gz: OK
Step #1: Installed Node.js v16.13.0
Step #1: INFO[0028] Taking snapshot of full filesystem...
Step #1: INFO[0030] Adding whiteout for /nodejs/lib/node_modules/npm/node_modules/npm-install-checks/CHANGELOG.md
Step #1: INFO[0030] Adding whiteout for /nodejs/lib/node_modules/npm/node_modules/os-tmpdir
Step #1: INFO[0030] Adding whiteout for /nodejs/lib/node_modules/npm/node_modules/npmlog/CHANGELOG.md
Step #1: INFO[0030] Adding whiteout for /nodejs/lib/node_modules/npm/node_modules/lodash.uniq
(...)
ERROR: gcloud crashed (IOError): [Errno 11] Resource temporarily unavailable
Thanks
[Update] I have upgrade the gcloud-sdk and the error message is a little different now, I get a
[Errno 11] write could not complete without blocking
in the middle of the long logging stream generated by the gcloud app deploy command.
Following the advise of other developers (related to python lib issues, link I found a workaround by adding the --no-user-output-enabled flag to the gcloud app deploy command. This prevents the error from happening.
But this is a temp workaround and I hope the GCP team will fix the issue in the CLI library.
I am deploying a trivial App Engine Standard Environment app. (Literally the shortest possible, a Python 3 "hello world".) I am using Macbook with zshell.
If I redirect standard error to file, I get an error (below) every time.
gcloud app deploy -q 2>>err.log
If I omit the redirection, it succeeds every time.
There is no difference between using > or >>. Redirecting with a pipe, e.g. to grep, does not cause the problem.
So this is a "solution" (by sending output through a passthrough grep) that does what I need and does not trigger the problem, but this is very roundabout.
gcloud app deploy -q 2>&1 >/dev/null |egrep "." >> err.txt
Note that I use -q, so waiting for my Y for approval is not the issue.
The error is this. (Identifiers were anonymized.)
..................failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build BUILD_ID status: FAILURE
Build error details: Failed to download at least one file. Cannot continue.
Full build logs: https://console.cloud.google.com/cloud-build/builds/BUILD_ID?project=PROJECT_ID
Looking at the logs, I see this.
starting build "BUILD_ID"
FETCHSOURCE
BUILD
Starting Step #0 - "fetcher"
Step #0 - "fetcher": Already have image (with digest): gcr.io/cloud-builders/gcs-fetcher
Step #0 - "fetcher": Fetching manifest gs://staging.joshua-playground.appspot.com/ae/BUILD_ID/manifest.json.
Step #0 - "fetcher": Processing 728 files.
Step #0 - "fetcher": Failed to fetch gs://staging.my-project.appspot.com/BUILD_ID, will no longer retry: fetching "gs://staging.my-project.appspot.com/BUILD_ID" with timeout 1h0m0s to temp file "/workspace/.download/staging.joshua-playground.appspot.com-BUILD_ID": err SHA mismatch, got "SHA_VALUE", want "SHA_VALUE"
Step #0 - "fetcher": Failed to download at least one file. Cannot continue.
Finished Step #0 - "fetcher"
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/gcs-fetcher" failed: step exited with non-zero status: 1
Google has confirmed this issue. Please track it here.
I have a Python 3 project which I am hosting on Google AppEngine Standard. Until a couple of days ago I was able to deploy normally (right since I did the initial setup in July 2019) until a couple of days ago. Now I get the following response:
starting build "abc"
FETCHSOURCE
BUILD
Starting Step #0 - "fetcher"
Step #0 - "fetcher": Already have image (with digest): gcr.io/cloud-builders/gcs-fetcher
Step #0 - "fetcher": Fetching manifest gs://staging.my-project.appspot.com/ae/xxx/manifest.json.
Step #0 - "fetcher": Processing 312 files.
Step #0 - "fetcher": Failed to fetch gs://staging.my-project.appspot.com/xxx, will no longer retry: fetching "gs://staging.my-project.appspot.com/xxx" with timeout 1h0m0s to temp file "/workspace/.download/staging.my-project.appspot.com-xxx": source-context.json SHA mismatch, got "xxx", want "yyy"
Step #0 - "fetcher": Failed to download at least one file. Cannot continue.
Finished Step #0 - "fetcher"
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/gcs-fetcher" failed: step exited with non-zero status: 1
Any idea why this would be happening and how to fix it?
P.S. I use the following command for deployment:
gcloud --project my-project app deploy app.yaml
After conversation with Google engineers (https://issuetracker.google.com/issues/154588981?pli=1) the following worked:
Remove the source-context.json file
Delete the bucket where the deployment files are, e.g. gs://staging.my-project.appspot.com
Deploy again
If you need the source-context.json file, you can follow these steps: https://www.google.com/url?q=https://cloud.google.com/debugger/docs/source-context&sa=D&usg=AFQjCNHMB7Dm_jISwG2AnpokQ7XN5GmLAw
This is what I did and it worked for me:
1.
Go to the link that in the error description:
2.
You'll see there some lines for staging fail, copy the name of the file in the first line:
3.
Go to the Google storage here and enter to a bucket with name that starts with "staging.***"
And inside that bucket I search for the string I copy in step 2:
4.
Deleted that file, and retry that steps for every line you'll see in the error details link (in my example there are 4 rows).
5.
Deploy again!
I had a similar problem after I changed a local file during a gcloud app deploy with a long upload (2200 files, because of a wrong .gcloudignore).
I fixed it by deleting the changed file in the cloud storage browser, and deploying again with gcloud app deploy.
For anyone using a NextJS project the solution may be to delete the .next folder and rebuild your project. It seems the some of the cache files from next can become out of sync.
From the root of your project:
rm -rf .next && next build
then redeploy as usual
We are not able to deploy our app anymore to App Engine Standard.
Steps to reproduce: we simply deploy our project with IntelliJ IDEA - we get the following error:
Beginning deployment of service [default]...
#============================================================#
#= Uploading 0 files to Google Cloud Storage =#
#============================================================#
File upload done.
Updating service [default]...
.................................................................................failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build 15c72c99-c2cc-4e4d-b910-0eee13a9cb5a status: FAILURE.
Error ID: 6AA2815A.
Error type: BuildError.
Error message: 21 Feb 2020 04:36:00 INFO Running as user 0, group 0
21 Feb 2020 04:36:00 INFO Arguments: ['--name=eu.gcr.io/XXXXXXX/app-engine-tmp/app/ttl-2h:af4f2b82-4138-4c9a-905f-6f9cf038e4cd', '--src=/workspace', '--base=eu.gcr.io/gae-runtimes/java8:java8_20191215_8_0_RC00']
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/bin/appender.par/__main__.py", line 90, in <module>
File "/bin/appender.par/__main__.py", line 76, in main
File "/bin/appender.par/containerregistry/client/v2_2/append_.py", line 74, in __init__
File "/bin/appender.par/containerregistry/client/v2_2/docker_image_.py", line 110, in uncompressed_blob
File "/usr/lib/python2.7/gzip.py", line 260, in read
self._read(readsize)
File "/usr/lib/python2.7/gzip.py", line 319, in _read
self._add_read_data( uncompress )
File "/usr/lib/python2.7/gzip.py", line 337, in _add_read_data
self.extrabuf = self.extrabuf[offset:] + data
MemoryError.
Failed to deploy '[2020-02-21 06:33:21] XXXX:war exploded. Project: XXXXXX. Version: vvvv': Deployment failed with exit code: 1
Please make sure that you are using the latest version of the Google Cloud SDK.
Run ''gcloud components update'' to update the SDK. (See: https://cloud.google.com/sdk/gcloud/reference/components/update.)
The Cloud SDK is managed by IDEA .. it seems to be the latest version (in Settings - Cloud SDK).
We have another smaller test project in IDEA which we can deploy without issues.
And we have another larger project in Eclipse which fails with a similar error - MemoryError.
I guess there must be some Memory issue in Google Cloud since the smaller project works fine, but the 2 larger projects do not work!!
This happens since yesterday .. we searched but could not find any similar issues .. so I guess this must be something new?
UPDATE:
It seems to work now .. I guess it was a temporary glitch in the Google deployment process.
For issues like this, it would be better to open a ticket to the Google Cloud Platform support team. They will be able to troubleshoot it better with access to the specifics of your environment and the logs.
When running gcloud preview app run from Google Compute Engine, running Ubuntu Trust 14.04 image, I get the following error:
(Note: I fetched the complete gcloud SDK from https://cloud.google.com/sdk/)
$ gcloud preview app run appengine-modules-guestbook/src/main/webapp
Java module found in [/home/yaraju_gmail_com/appengine-modules-sample-java/appengine-modules-guestbook/src/main/webapp]
INFO: Skipping SDK update check.
INFO: Starting API server at: http://localhost:46920
INFO: Starting module "default" running at: http://localhost:8080
INFO: Starting admin server at: http://localhost:8000
Exception in thread Instance Adjustment:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 1492, in _loop_adjusting_instances
self._adjust_instances()
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 1469, in _adjust_instances
self._add_instance(permit_warmup=True)
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 1354, in _add_instance
if not inst.start():
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/instance.py", line 274, in start
self._runtime_proxy.start()
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/http_runtime.py", line 270, in start
stderr=subprocess.PIPE)
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/safe_subprocess.py", line 132, in start_process_file
stderr=stderr)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
What can I do to fix this?
You might be missing one of the dependencies.
Ensure that you have run the following:
sudo apt-get install openjdk-7-jdk
gcloud components update gae-java