Issue deploying to AppEngine (Flex) from Cloud Build - google-app-engine

Question

I'm facing issue about deploying to AppEngine Flex Environment via Cloud Build. It used to work fine but not today. here's trace shown on Cloud Build's logs.
... (skipped composer's things) ...
Step #1 Updating service [legacy-api] (this may take several minutes)...
Step #1: ..............................................................................................................................................................................................................................................................................................failed.
Step #1: ERROR: (gcloud.app.deploy) Error Response: [13] Flex operation projects/MY-PROJECT/regions/asia-east2/operations/... error [INTERNAL]: An internal error occurred while processing task /appengine-flex-v1/insert_flex_deployment/flex_create_resources>2019-11-27T07:24:52.924Z46964.jo.8: Deployment Manager operation …/operation-… errors: [code: "RESOURCE_ERROR"
Step #1: location: "/deployments/…/resources/…-00it"
Step #1: message: "{\"ResourceType\":\"compute.v1.instanceTemplate\",\"ResourceErrorCode\":\"400\",\"ResourceErrorMessage\":{\"code\":400,\"errors\":[{\"domain\":\"global\",\"message\":\"Invalid value for field \'resource.properties.labels\': \'\'. Label value \'Infinity\' violates format constraints. The value can only contain lowercase letters, numeric characters, underscores and dashes. The value can be at most 63 characters long. International characters are allowed.\",\"reason\":\"invalid\"}],\"message\":\"Invalid value for field \'resource.properties.labels\': \'\'. Label value \'Infinity\' violates format constraints. The value can only contain lowercase letters, numeric characters, underscores and dashes. The value can be at most 63 characters long. International characters are allowed.\",\"statusMessage\":\"Bad Request\",\"requestPath\":\"https://compute.googleapis.com/compute/v1/projects/.../global/instanceTemplates\",\"httpMethod\":\"POST\"}}"
Step #1: ]
Here's my cloudbuild.yaml
steps:
- name: 'gcr.io/$PROJECT_ID/secrets:latest'
entrypoint: sh
args:
- "-c"
- |
cat /secrets/$_ENV/environments/${_SERVICE_NAME}.env > .env
- name: 'gcr.io/cloud-builders/gcloud'
args: ["app", "deploy", "--version=$SHORT_SHA", "--promote", "--stop-previous-version", "./app.yaml"]
timeout: 1200s
substitutions:
_ENV: staging
_SERVICE_NAME: legacy-api
here's my app.yaml
service: legacy-api
runtime: php
env: flex
runtime_config:
document_root: public
enable_stackdriver_integration: true
resources:
cpu: 4
memory_gb: 8
beta_settings:
cloud_sql_instances: ${CLOUD_SQL_INSTANCE}
network:
name: default
now, this blocked my development processes. please help.
Thanks in advance!
edit 2019-11-27 23:00 GMT+0700 (12hours after problem first seen)
things get worse, i have no idea what to do. then, i tried to change deployment destination to Standard Environment. and this is what I get.
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/gcloud
Step #1: Services to deploy:
Step #1:
Step #1: descriptor: [/workspace/app.yaml]
Step #1: source: [/workspace]
Step #1: target project: [....]
Step #1: target service: [legacy-api-std]
Step #1: target version: [201911272240]
Step #1: target url: [https://legacy-api-std-dot-....appspot.com]
Step #1:
Step #1:
Step #1: Do you want to continue (Y/n)?
Step #1: Beginning deployment of service [legacy-api-std]...
Step #1: Created .gcloudignore file. See `gcloud topic gcloudignore` for details.
Step #1: #============================================================#
Step #1: #= Uploading 1 file to Google Cloud Storage =#
Step #1: #============================================================#
Step #1: File upload done.
Step #1: ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: This deployment has too many files. New versions are limited to 10000 files for this app.
Step #1: - '#type': type.googleapis.com/google.rpc.BadRequest
Step #1: fieldViolations:
Step #1: - description: This deployment has too many files. New versions are limited to 10000
Step #1: files for this app.
Step #1: field: version.deployment.files[...]
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/gcloud" failed: exit status 1
update 2019-11-28 08:40:00+0700 (almost 24 hours since the problem first seen
cases opened in Google Cloud Support 8 hours ago
still no miracle here

Answer 1

I have been able to find out that the issue is known in GCP and the team is currently working on it.
The error is due to the fact that the SHORT_SHA that you are using, is matching one of the regexs used by the internal Deployment Manager YAML parser. At the moment, this known issue is caused by passing a SHA string that the Deployment Manager YAML parser will evaluate as a float rather than a string, passing a timestamp as a value also causes the parser to incorrectly evaluate the string as a timestamp.
Then adding a random string at the end of the SHORT_SHA will force it to be a string and therefore avoid the issue:
"${SHORT_SHA}xyz"
Regex that is being used for floating point numbers can be found here.
I have also created this PIT to keep track of the engineer's investigation. Further information about it will be shared in this thread.