We are using google app enigne (using version 1.9.18 while build & deploying app, language is Java)
We are using cron retry parameters in cron.yaml like following
retry_parameters:
min_backoff_seconds: 2.5
max_backoff_seconds: 5
max_doublings: 5
While deploying we are getting following exception
unable to find retry_parameters on com.google.apphosting.utils.config.CronXml$Entity
Why we getting this exception, do we need to update our app engine version ??
retry_parameters is not a global setting; It pertains to each cron job separately. Here is the syntax:
cron:
- description: "retry demo"
url: /retry
schedule: every 10 mins
retry_parameters:
min_backoff_seconds: 2.5
max_doublings: 5
Related
I am attempting to setup a CI Pipeline using Google Cloud Build.
I am attempting to deploy a MeteorJS app which has a lengthy build time - the default build timeout for GCB is 10 minutes and it was recommended here that I increase the timeout.
I have setup my cloudbuild.yaml file with the timeout option increased to 20 minutes:
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy']
timeout: 1200s
I have a Trigger setup in GCB connected to a Bitbucket Repo and when I push a change and the Trigger fires, I get 2 new builds - one coming from Bitbucket and one whose source is Google Cloud Storage.
Once 10 minutes of build time has elapsed, the build from Cloud Storage will timeout which will cause the Bitbucket build to fail as well with Error Response: [4] DEADLINE_EXCEEDED
Occasionally, for whatever reason, the Cloud Storage build will finish in under 10 minutes which will allow the Bitbucket build to finish successfully and deploy.
If I attempt to cancel/stop the Cloud Storage build, it will also stop the Bitbucket build.
The screenshot below shows 2 attempts of the exact same build with differing results.
I do not understand where this second Cloud Storage Build is coming from, but it does not seem to be affected by the settings in my yaml file or my global GCP settings.
I have attempted to run the following commands from the gcloud CLI:
gcloud config set app/cloud_build_timeout 1200
gcloud config set builds/timeout 1200
gcloud config set container/build-timeout 1200
I have also attempted to use a high CPU build machine to speed up the process but it did not seem to have any effect.
Any insight would be greatly appreciated - I feel that I have exhausted every possible combination of Google Search keywords I can think up!
This timeout error comes from app engine deployment itself which has 10 min timeout by default.
You will need to update app/cloud_build_timeout property inside container itself like this:
steps:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args: ['-c', 'gcloud config set app/cloud_build_timeout 1200 && gcloud app deploy']
timeout: 1200s
Update
Actually simpler solution:
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy']
timeout: 1200s
timeout: 1200s
I am deploying a node.js server to Google App Engine from Bitbucket pipeline environment and the last command in the script is: gcloud -q app deploy app.yaml --no-promote --verbosity=debug
The logs show that the service is deployed successfully but the script is not terminating, this is the last part of the log:
> DEBUG: Reading GCS logfile: 206 (read 10 bytes) PUSH DONE DEBUG:
> Operation [...] complete. Result: {...} DEBUG: Reading GCS logfile:
> 416 (no new content; keep polling)
> -------------------------------------------------------------------------------- DEBUG: Converted YAML to JSON: "{...}" DEBUG: Operation [...] not
> complete. Waiting to retry. Updating service [default] (this may take
> several minutes)... .DEBUG: Operation [...] not complete. Waiting to
> retry. ......DEBUG: Operation [...] not complete. Waiting to retry.
> .......DEBUG: Operation [...] not complete. Waiting to retry.
> ......DEBUG: Operation [...] not complete. Waiting to retry.
> .......DEBUG: Operation [...] not complete. Waiting to retry.
> .......DEBUG: Operation [...] not complete. Waiting to retry.
I tried to add readiness_check and liveness_check to app.yml but it didn't change the behaviour.
readiness_check:
path: "/api/public/logout"
check_interval_sec: 5
timeout_sec: 4
failure_threshold: 2
success_threshold: 2
app_start_timeout_sec: 300
liveness_check:
path: "/api/public/logout"
check_interval_sec: 30
timeout_sec: 4
failure_threshold: 2
success_threshold: 2
The main unknown here is what criteria does gcloud app deploy uses to determine termination condition?
Also, is there any bypass to this problem?
Update
The problem happens also when running the gcloud app deploy command from local environment (my laptop).
The problem does NOT happen when removing the --no-promote flag.
The gcloud app deploy command expects a well-formed and valid app.yml file, this is what determines its termination condition.
As you confirmed the deployment worked without the --no-promote flag, it could mean that something in the configuration expects the application to be already deployed and running, thus preventing the script to complete.
Another possible cause would be that the Google Cloud SDK version specified in bitbucket-pipelines.yml is an older one. Make sure you work with the latest. This consideration applies extensively to all dependencies in package.json, which might be conflicting with one another, especially when using older versions of Node.js.
This guide can help at building a sound configuration for Bitbucket-based deployments; although the example given is with Python, it might as well be used as a template for processing a Node.js pipeline.
Nb. in this solution, the Google Cloud SDK version is an older one (127.0.0), which will make this deployment fail, so it should be replaced with the latest (228.0.0 or higher). Also the guide omits another required API activation: Cloud Build API. I've notified the team to amend the solution.
I've tested several scenarios with a simple Node.js server, and could not reproduce the issue. Check my Github repository for the code.
For further help on this topic, please provide more hints, such as the content of the app.yml, bitbucket-pipelines.yml, and package.json files, as well as a description of the state of App Engine (services, versions).
In order to deploy the test repository to App Engine from Bitbucket, make sure the following is done on the project:
Enable API's:
App Engine Admin
Cloud Build
Create a Service Account with following permissions, and generate an API Key:
App Engine: Admin
Cloud Build: Editor
Storage: Object Admin
we updated our google app engine health checks from the legacy version to the new version using and now our deployments are failing. Nothing else on the project has changed. We tested the default settings and then extended checks just in case.
This is the error:
ERROR: (gcloud.app.deploy) Error Response: [4] Your deployment has failed to become healthy in the allotted time and therefore was rolled back. If you believe this was an error, try adjusting the 'app_start_timeout_sec' setting in the 'readiness_check' section.
This is our app.yaml:
liveness_check:
check_interval_sec: 120
timeout_sec: 40
failure_threshold: 5
success_threshold: 5
initial_delay_sec: 500
readiness_check:
check_interval_sec: 120
timeout_sec: 40
failure_threshold: 5
success_threshold: 5
app_start_timeout_sec: 1500
Unfortunately, no matter the configuration, both the readiness and liveness checks are throwing 404s.
What could be causing the problem? and how can we debug this?
Is it possible to rollback to the legacy health checks?
This is usually caused when the application is still reading from the legacy health check flags and/or deploying the app using gcloud app deploy without enabling the updated health checks first. You can check this by:
1- Making sure the legacy health_check flag does not exist on your app.yaml.
2- Run gcloud beta app describe to see whether splitHealthChecks flag is set to true under featureSettings.
By default, HTTP requests from updated health checks are not forwarded to your application container. If you want to extend health checks to your application, then specify a path for liveness checks or readiness checks.
You can then enable updated health checks by using gcloud beta app update --split-health-checks --project [your-project-id]. See this public issue tracker or this article about Updated Health Checks about for more details.
In my case, I solved this issue by manually increasing memory allocation?
resources:
cpu: 1
memory_gb: 2
disk_size_gb: 10
Found this solution in a google forum:
https://groups.google.com/forum/#!topic/google-appengine/Po_-SkC5DOE
For those of you who want to migrate to the default settings for splitted health checks, follow these steps:
1) Remove health_check, liveness_check and readiness_check sections from your app.yaml file
2) Deploy to a newer version, This is important. So, for example, if your current version is production, change it to something else like prod in the command gcloud app deploy --version [new-version-name]
I'm trying to set up some Cron jobs on Google App Engine. This is my cron.yaml file:
cron:
- description: "Progress update job"
url: /cron-jobs/update-progress
schedule: every 24 hours
When I try to deploy that, I get the following error:
$ appcfg.py update ./
06:52 PM Application: my-app-id; version: 1
06:52 PM Host: appengine.google.com
Error parsing yaml file:
Missing required value 'schedule'.
in "./cron.yaml", line 3, column 1
I copied the cron.yaml almost exactly from the documentation page, so I'm not sure what I'm doing wrong here.
Edit: It works now, and don't know what's different. Retyped the exact same file, and now it works. Maybe, since I copy-pasted originally.
I used to be able to deploy, but something in my local environment has changed. Other devs I work with can still deploy from the same repo, so I know it must be something on my mac, but I can't figure out what. The deploy actually works up until the point of updating the crons:
[localhost] local: appcfg.py --oauth2 --version=prod update_cron .
01:39 PM Host: appengine.google.com
Error parsing yaml file:
Unable to assign value 'America/Chicago' to attribute 'timezone':
timezone 'America/Chicago' is unknown
in "./cron.yaml", line 5, column 13
Fatal error: local() encountered an error (return code 1) while executing 'appcfg.py --oauth2 --version=prod update_cron .'
Aborting.
cron.yaml:
cron:
- description: remove orphaned revisions
url: /bible_importer/revisions/orphaned
schedule: every day 04:00
timezone: America/Chicago
target: import
- description: remove orphaned fragment sets
url: /bible_importer/fragments/orphaned
schedule: every day 04:30
timezone: America/Chicago
target: import
Thanks in advance!
If you look into GAE's SDK for crons, it's using pytz to do the timezone conversion, which seems to Raise the exception you are watching even when it doesn't find the module when calling timezone() for the first time.
It also seems that [pytz is not included directly in GAE's SDK] due to it's heavy weight.
So all points to you having the need to install this lib independently.
Hope you get it fixed.