I deployed a simple nodejs server on Google app engine flex.
When it has 1 instance running, it is getting 3 times as much liveness and readyness checks as it should be reiceving considering the configuration on my app.yml file.
The documentation says:
If you examine the nginx.health_check logs for your application, you might see health check polling happening more frequently than you have configured, due to the redundant health checkers that are also following your settings. These redundant health checkers are created automatically and you cannot configure them.
Still this does look like an aggressive behaviour. Is this normal?
My app.yml config :
runtime: nodejs
env: flex
service: web
resources:
cpu: 1
memory_gb: 3
disk_size_gb: 10
automatic_scaling:
min_num_instances: 1
cpu_utilization:
target_utilization: 0.6
readiness_check:
path: "/readiness_check"
timeout_sec: 4
check_interval_sec: 5
failure_threshold: 2
success_threshold: 1
app_start_timeout_sec: 300
liveness_check:
path: "/liveness_check"
timeout_sec: 4
check_interval_sec: 30
failure_threshold: 2
success_threshold: 1
Yes, this is normal. Three different locations are checking health of your service. You have configured the health check to be every five seconds. If you want less health check traffic, change check_interval_sec: 5 to be a larger number.
Related
I'm running a very simple node app in a GAE Flex custom instance instance.
All of a sudden, seemingly out of nowhere, it shuts down causing a short period of 503s, before eventually coming back up:
I'm absolutely certain nobody did this manually.
What's going on? Are GAE apps expected to randomly shut down and restart?
Here's my config:
runtime: custom
api_version: '1.0'
env: flexible
threadsafe: true
automatic_scaling:
cool_down_period: 120s
min_num_instances: 1
max_num_instances: 15
cpu_utilization:
target_utilization: 0.5
network: {}
liveness_check:
initial_delay_sec: 300
path: /
check_interval_sec: 30
timeout_sec: 4
failure_threshold: 4
success_threshold: 2
readiness_check:
path: /
check_interval_sec: 5
timeout_sec: 4
failure_threshold: 2
success_threshold: 2
app_start_timeout_sec: 300
It turns out that GAE flex instances automatically reboot once a week. So when you have only one instance running, this behaviour is to be expected.
I have a Flask app that deploys fine in the Google App Engine Flexible environment but some new updates have made it relatively resource intensive (Was receiving a [CRITICAL] Worker Timeout message.) In attempting to fix this issue I wanted to increase the number of CPUs for my app.
app.yaml:
env: flex
entrypoint: gunicorn -t 600 --timeout 600 -b :$PORT main:server
runtime: python
threadsafe: false
runtime_config:
python_version: 2
automatic_scaling:
min_num_instances: 3
max_num_instances: 40
cool_down_period_sec: 260
cpu_utilization:
target_utilization: .5
resources:
cpu: 3
After some time I receive:
"Updating service [default] (this may take several minutes)...failed.
ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred during deployment."
Is there some sort of permission issue preventing me from increasing the CPUs? Or is my app.ymal invalid?
You can not set the number of cores(CPU) to odd numbers except 1. It should be even.
I recently asked a question: Deploying to google app engine failed
I got my app deployed, however, we are incurring a large sum for this project. (wiki.js using third-party DB mLab).
I'm wondering if it has to do with the config I put in app.yaml, namely, the memory expansion and resources
This is what the google support person said: to add:
resources:
cpu: 2
memory_gb: 4.0
disk_size_gb: 20
health_check:
enable_health_check: False
My app.yaml (from google console) is:
runtime: nodejs
api_version: '1.0'
env: flexible
threadsafe: true
automatic_scaling:
min_num_instances: 2
max_num_instances: 20
cpu_utilization:
target_utilization: 0.5
resources:
cpu: 2
memory_gb: 4
disk_size_gb: 20
health_check:
enable_health_check: false
Google Cloud Support person here again!
Find a detailed description of costs going to Google Cloud Platform Console home --> Billing --> View detailed charges or following this guide. Update the post with the expensive fields without sharing private data, please, so I can understand what is going on.
Find general information about quotas here.
P.S.: I found a couple of solutions and I suggested you to TRY them since they weren't working for me.
I've noticed that when I'm upgrading the vm tag to the new env one, that the auto scaler creates an infinite amount of instances during the deployment process. This only happens when I'm using the new env flag. This is my conf file:
runtime: custom
vm: true
service: default
threadsafe: true
health_check:
enable_health_check: True
check_interval_sec: 5
timeout_sec: 4
unhealthy_threshold: 2
healthy_threshold: 2
restart_threshold: 60
automatic_scaling:
min_num_instances: 1
cool_down_period_sec: 60
cpu_utilization:
target_utilization: 0.9
It would be great if anyone could help me because I'm unable to migrate my VM's because of this problem.
Cheers
Created a new Flexible Environment App and made successful deploy with latest GCloud version (current is 133) but get "503 Server Error" without any error logs.
Source code: https://github.com/AIMMOTH/scala-stack-angular/tree/503-error
App link: https://scala-stack-angular-us.appspot.com
Error page:
Error: Server Error
The service you requested is not available yet.
Please try again in 30 seconds.
Version info:
Version Status Traffic Allocation Instances Runtime Environment Size Deployed Diagnose
20161108t190158 Serving 100 % 2 custom Flexible
I had a filter responding to /_ah/* and broke Google App Engine.
For me, it was because of wrong settings in app.yaml:
vm: true # the flexible environment
runtime: java # Java 8 / Jetty 9.3 Runtime
service: default
threadsafe: true # handle multiple requests simultaneously
resources:
cpu: .5 # number of cores
memory_gb: 1.3
disk_size_gb: 10 # minimum is 10GB and maximum is 10240GB
health_check:
enable_health_check: true
check_interval_sec: 5 # time interval between checks (in seconds)
timeout_sec: 4 # health check timeout interval (in seconds)
unhealthy_threshold: 2 # an instance is unhealthy after failing this number of consecutive checks
healthy_threshold: 2 # an unhealthy instance becomes healthy again after successfully responding to this number of consecutive checks
restart_threshold: 60 # the number of consecutive check failures that will trigger a VM restart
automatic_scaling:
min_num_instances: 1
max_num_instances: 1
cool_down_period_sec: 120 # time interval between auto scaling checks. It must be greater than or equal to 60 seconds.
# The default is 120 seconds
cpu_utilization:
target_utilization: 0.5 # CPU use is averaged across all running instances and is used to decide when to reduce or
# increase the number of instances (default 0.5)
handlers:
- url: /.* # regex
script: ignored # required, but ignored
secure: always # https
beta_settings:
java_quickstart: true # process Servlet 3.1 annotations
use_endpoints_api_management: true # enable Google Cloud Endpoints API management
I removed use_endpoints_api_management: true and everything works fine.