I have a Flask app that deploys fine in the Google App Engine Flexible environment but some new updates have made it relatively resource intensive (Was receiving a [CRITICAL] Worker Timeout message.) In attempting to fix this issue I wanted to increase the number of CPUs for my app.
app.yaml:
env: flex
entrypoint: gunicorn -t 600 --timeout 600 -b :$PORT main:server
runtime: python
threadsafe: false
runtime_config:
python_version: 2
automatic_scaling:
min_num_instances: 3
max_num_instances: 40
cool_down_period_sec: 260
cpu_utilization:
target_utilization: .5
resources:
cpu: 3
After some time I receive:
"Updating service [default] (this may take several minutes)...failed.
ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred during deployment."
Is there some sort of permission issue preventing me from increasing the CPUs? Or is my app.ymal invalid?
You can not set the number of cores(CPU) to odd numbers except 1. It should be even.
Related
What causes "Request was aborted after waiting too long to attempt to service your request"?
It seems the result of some sort of internal timeout, but I don't know where this is configured.
We're currently using autoscaling, and that error was the result of a temporary increase in the number of tasks in our taskqueue. Shouldn't autoscaling have created more instances to handle that request?
Also, if a task in Cloud Tasks fails with "Request was aborted after waiting too long to attempt to service your request", is that task retried, or is it removed from the queue?
Edit: I found the problem.
This was the configuration for scailing in our app.yaml:
basic_scaling:
max_instances: 2
I found this when I was proxying to view a munin node with many graphs on an f1-micro backend. Responses will fail with a 529 error if they are waiting for longer than (min|max)_pending_latency - probably it is trying to create a new instance, because the minimum is violated, but finds it cannot.
The default appears to be 5s. You can set it in app.yaml to a max of 15s.
automatic_scaling:
min_pending_latency: 15s
max_pending_latency: 15s
Once I did that I stopped getting the errors for requests waiting for 6s. Of course, I'm sure Google would prefer you increase the number of scaling instances, or use a faster node. But maybe you only want to scale to one or two, or 15s is an acceptable latency for what you're trying to do.
For reference, my full app.yaml:
runtime: php73
service: munin
instance_class: F1
automatic_scaling:
max_instances: 1
min_instances: 0
target_cpu_utilization: 0.95
target_throughput_utilization: 0.95
max_concurrent_requests: 80
max_pending_latency: 15s
handlers:
- url: .*
script: auto
secure: always
I recently asked a question: Deploying to google app engine failed
I got my app deployed, however, we are incurring a large sum for this project. (wiki.js using third-party DB mLab).
I'm wondering if it has to do with the config I put in app.yaml, namely, the memory expansion and resources
This is what the google support person said: to add:
resources:
cpu: 2
memory_gb: 4.0
disk_size_gb: 20
health_check:
enable_health_check: False
My app.yaml (from google console) is:
runtime: nodejs
api_version: '1.0'
env: flexible
threadsafe: true
automatic_scaling:
min_num_instances: 2
max_num_instances: 20
cpu_utilization:
target_utilization: 0.5
resources:
cpu: 2
memory_gb: 4
disk_size_gb: 20
health_check:
enable_health_check: false
Google Cloud Support person here again!
Find a detailed description of costs going to Google Cloud Platform Console home --> Billing --> View detailed charges or following this guide. Update the post with the expensive fields without sharing private data, please, so I can understand what is going on.
Find general information about quotas here.
P.S.: I found a couple of solutions and I suggested you to TRY them since they weren't working for me.
We are facing issues with taskqueues after recently updating the Endpoints API to version 2 in Google App Engine - Python. Following are the issues faced with respect to taskqueus,
Taskqueue doesn't get added to the queue at all , Just gets ignored and never executed.
Taskqueue get terminated with error - "Process terminated because the backend was stopped."
The most critical error is the first one where the task is just ignored and not added to the queue itself.
Details on the codebase and logs are attached along.
It would be great if someone can help us out here.
app.yaml (Server Settings)
#version: 1
runtime: python27
api_version: 1
threadsafe: true
instance_class: F4
automatic_scaling:
min_idle_instances: 1
max_idle_instances: 4 # default value
min_pending_latency: 500ms # default value
max_pending_latency: 900ms
max_concurrent_requests: 50
queue.yaml
- name: allocateStore
rate: 500/s
bucket_size: 500
max_concurrent_requests: 1000
retry_parameters:
task_retry_limit: 0
Adding task to queue:
taskqueue.add(queue_name='allocateStore', url='/tasksStore/allocateStore')
Thanks,
Navin Lr
When trying to deploy my app engine using flexible environment then i am getting error.
ERROR: (gcloud.preview.app.deploy) INVALID_ARGUMENT:
The beta setting machine_type cannot be set in an App Engine Flexible Environment deployment.
My app.yaml is given below
runtime: nodejs
#vm: true
env: flex
# [END runtime]
network:
instance_tag: app-tag
name: network-tag
instance_class: F1
automatic_scaling:
min_num_instances: 1
max_num_instances: 2
cool_down_period_sec: 60
beta_settings:
machine_type: f1-micro
handlers:
- url: /.*
script: IGNORED
secure: always
# Temporary setting to keep gcloud from uploading node_modules
skip_files:
- ^node_modules$
Also can anyone please tell me what is the difference between vm: true and env: flex because both set app engine environment to flexible ??
When changing from vm: true to env: flex you're actually switching to the latest infra version, see Upgrading to the Latest App Engine Flexible Environment Beta Release.
The machine type is no longer configured that way. Instead you'd configure a custom instance shape via its resources:
Resource settings
These settings control the computing resources. App Engine assigns a
machine type based on the amount of CPU and memory you've
specified. The machine is guaranteed to have at least the level of
resources you've specified, it might have more.
You can specify up to eight volumes of tmpfs in the resource settings.
You can then enable workloads that require shared memory via tmpfs and
can improve file system I/O.
For example:
resources:
cpu: 2
memory_gb: 1.3
disk_size_gb: 10
volumes:
- name: ramdisk1
volume_type: tmpfs
size_gb: 0.5
Created a new Flexible Environment App and made successful deploy with latest GCloud version (current is 133) but get "503 Server Error" without any error logs.
Source code: https://github.com/AIMMOTH/scala-stack-angular/tree/503-error
App link: https://scala-stack-angular-us.appspot.com
Error page:
Error: Server Error
The service you requested is not available yet.
Please try again in 30 seconds.
Version info:
Version Status Traffic Allocation Instances Runtime Environment Size Deployed Diagnose
20161108t190158 Serving 100 % 2 custom Flexible
I had a filter responding to /_ah/* and broke Google App Engine.
For me, it was because of wrong settings in app.yaml:
vm: true # the flexible environment
runtime: java # Java 8 / Jetty 9.3 Runtime
service: default
threadsafe: true # handle multiple requests simultaneously
resources:
cpu: .5 # number of cores
memory_gb: 1.3
disk_size_gb: 10 # minimum is 10GB and maximum is 10240GB
health_check:
enable_health_check: true
check_interval_sec: 5 # time interval between checks (in seconds)
timeout_sec: 4 # health check timeout interval (in seconds)
unhealthy_threshold: 2 # an instance is unhealthy after failing this number of consecutive checks
healthy_threshold: 2 # an unhealthy instance becomes healthy again after successfully responding to this number of consecutive checks
restart_threshold: 60 # the number of consecutive check failures that will trigger a VM restart
automatic_scaling:
min_num_instances: 1
max_num_instances: 1
cool_down_period_sec: 120 # time interval between auto scaling checks. It must be greater than or equal to 60 seconds.
# The default is 120 seconds
cpu_utilization:
target_utilization: 0.5 # CPU use is averaged across all running instances and is used to decide when to reduce or
# increase the number of instances (default 0.5)
handlers:
- url: /.* # regex
script: ignored # required, but ignored
secure: always # https
beta_settings:
java_quickstart: true # process Servlet 3.1 annotations
use_endpoints_api_management: true # enable Google Cloud Endpoints API management
I removed use_endpoints_api_management: true and everything works fine.