using push queues and flexible environment on Google AppEngine I get 403 (Forbidden) error when a task (to be executed on backend service), created with default service, is executed. The task is successfully pushed to queue, confirmed locally, but the execution of the task(s) fails with log:
INFO 2020-12-24 13:42:39,897 module.py:865] default: "POST /tasks/test-handler HTTP/1.1" 403 31
WARNING 2020-12-24 13:42:39,897 taskqueue_stub.py:2158] Task task2 failed to execute. The task has no remaining retries. Failing permanently after 1 retries and 0 seconds
The same happens both locally and on production. However, if a task is created with a cron job then the execution works just fine. I am using dev_appserver.py with Go 1.11 with the following .yaml definitions:
# backend service
service: backend
runtime: go111
instance_class: F2
inbound_services:
- warmup
- default
handlers:
- url: /tasks/.*
login: admin
redirect_http_response_code: 301
# default app service
service: default
runtime: go111
instance_class: F2
inbound_services:
- warmup
handlers:
- url: /api/.*
script: auto
secure: always
redirect_http_response_code: 301
Initial API request comes to an /api endpoint which then succesfully pushes a queue message using:
t := taskqueue.NewPOSTTask(taskURL, url.Values{
"testParam": {strconv.Itoa(testParam)},
})
if _, err := taskqueue.Add(ctx, t, "test-queue"); err != nil {
return ErrPublishingTaskToQueue
}
My queue.yaml definition (in reality I have many more):
total_storage_limit: 120M
queue:
- name: test-queue
rate: 1/s
bucket_size: 100
max_concurrent_requests: 10
retry_parameters:
task_retry_limit: 1
Any ideas why I'd be getting 403 (Forbidden) statuses on task execution if a task is not created via a cron job? The documentation and existing resources on this matter do not help much :/
Managed to make it work. If anyone struggles with getting 403 responses on task execution for push queues on Google AppEngine make sure that you set the right target service. In my example above I was missing target: backend in queue.yaml:
total_storage_limit: 120M
queue:
- name: test-queue
rate: 1/s
bucket_size: 100
max_concurrent_requests: 10
target: backend
retry_parameters:
task_retry_limit: 1
The problem was that the tasks were created with default service which by default means they hit the default service, but should hit backend service. Unfortunately default service had the required endpoint deployed as well, so I got 403 instead of 404.
More details on the target field:
https://cloud.google.com/appengine/docs/standard/python/config/queueref#target
Related
I have a discord bot that is being hosted on Google App Engine. It will work and run, and then roughly every ~14 min, the bot goes offline, and I see these errors:
Upon further review of the error logs, this is the output:
logMessage: "This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application."
severity: "INFO"
time: "2021-10-03T16:29:18.831860Z"
}
1: {
logMessage: "The warmup request failed. Please check your warmup handler implementation and make sure it's working correctly."
severity: "INFO"
time: "2021-10-03T16:29:18.831862Z"
}
2: {
logMessage: "Process terminated because it failed to respond to the start request with an HTTP status code of 200-299 or 404."
severity: "ERROR"
time: "2021-10-03T16:29:18.831863Z"
My app.yaml file is as follows:
runtime: python38
instance_class: B1
manual_scaling:
instances: 1
entrypoint: python3 bot.py
I'm quite new to GCP and hosting web services, so I am quite lost. Any help here is deeply appreciated.
You need to provide a url handler for /_ah/start. (Might as well also provide for /_ah/stop and /_ah/warmup too). Those are calls GAE will make to start and stop your app. They should return an http response of 200. Here is an example, in Flask:
#app.route('/_ah/start')
#app.route('/_ah/stop')
#app.route('/_ah/warmup')
def warmup():
# Handle your warmup logic here, e.g. set up a database connection pool
return '', 200, {}
EDIT: Valid responses are 200–299 or 404
I am trying to get my dataflow job scheduled via cron.yaml in an AppEngine flexible environment. This works flawlessly when I leave my endpoint unprotected. However, when trying to secure the endpoint, I see 403 status responses, even when triggering it from within the TaskQueues interface.
My app.yaml looks like this:
runtime: java
env: flex
handlers:
- url: /.*
script: this field is required, but ignored
- url: /dataflow/schedule
script: this field is required, but ignored
login: admin
runtime_config:
jdk: openjdk8
resources:
cpu: .5
memory_gb: 1.3
disk_size_gb: 10
manual_scaling:
instances: 1
Secure handlers (like login: admin) do not work on App Engine Flexible, that is why the 403.
For securing that handler, you can check the request header "X-AppEngine-Cron" in your app, which is a trusted header only set by traffic coming from App Engine.
Using Google App Engine, I have an application myapp as a default service that adds a task to a task queue and launch a background service worker called optimize. Although myapp is running fine, unfortunately I always see a POST 404 error in the myapp log when the AppEngine task queue tries to launch the URL /optimize-dot-myapp.appspot.com/index.php/optimize. Of course because of the 404 error the task queue keeps retrying. My current optimize.yaml file contains the following. Any thoughts?
# optimize.yaml configuration for Google App Engine
# Full details at: https://cloud.google.com/appengine/docs/php/config/appref
runtime: php55
api_version: 1
service: optimize
handlers:
# Serve php scripts.
- url: /index.php/optimize
script: index.php/optimize
The default app.yaml file contains the following:
# app.yaml configuration for Google App Engine
# Full details at: https://cloud.google.com/appengine/docs/php/config/appref
runtime: php55
api_version: 1
handlers:
# Serve php scripts.
- url: /(.+\.php).*
script: \1
- url: /
script: index.php
# All URLs beginning with /assets are treated as paths to
# static files in the assets/ directory.
- url: /assets
static_dir: assets
In case it's useful, the optimize worker is started in the default task queue with the following PHP:
// Start the background worker
// API details: https://cloud.google.com/appengine/docs/php/refdocs/classes/google.appengine.api.taskqueue.PushTask
$url = '/optimize-dot-myapp.appspot.com/index.php/optimize';
$task = new PushTask($url, $param);
$task_name = $task->add();
You have collisions in the URL path patterns. The /index.php/optimize path matches both the /index.php/optimize url pattern from optimize.yaml and the /(.+\.php).* pattern from app.yaml. Probably the request ends up in the default service instead of the optimize one. Easy to confirm: check the app logs, you can select a specific service and you'll see which service got the request.
I would add a dispatch.yaml file to clarify things and eliminate the possibility of ambiguous routing (no need to specify the default module, anything not matching dispatch rules is sent to the default module):
application: my_app
dispatch:
- url: "*/optimize/*"
module: optimize
Then adjust the url patterns accordingly in optimize.yaml (they should all start with /optimize):
- url: /optimize/index.php
script: index.php
Note: the index.php file mentioned above would be in the optimize service dir, not in the default service one. Assuming here that each service has its own dir, as mentioned in Can a default service/module in a Google App Engine app be a sibling of a non-default one in terms of folder structure?
And in the task enqueueing code the url should only contain the request path, not the hostname (which is interpreted as part of the path, thus causing a mismatch with the handler's url pattern). You want:
$url = '/optimize/index.php';
In Google App Engine, applications define task queues in a configuration file called queue.yaml. You can use queue.yaml to configure both push queues and pull queues.
The following a basic example that defines a named queue and overrides the default processing rate:
queue:
- name: my-push-queue
rate: 1/s
The following is a more complex example of a queue.yaml configuration that demonstrates setting up task retries and modifying the default processing rate.
queue:
- name: fooqueue
rate: 1/s
retry_parameters:
task_retry_limit: 7
task_age_limit: 2d
- name: barqueue
rate: 1/s
retry_parameters:
min_backoff_seconds: 10
max_backoff_seconds: 200
max_doublings: 0
- name: bazqueue
rate: 1/s
retry_parameters:
min_backoff_seconds: 10
max_backoff_seconds: 200
max_doublings: 3
Please read this documentation for further details
I have a module that works fine when I push it to app engine. When it works it logs stuff nicely and the logs are accessible in the console logs viewer. But then after a while it just stops working and when I try to access any url give me 500 server errors with no info (it just says waiting 30 seconds might be a good idea). When this happens nothing gets logged for requests.
If I restart the module (by pushing my code to app engine) then it works for a little while again.
The module is running a Pyramid app and the configuration file looks a little something like:
application: my_app
module: my_module
version: dev
runtime: python27
api_version: 1
threadsafe: false
instance_class: B2
basic_scaling:
max_instances: 2
idle_timeout: 10m
handlers:
- url: /actions/.*
script: my_module.application
login: admin
- url: /.*
script: my_module.application
builtins:
- appstats: off
libraries:
- name: webob
version: latest
- name: setuptools
version: latest
includes:
- mapreduce/include.yaml
I think what is happening is that it's hitting the idle timeout and shutting down. I need requests to the module to turn it back on again. How do I do that?
Let me know if you need more info, I'm an app engine noob at this stage. Any help would be greatly appreciated.
When a module start, App Engine call the url /_ah/start. You mustn't handle this request.
In you my_module.application you need to add in the handler who match this request :
def get(self):
# Let module start
if "X-Appengine-Cron" in self.request.headers or "X-AppEngine-TaskName" in self.request.headers or "X-Appengine-Failfast" in self.request.headers:
return
Even if you hit the idle timeout, AppEngine will spin a new instance when new request is coming in.
Use Cloud Debugger to inspect the state of your application. The debugger makes it easier to view the application state and understand what happens after your app has been running for while.
Check the docs on startup state https://cloud.google.com/appengine/docs/python/modules/#Python_Instance_states
you current default handler /.* should be able deal with a /_ah/start if youe handler can gracefully deal with a 404.
Thats how I handle startups. Goes through the main handler which can deal with non existent url requests using the default pyramid not found.
I have a config.add_notfound_view(notfound) registered.
I'm getting this error from a request called from the task queue. It suggests I need to change the app.yaml handler but I think I have the correct handler
Here is the error log entry:
2011-11-17 13:30:35.849 /tasks/kacher 302 209ms 0kb
0.1.0.1 - - [17/Nov/2011:13:30:35 -0800] "GET /tasks/kacher HTTP/1.1" 302 0 - - "rawload.XXX.appspot.com" ms=209 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.000032 queue_name=default task_name=cf2e2f1d39d108b3972a1da8c6532fea
W2011-11-17 13:30:35.842
Request failed because URL requires user login. For requests invoked within App Engine (offline requests like Task Queue, or webhooks like XMPP and Incoming Mail), the URL must require admin login (or no login).
This is the code to call the task:
taskqueue.add(url='/tasks/kacher',target='rawload',method='GET')
Here is my app.yaml, with the task urls as login:admin which seems correct (to me):
- url: /tasks.*
script: main.py
login: admin
- url: .*
script: main.py
login: required
secure: always
There's a known bug in the dev_appserver with tasks that require admin login sometimes failing like this. Try logging your browser session in as an admin before accessing the URL that enqueues the task, or try uploading your app to production to see if you experience the issue there.