Google appengine cloud tasks alpha handler http 404 error - google-app-engine

Question

When I send a task to a task queue it keeps failing and shows a http 404 (not found) error in the logs.
The project has been whitelisted for cloud tasks alpha on flexible.
I can send HTTP post requests to /endpointpath & /tasks/worker locally without any errors.
The endpoint works fine and adds the task to the task queue.
13:37:41.300 POST 200 0 B 422 ms curl/7.54.0 /endpointspath?key=keyremoved 0.0.0.0 - "POST endpointspath?key=keyremoved" 200 undefined "-" "curl/7.54.0"
The app is running as the default service.
app.go main func:
func main() {
r := mux.NewRouter()
r.HandleFunc("/", handler)
r.HandleFunc("/_ah/health", healthCheckHandler)
// Task handlers
r.Path("/tasks/worker").Methods("POST", "GET", "PUT").HandlerFunc(workerTaskHandler)
// Endpoints
r.Path("/endpointpath").Methods("POST").HandlerFunc(searchHandler)
http.Handle("/", r)
port := 8080
if portStr := os.Getenv("PORT"); portStr != "" {
port, _ = strconv.Atoi(portStr)
}
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", port), nil))
}
abbreviated app.yaml:
runtime: go
env: flex
handlers:
- url: /tasks/.*
script: _go_app
- url: /.*
script: _go_app
Logs HTTP 404 response when queue dispatches request to worker:
10.0.0.1 - "POST /tasks/worker" 404 200 "-" "AppEngine-Google; (+http://code.google.com/appengine)"
Expand all | Collapse all {
httpRequest: {
latency: "0s"
referer: "-"
remoteIp: "10.0.0.1"
requestMethod: "POST"
requestUrl: "/tasks/worker"
responseSize: "200"
status: 404
userAgent: "AppEngine-Google; (+http://code.google.com/appengine)"
}
insertId: "......."
jsonPayload: {
appLatencySeconds: "-"
latencySeconds: "0.000"
time: null
trace: "......."
}
labels: {
appengine.googleapis.com/instance_name: "......"
appengine.googleapis.com/trace_id: "......."
compute.googleapis.com/resource_id: "......."
compute.googleapis.com/resource_name: "......"
compute.googleapis.com/zone: "us-central1-b"
}
logName: "projects/projectname/logs/appengine.googleapis.com%2Fnginx.request"
receiveTimestamp: "2017-12-09T10:56:14.794726383Z"
resource: {
labels: {
module_id: "default"
project_id: "projectname"
version_id: "....."
}
type: "gae_app"
}
timestamp: "2017-12-09T10:56:10.301Z"
}
The closest I can get GAE to find the tasks/worker url is by setting login:admin in app.yaml (even tho flex doesn't use this for authentication). This returns a 403 unauthorised error.
handlers:
- url: /tasks/.*
script: _go_app
login: admin
Here is the 403 response in the logs
{
httpRequest: {
latency: "0s"
referer: "-"
remoteIp: "10.0.0.1"
requestMethod: "POST"
requestUrl: "/tasks/worker"
responseSize: "162"
status: 403
userAgent: "AppEngine-Google; (+http://code.google.com/appengine)"
}
insertId: "....."
jsonPayload: {
appLatencySeconds: "-"
latencySeconds: "0.000"
time: null
trace: "....."
}
labels: {
appengine.googleapis.com/instance_name: "...."
appengine.googleapis.com/trace_id: "...."
compute.googleapis.com/resource_id: "...."
compute.googleapis.com/resource_name: "....."
compute.googleapis.com/zone: "us-central1-b"
}
logName: "projects/projectname/logs/appengine.googleapis.com%2Fnginx.request"
receiveTimestamp: "2017-12-09T13:35:59.986118082Z"
resource: {
labels: {
module_id: "default"
project_id: "projectname"
version_id: "....."
}
type: "gae_app"
}
timestamp: "2017-12-09T13:35:54.764Z"
}
Not sure if it's related but projectname.appspot.com/_ah/health returns this error:
{
"code": 5,
"message": "Method does not exist.",
"details": [
{
"#type": "type.googleapis.com/google.rpc.DebugInfo",
"stackEntries": [],
"detail": "service_control"
}
]
}

Answer 1

It turns out endpoints can't run on the same service as task handlers. Task handler url requests are blocked by the ESP proxy if they run on the same service in the flexible environment, and the service has the endpoints service enabled.
Run task handlers on a separate service and do not set "endpoints_api_service:" in the task handler service app.yaml file.
Doing so will prevent the queue from being able to dispatch to workers in the flexible environment.
This isn't mentioned in the app engine documentation which is kinda bizarre.
The "/_ah/health" issue was caused by this path not being set in the open api file. If this path isn't set the url isn't recognised by the proxy.