Why is Google App Engine throwing access forbidden errors? - google-app-engine

Question

Could really use some help here. I have a GAE NodeJS app in the standard environment. Until a few days ago (09/23) it was running just fine, it would respond to requests as expected, etc.
Today, the app responds with 403's when I try to make any request to my appspot url. I'm 100% certain this is not a code issue, as if I deploy the same code to GAE in another project, it works fine. Furthermore, the only firewall rule is a wildcard to allow all traffic.
Edit: adding the only relevant-looking log entry I see from the project:
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {},
"authenticationInfo": {
"principalEmail": "address#domain.com"
},
"requestMetadata": {
"callerIp": "x.x.x.x",
"requestAttributes": {
"time": "2021-09-23T15:04:05.198927Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "appengine.googleapis.com",
"methodName": "google.appengine.v1.Services.UpdateService",
"authorizationInfo": [
{
"resource": "apps/my-google-cloud-project-id/services/default",
"permission": "appengine.services.update",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "apps/my-google-cloud-project-id/services/default",
"serviceData": {
"#type": "type.googleapis.com/google.appengine.v1.AuditData",
"updateService": {
"request": {
"name": "apps/my-google-cloud-project-id/services/default",
"service": {
"networkSettings": {
"ingressTrafficAllowed": "INGRESS_TRAFFIC_ALLOWED_INTERNAL_AND_LB"
}
},
"updateMask": "networkSettings"
}
}
},
"resourceLocation": {
"currentLocations": [
"us-east1"
]
}
},
"insertId": "an-id",
"resource": {
"type": "gae_app",
"labels": {
"project_id": "my-google-cloud-project-id",
"zone": "",
"module_id": "default",
"version_id": ""
}
},
"timestamp": "2021-09-23T15:04:05.131761Z",
"severity": "NOTICE",
"logName": "projects/my-google-cloud-project-id/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "some-operation-uuid",
"producer": "appengine.googleapis.com/admin",
"first": true
},
"receiveTimestamp": "2021-09-23T15:04:05.495890906Z"
}
I don't recall making this change, and I'm not sure what the ingressTrafficAllowed value was before.

Answer 1

Somehow the ingress setting on the GAE service got changed. I believe that issue was fixed by going to GCP console > App Engine > Services > select affected service(s) -> Edit ingress setting from the top, and select the appropriate value.
I say I believe this fixed the issue as I was still getting 403's on my appspot url after doing this, and ultimately I ended up deleting and re-creating the project from scratch, which got everything working again. Clearly there was some misconfiguration somewhere in my project, but GCP does not make it easy to diagnose what the issue might be.