Google cloud app engine flex environment deployment failure (NestJS) - Error code 13 (Internal Error Occurred) - google-app-engine

I've done a lot of researching on this error and found some posts here and on github but none of the resolutions they suggest are working for me. I'm receiving a pretty ambiguous error when trying to deploy my application to a flex environment on google cloud service app engine.
It works fine when I deploy to a standard app service in the app engine, but when I try to deploy to a flex service I receive this error:
ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred during deployment.
I know it is nothing wrong with my application code nor my app.yaml because when I do the same deployment on another google cloud account, the service deploys fully, publishes, and I'm able to use the API. It is definitely something failing with my google cloud setup for this specific account. I'm pretty confident that I have all of the apis enabled and correct service account roles applied to this account that is not working, but obviously something is different.
One suggestion I read was to apply the role "App Engine flexible environment Service Agent" to my app engine default service account. That role did not exist on my other account that is working, but I tried anyway - no joy.
Here are my roles assigned to my service accounts on both my working google cloud account, and the non-working one.
Here is a comparison between my enabled APIs on both accounts:
I set both of these cloud accounts up and I do not recall doing anything differently for the one that is working than I did for the one that is not.
I've tried deploying through my cloud build trigger and manually running gcloud app deploy.
I also tried the latter with --verbosity=debug which gives me only a little more information about the failure:
"done": true,
"error": {
"code": 13,
"message": "An internal error occurred during deployment."
},
"metadata": {
"#type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
"endTime": "2022-08-14T00:25:57.780Z",
"insertTime": "2022-08-14T00:25:54.623Z",
"method": "google.appengine.v1.Versions.CreateVersion",
"target": "apps/my-app/services/your-app/versions/20220814t002258",
"user": "me#you.com"
},
This is a resource I've found similar to my issue: https://github.com/GoogleCloudPlatform/nodejs-getting-started/issues/153
I've tried every suggestion there and still having the problem.
I'm 100% certain it's not the code itself nor the app.yaml because it works fine on my other account. Any other suggestions of places to look would be greatly appreciated.
TIA

Related

VPC Network error when configuring App Engine for Google Tag Manager Server Side

I'm trying to set up ServerSide Google Tag Manager, which is a service that runs on GCP App Engine.
The setup is done with a few Cloud Shell commands, which are describede here:
https://developers.google.com/tag-platform/tag-manager/server-side/script-user-guide#create_a_tagging_server_or_reconfigure_an_existing_tagging_server
However, when I did it originally (even as GCP project owner), I got this error:
I was advised that, to solve this, I needed to manually create a VPC Network called default. So I did exactly, that, with the default network settings.
When I ran the App Engine setup commands again, I got further, but ran into a new error:
ERROR: (gcloud.app.deploy) Error Response: [3] Flex operation projects/projectid/regions/europe-west1/operations/f7d7489b-f4ef-4b8b-ac26-b9b945d3ebfc error [INVALID_ARGUMENT]: An internal error occurred while processing task /app-engine-flex/insert_flex_deployment/flex_create_resources>2022-05-12T09:17:00.642Z6448.wa.0: 'subnetwork_name' must be specified for custom mode network 'default'.
I tried the setup commands with and without a defined subnetwork in the new VPC network I created. The same error persisted regardless.
I have not been able to find any solution to this online so far, so hoping that someone here will be able to assist.
Thanks in advance!
Simon
As you created your network as "custom", you need to create by yourself at least one subnet in the same region where your app was created. After that, you must configure the subnet in the app.yaml file:
network:
name: projects/PROJECT_ID/global/networks/NETWORK_NAME
subnetwork_name: SUBNETWORK_NAME
The answer to this was that the default service in the app engine (Services in the left-hand menu in the app engine console) failed to automatically create the VPC- and subnetworks.
By manually recreating the defaultapp engine service, the correct VPC networks and subnetworks were created, and the problem was solved.

Google app engine deployment using wrong service account?

I have this app that's been running on Google App Engine for a good couple of years. Today I needed to modify its app.yaml file, but when I ran "gcloud app deploy" I got this error:
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [7] Failed to create cloud build: service account "redacted#cloudbuild.gserviceaccount.com" has insufficient permission to execute the build on project "redacted"
It appears that the default service account no longer has the necessary permissions. In fact, it may have been deleted completely, because I don't see it listed under gcloud projects get-iam-policy xxxx. What I see instead is this:
serviceAccount:service-redacted#gcp-sa-cloudbuild.iam.gserviceaccount.com
with a correct-looking role cloudbuild.serviceAgent.
I tried googling any changes related to GAE service accounts but didn't find any useful references. I followed the solution to this question, but it didn't help.
The question is, how do I change the permissions of the service account mentioned in the original error message, or, how do I tell GAE to use another (correct) service account when deploying my app?
According to [this][1] documentation
...To deploy new versions, a member must have the Service Account User (roles/iam.serviceAccountUser) role on the App Engine default service account, and the Cloud Build Editor (roles/cloudbuild.builds.editor) and Cloud Storage Object Admin (roles/storage.objectAdmin) roles on the project.....
Using the IAM page you can add the email address of the person deploying (the one you have authenticated via gcloud) and add the above roles to that email
[1]: https://cloud.google.com/iam/docs/understanding-roles?hl=en#app-engine-roles

How to run a Cloud Scheduler job with App Engine HTTP when protected by IAP

I have a Python app on Google App Engine Standard which is secured using Google Cloud Identity Aware Proxy (IAP).
I would like to trigger a part of my app every day by using Cloud Scheduler. (It is calling an API, doing calculations, and stores results in Google Cloud Storage.)
To do so, I tried triggering a POST request on an "App Engine HTTP". example URL: "/refresh_data"
When running the job, I get the following error:
jsonPayload: {
status: "FAILED_PRECONDITION"
relativeUrl: "/refresh_data"
jobName: "..."
#type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
targetType: "APP_ENGINE_HTTP"
}
I can't find any documentation relative to the "FAILED_PRECONDITION" error so I'm feeling kind of stuck here.
As an alternative, I tried to trigger a POST request to a simple "HTTP" and by granting the Owner access level in IAP to the service account I am using for Cloud Scheduler. This time, the error message I get is the following :
jsonPayload: {
status: "UNAUTHENTICATED"
#type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
jobName: "..."
targetType: "HTTP"
url: "https:.../refresh_data"
}
I really don't understand how to make this work... The Cloud Scheduler documentation barely documents the "App Engine HTTP" option, and nothing is documented relative to te use of IAP...
Any help would be much appreciated :)
It's the dark side of IAP. I sent this feedback to Google months ago. Same thing with Pubsub, you can't trigger and App Engine IAP protected from these serverless product, even if you use a service account with the correct authorization.
You can check this relevant documentation from IAP on how to authenticate with a Service Account.
Whenever you are using Cloud Scheduler, the requests will be done with its Service Account, therefore the guide to follow should be the one linked above as you are basically trying to authenticate programmatically rather than with the Google sign-in.
That being said, you would need to generate the OIDC token prior to making a request to an IAP-secured endpoint. Once you have the OIDC token, it needs to be included in the Authorization: Bearer header.
I had a similar problem when upgrading my GAE app from python 2.7 to python3 (standard).
I got the same error message as you (status: "FAILED_PRECONDITION") when running my previous cron.yaml jobs set up as cloud scheduler jobs.
And the upload of previous cron.yaml files did not work to run either.
Then I found out that just adding an ending '/' on the url fixed it.
So a cron like:
cron:
- description: competition participants pilot list update
url: /daily1/
schedule: every 8 hours from 05:00 to 21:00
worked after being uploaded with cloud SDK:
gcloud app deploy cron.yaml --project my-gae-project
(and I forgot that) I also had to:
gcloud services enable cloudscheduler.googleapis.com --project my-gae-project
I got this working recently for HTTP endpoints behind an IAP (we use cloud run, but it should be similar for any IAP protected HTTP endpoint). This is the Cloud Scheduler configuration that worked for me:
Auth Header: "Add OIDC Token"
Service account: Select a service account with the role "IAP-secured Web App User" (if you recently added this permission, it can take a few minutes to propagate)
Audience: Your IAP's Client ID (from here: https://console.cloud.google.com/apis/credentials)

Insufficient authentication scopes from Jenkins build via Google Compute Engine

I searched the web for a while, but can't seem to find the right answer.
I have created a VM Instance on Google Compute Engine and I am running Jenkins on it. This instance checks a code repository I have and when a change occurs, I want to run the following command
gcloud --project=test preview app deploy -q app.yaml --version=dev
When I want to trigger a build, or it triggers it by himself, I get the following error:
Beginning deployment...
ERROR: Error Response: [403] Request had insufficient authentication scopes.
ERROR: (gcloud.preview.app.deploy) Could not retrieve the default Google Cloud Storage bucket for [test].
Please try again or use the [bucket] argument.
The VM instance does have acces to the storage as you can see in the following image, so I don't understand why this error pops up.
Go to the App Engine developer console https://appengine.google.com/ and in
Application Settings > Cloud integration > and turn ON the Google Cloud Storage
so this will set up the default bucket for you. This something that the new Google Developer Console will do automatically.If the project is created on the old console you have to do it manually.

Google App Engine: "The backend is locked down"

I am trying to test out the Google App Engine. I have created an appspot domain, downloaded the sample project and put in all credentials required. I have also enabled Google Cloud Messaging for Android.
When I hit the send button (in the samples GuestBookActivity), I get the message:
"The backend is locked down. The administrator can change the authentication/authorization settings on [my appspot domain]".
Does anyone know about this issue and how to resolve it?
As #Peter Knego said, see point 8 here: https://developers.google.com/cloud/samples/mbs/deploy_backend
I had forgot to open my server for development use.

Resources