App Engine deploy: Permissions error fetching application - google-app-engine

We are using a service account to deploy our app to App Engine using Travis.
On every merged PR, Travis pulls the code from our GitHub repository, and pulls a Docker image which contains Google Cloud SDK and executes the gcloud app deploy command. We use a Service Account to perform the deployment with "Project Owner" role.
Everything used to work fine until I added a new service to the project which automates SSL certificate generation and renewal, along with a dispatch.yaml file to route traffic incoming from Let's Encrypt for domain verification. I needed to add more permissions to allow updating the SSL certificates we use for our custom domain. I removed the current service account, and created a new one with a new private key. I created a new role with the required permissions to update and view SSL certificates in addition to the previous permissions (all appengine.* permissions). I assigned the new role and the Project Owner role to the new account. After these changes, the deployment fails with the following error when executing the deploy command:
Permissions error fetching application [apps/hollowverse-c9cad]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.
I used the same service account on my local machine with logging level set to debug. I got this error:
DEBUG: HttpError accessing <https://appengine.googleapis.com/v1/apps/hollowverse-c9cad?alt=json>: response: <{'status': '403', 'content-length': '335', 'x-xss-protection': '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Wed, 02 Aug 2017 14:33:50 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443"; ma=2592000; v="39,38,37,36,35"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 403,
"message": "Operation not allowed",
"status": "PERMISSION_DENIED",
"details": [
{
"#type": "type.googleapis.com/google.rpc.ResourceInfo",
"resourceType": "gae.api",
"description": "The \"appengine.applications.get\" permission is required."
}
]
}
}
>
DEBUG: (gcloud.beta.app.deploy) Permissions error fetching application [apps/hollowverse-c9cad]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.
The description says that appengine.applications.get is required to perform the deployment. Looking at the permissions granted to the role assigned to the Travis account we use to deploy, appengine.applications.get is clearly granted:
I assigned every possible App Engine and Project role to the account, but deployment still fails with the same error. However, using the default service account, which is automatically created for every new project on GCP, seems to be working.

I removed the current service account, and created a new one with a new private key.
This is where it went wrong. The new account had the same ID as the previous one. Although I could not find this behavior documented anywhere, it looks like that once an ID is used for a service account, it cannot be used again for a new account, even if the previous one is removed.
We created a new account with a new ID (travis2#hollowverse-c9cad.iam.gserviceaccount.com) instead of travis#hollowverse-c9cad.iam.gserviceaccount.com) and the issue is now resolved.

Related

ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: unable to resolve source

I am trying to deploy a go 1.11 runtime that used to work, but recently I've been getting: ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: unable to resolve source errors.
Nothing in my app.yaml has changed, and the error message isn't helpful to understand what the issue could be. I ran it with the --verbosity=debug flag and get:
Building and pushing image for service [apiv1]
DEBUG: Could not call git with args ('config', '--get-regexp', 'remote\\.(.*)\\.url'): Command '['git', 'config', '--get-regexp', 'remote\\.(.*)\\.url']' returned non-zero exit status 1
INFO: Could not generate [source-context.json]: Could not list remote URLs from source directory: /var/folders/18/k3w6w7f169xg4mypdwj7p4_c0000gn/T/tmp6IkZKx/tmphibUAo
Stackdriver Debugger may not be configured or enabled on this application. See https://cloud.google.com/debugger/ for more information.
INFO: Uploading [/var/folders/18/k3w6w7f169xg4mypdwj7p4_c0000gn/T/tmpVHKXol/src.tgz] to [staging.wildfire-app-backend.appspot.com/asia.gcr.io/wildfire-app-backend/appengine/apiv1.20190506t090359:latest]
DEBUG: Using runtime builder root [gs://runtime-builders/]
DEBUG: Loading runtimes manifest from [gs://runtime-builders/runtimes.yaml]
INFO: Reading [<googlecloudsdk.api_lib.storage.storage_util.ObjectReference object at 0x105ca9b10>]
DEBUG: Resolved runtime [go1.11] as build configuration [gs://runtime-builders/go-1.11-builder-20181217154124.yaml]
INFO: Using runtime builder [gs://runtime-builders/go-1.11-builder-20181217154124.yaml]
INFO: Reading [<googlecloudsdk.api_lib.storage.storage_util.ObjectReference object at 0x105b03b50>]
DEBUG: (gcloud.app.deploy) INVALID_ARGUMENT: unable to resolve source
Traceback (most recent call last):
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 985, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 795, in Run
resources = command_instance.Run(args)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/surface/app/deploy.py", line 90, in Run
parallel_build=False)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 636, in RunDeploy
flex_image_build_option=flex_image_build_option)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 411, in Deploy
image, code_bucket_ref, gcr_domain, flex_image_build_option)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 287, in _PossiblyBuildAndPush
self.deploy_options.parallel_build)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/deploy_command_util.py", line 450, in BuildAndPushDockerImage
return _SubmitBuild(build, image, project, parallel_build)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/deploy_command_util.py", line 483, in _SubmitBuild
build, project=project)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/api_lib/cloudbuild/build.py", line 149, in ExecuteCloudBuild
build_op = self.ExecuteCloudBuildAsync(build, project)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/api_lib/cloudbuild/build.py", line 133, in ExecuteCloudBuildAsync
build=build,))
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/third_party/apis/cloudbuild/v1/cloudbuild_v1_client.py", line 205, in Create
config, request, global_params=global_params)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/third_party/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/third_party/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/third_party/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
HttpBadRequestError: HttpError accessing <https://cloudbuild.googleapis.com/v1/projects/wildfire-app-backend/builds?alt=json>: response: <{'status': '400', 'content-length': '114', 'x-xss-protection': '0'
, 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Mon, 06 May 2
019 16:04:41 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443"; ma=2592000; v="46,44,43,39"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 400,
"message": "unable to resolve source",
"status": "INVALID_ARGUMENT"
}
}
>
ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: unable to resolve
Any advice would be useful, I also tried it with gcloud beta, I rotated my credentials and was of no use. My user has Owner role, but I added individually all the roles that might be necessary
App Engine Admin
App Engine Code Viewer
App Engine Deployer
App Engine Service Admin
Project Billing Manager
Cloud Build Service Account
Cloud Build Editor
Cloud Build Viewer
Owner
Storage Admin
Try disabling and enabling the Cloud Build API in your project so a new service account is created, then try to deploy again.
This ensures that Cloud Build has permission to start builds (the service account name should look like project-number#cloudbuild.gserviceaccount.com).
I found that it took a while for the API permissions to propagate fully.
I had this error after enabling the App Engine Admin API, but I waited a few minutes and tried again and it worked fine.
Cloud Build uses a service account instead of your user credential, so adding roles to your user won't help in this case.
You should be able to manually add the service account, [project-number]#cloudbuild.gserviceaccount.com, back to your account without re-enabling the API. Some users are known to delete random service accounts they don't recognize, which can sometimes cause this issue.
In summary, check if the service account exists. If so, give it all the roles it needs; if not, then create it and give it all the role it needs.

Permission denied during gcloud app deploy using Google Cloud SDK

It is insanely hard, to deploy an app to Google App Engine, using Google Cloud SDK.
I had tried the below 2 commands
C:\Users\yccheok\Desktop\jstock-android-appengine>gcloud config set project jstock-android
Updated property [core/project].
C:\Users\yccheok\Desktop\jstock-android-appengine>gcloud app deploy app.yaml --log-http --verbosity=debug
DEBUG: Running [gcloud.app.deploy] with arguments: [--log-http: "true", --verbosity: "debug", DEPLOYABLES:1: "['app.yaml']"]
DEBUG: No staging command found for runtime [python27] and environment [STANDARD].
DEBUG: API endpoint: [https://appengine.googleapis.com/], API version: [v1]
=======================
==== request start ====
uri: https://appengine.googleapis.com/v1/apps/jstock-android?alt=json
method: GET
== headers start ==
Authorization: Bearer ya29.GlxEBb1XVP1JK93-ARiaN_ZgiMbvZmw5KWfvJVfibDJ4FK_ZaMRoU1jVDTiWzsY606GSduJKJd9Nm8zA-_Iql5mGn4AMk4QVl8mPRycfekeZnOOHtbUvpkBMgOLOQA
accept: application/json
accept-encoding: gzip, deflate
content-length: 0
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/184.0.0 command/gcloud.app.deploy invocation-id/c9ae232d33b346d787b95a36e28c38c0 environment/None environment-version/None interactive/True python/2.7.13 (Windows NT 10.0.16299)
== headers end ==
== body start ==
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: hq=":443"; ma=2592000; quic=51303431; quic=51303339; quic=51303338; quic=51303337; quic=51303335,quic=":443"; ma=2592000; v="41,39,38,37,35"
cache-control: private
content-length: 335
content-type: application/json; charset=UTF-8
date: Tue, 16 Jan 2018 19:16:21 GMT
server: ESF
status: 403
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
{
"error": {
"code": 403,
"message": "Operation not allowed",
"status": "PERMISSION_DENIED",
"details": [
{
"#type": "type.googleapis.com/google.rpc.ResourceInfo",
"resourceType": "gae.api",
"description": "The \"appengine.applications.get\" permission is required."
}
]
}
}
-- body end --
total round trip time (request+response): 1.796 secs
---- response end ----
----------------------
DEBUG: (gcloud.app.deploy) Permissions error fetching application [apps/jstock-android]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\calliope\cli.py", line 797, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\calliope\backend.py", line 757, in Run
resources = command_instance.Run(args)
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\surface\app\deploy.py", line 65, in Run
parallel_build=False)
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 543, in RunDeploy
app = _PossiblyCreateApp(api_client, project)
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 703, in _PossiblyCreateApp
api_client._FormatApp()))) # pylint: disable=protected-access
HttpException: Permissions error fetching application [apps/jstock-android]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.
ERROR: (gcloud.app.deploy) Permissions error fetching application [apps/jstock-android]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.
C:\Users\yccheok\Desktop\jstock-android-appengine>
Then, I went through https://cloud.google.com/appengine/docs/admin-api/accessing-the-api , it mentioned I need to use Admin API. So, I do it step by step carefully.
Step 1
Step 2
It mentions Admin API is enabled. Now I need credential.
Step 3
Step 4
OK. Now they mention I don't need create new credential. I can use Application Default Credentials ?!
Step 5
So, I went to https://developers.google.com/identity/protocols/application-default-credentials?hl=en_GB . I learn that I need to run
C:\Users\yccheok\Desktop\jstock-android-appengine>gcloud auth application-default login
Your browser has been opened to visit:
https://accounts.google.com/o/oauth2/auth?redirect_uri=http%3A%2F%2Flocalhost%3A8085%2F&prompt=select_account&response_type=code&client_id=764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform&access_type=offline
Credentials saved to file: [C:\Users\yccheok\AppData\Roaming\gcloud\application_default_credentials.json]
These credentials will be used by any library that requests
Application Default Credentials.
Step 6
Step 7
Still, after completing the above 7 steps, I still get the exact same error message, when trying to run
gcloud app deploy app.yaml --log-http --verbosity=debug
Can anyone let me know, what step I'm still require, in order to deploy my Python app to Google App Engine, using Google Cloud SDK?
The Admin API is for programmatically deploying the app, not for deploying using gcloud app deploy, for which you don't even need the Admin API enabled for your app.
From Deploying a Python App:
To programmatically deploy your apps, use the Admin API.
Before you begin
Before you can deploy your app:
The Owner of the GCP project must create the App Engine application.
Ensure that your user account includes the required privileges.
(but I can see how the above could be mis-intrepreted as an invitation to use the Admin API)
Most likely the account actually used by gcloud app deploy is missing or doesn't have the required permissions.
You can check the acount used with gcloud auth list. In my case the account is my email address, not a service account (I'm not sure if a service account can be used).
Use gcloud auth login (and maybe gcloud auth revoke) if you need a different account.
And you can check the account's privileges (if any) on the project/app on the IAM Page.
One additional note on this,
When you enable the App Engine API and the cloud builder api, make sure the Cloud Build Service Account also has access to the project.
I ran into that problem after enabling the correct apis.
This was using a build trigger. I could deploy locally from a command line because I was authenticated as myself. However, if you are using a build trigger, it will use the build service account, which needs access.
Hope this helps.
I had this issue. In my case it was solved by setting the project using the project ID, rather than the project name. See this answer
gcloud app deploy ERROR: Permissions error fetching application [apps/<PROJECT_NAME>]

gcloud app deploy give 400 / forbidden error / cannot push img to google container registry

I am trying to deploy .net core application from google compute vm to google app engine using gcloud app deploy. I get the following error
> WARNING: We couldn't validate that your project is ready to deploy to App Engine Flexible Environment. If deployment fails, please check the following mess
age and try again:
Server responded with code [400]:
Bad Request Unexpected HTTP status 400.
Failed Project Preparation (app_id='s~project-id'). Out of retries. Last error: Temporary error occurred while verifying project: TEMPORARY_ERROR: Unabl
e to check API status
Beginning deployment of service [default]...
WARNING: Deployment of App Engine Flexible Environment apps is currently in Beta
Building and pushing image for service [default]
Some files were skipped. Pass `--verbosity=info` to see which ones.
ERROR: (gcloud.app.deploy) Could not copy [/tmp/tmpLwvVOb/src.tgz] to [us.gcr.io/project-id/appengine/default.20170118t043919:latest]: HttpError accessing
<https://www.googleapis.com/resumable/upload/storage/v1/b/staging.project-id.appspot.com/o?uploadType=resumable&alt=json&name=us.gcr.io%2Fcasepro-v3%2Fappe
ngine%2Fdefault.20170118t043919%3Alatest>: response: <{'status': '403', 'content-length': '166', 'vary': 'Origin, X-Origin', 'server': 'UploadServer', 'x-g
uploader-uploadid': 'AEnB2UqprxH-2tIhsSZdGxDOtS8UnWSI29YTo4kaptNK67SWJpLVqR0zEtCAHgFyE64wj1HfCyUL5sy9z4AZkTRFYuxXfdw5TA', 'date': 'Wed, 18 Jan 2017 04:40:0
0 GMT', 'alt-svc': 'quic=":443"; ma=2592000; v="35,34"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"errors": [
{
"domain": "global",
"reason": "forbidden",
"message": "Forbidden"
}
],
"code": 403,
"message": "Forbidden"
}
}
>. Please retry.
I have already enabled billing api, app engine admin api and storage api. Service a/c that is being used has editor rights. VM instance has been created using cloud launcher for Jenkins Bitnami package. I am trying to deploy app from command line from the vm before I configure Jenkins to do the same.
What to do to resolve this?
The problem is that gcloud app deploy is trying to deploy to the project id 'project-id', which cannot be your project id.
Try setting the project like this:
gcloud config set project MY-PROJECT-ID
Then, retry the gcloud app deploy command.
If this fails, please reply with your full gcloud command line, and the results of these two commands:
gcloud config list
gcloud version

Keycloak 403 (Forbidden) on Keycloak.loadUserProfile()

I am new to Keycloak. I have tried to get the user information using below code:
keycloakAuth.loadUserProfile().success(function(profile) {
debugger
console.log(profile);
}).error(function(res) {
debugger
console.log('Failed to load profile');
});
installation json is :
{
"realm": "CheckRealm",
"realm-public-key": "MIIBIjASDFJJK677132HJJAOCAQ8AMIIBCgKCAQEAgo4deAfr8BeqWOiCsddwMtH5nh8EK2cKIeInpt7LnoCyMsGj1HTP835HpslOURrR6Bgc42V7r6J/MIHqx3+KESTqpcQSe9ll6eUjzaMbIX2GPmy9OnviH6srUgOlDAKhL+/SDh/iv8RfErAPO9cxnoBWUUTyfyes1YRn34KILBdHZoaWk5mteJx9aV4bfA5tGTT6aF8o1NkCX1OUfXiaAD5sqKZV5vbI+QsOUsNshvGfE5JR2EpwZbWH/vRQVusxLURjC51v96ieQ8zUME5LwAQ0TgZcspHTb4Y+KuYRTuDQKuxRUYFeNbvqUGeT2s2sHMmWOQIDAQAB",
"auth-server-url": "https://something.com/auth",
"url": "https://something.com/auth",
"ssl-required": "external",
"resource": "sample-tracker",
"clientId": "sample-tracker",
"credentials": {
"secret": "bnmbsdk87wq"
}
}
I got the mentioned error :Failed to load resource: the server responded with a status of 403 (Forbidden)
To solve this situation you need to configure your keycloak or RedHat Single Sign-On (if you are using the supported version).
Open the keycloak admin console, for example: http://localhost:8180/auth/admin. Then click on Configure > clients and select the account client. The account client must be enabled.
Next, still in Configure > clients, select the client that you are trying to access and click on the scope tab. Add the view-profile role of account application.
The user also needs to have this permission.
There is a resolved issue about this in https://issues.jboss.org/browse/LIVEOAK-204 .

Permission denied writing from App Engine to associated Cloud Storage bucket

We have a project that runs on App Engine and creates files on Cloud Storage. The two are connected as being part of the same cloud platform project.
In App Engine we have a "Google APIs Console Project Number", and in Cloud Console -> Credentials we have that project number listed under "Client ID" (1[..........].apps.googleusercontent.com) and "Email Address" (1[..........]#developer.gserviceaccount.com).
Every morning, we have some cron jobs that upload files to our Cloud Storage bucket. This has worked flawlessly since September 2013 but as of this morning (Oct 16, 2014) we're getting "permission denied" errors from Cloud Storage.
We're using the cloudstorage client library, which raises cloudstorage.ForbiddenError. Here's the log & exception output:
Expect status [201] from Google Storage. But got status 403.
Path: u'/bucketname/icon_20141016.png'.
Request headers: {'x-goog-resumable': 'start', 'x-goog-api-version': '2', 'content-type': 'image/png', 'accept-encoding': 'gzip, *'}.
Response headers: {'alternate-protocol': '443:quic,p=0.01', 'content-length': '151', 'via': 'HTTP/1.1 GWA', 'x-google-cache-control': 'remote-fetch', 'vary': 'Origin', 'server': 'UploadServer ("Built on Oct 9 2014 15:35:27 (1412894127)")', 'date': 'Thu, 16 Oct 2014 11:56:10 GMT', 'content-type': 'application/xml; charset=UTF-8'}.
Extra info: None.
Since we're using the Cloud platform connection between the two services, I feel like I can only diagnose the problem on my production App Engine instance. I would prefer not to deploy new versions and risk breaking a production server. This also appears to be a Cloud Storage issue this morning, but the only status page I could find says everything is working fine.
As #tx802 suggested, I checked the bucket ACLs carefully.
$ gsutil getacl gs://bucket
<Entry>
<Scope type="UserByEmail">
<EmailAddress>1[..........]#developer.gserviceaccount.com</EmailAddress>
</Scope>
<Permission>FULL_CONTROL</Permission>
</Entry>
I looked at the App Engine application settings and saw the service account is actually appname#appspot.gserviceaccount.com, so I gave that account full control:
$ gsutil chacl -u appname#appspot.gserviceaccount.com:FC gs://bucket
I'm not sure what changed since yesterday's cron run, but now it succeeds.

Resources