I am following this tutorial to use Google Cloud Storage from my AppEngine application. Older storage libraries are working fine, but after migrating to Google Cloud Java Libraries, I can't read my buckets anymore.
This is the tutorial I am following: https://github.com/googleapis/java-storage
In which the key part is the usage of Application Default Credentials: https://github.com/googleapis/java-storage#creating-an-authorized-service-object
To make authenticated requests to Google Cloud Storage, you must create a service object with credentials. You can then make API calls by calling methods on the Storage service object. The simplest way to authenticate is to use Application Default Credentials. These credentials are automatically inferred from your environment, so you only need the following code to create your service object:
Storage storage = StorageOptions.getDefaultInstance().getService();
I hope to be able to do this because this is running inside AppEngine, so Storage lives in the same project as the AppEngine code.
I am using the default service account. This service account has "owner" role in IAM configuration, and "storage object administrator" in the bucket itself.
However, I keep getting this login error:
com.google.cloud.storage.StorageException: Anonymous caller does not have storage.buckets.get access to the Google Cloud Storage bucket.
at com.google.cloud.storage.StorageException.translate(StorageException.java:118)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:287)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:479)
at com.google.cloud.storage.StorageImpl.lambda$get$4(StorageImpl.java:270)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.Retrying.run(Retrying.java:54)
at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1406)
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:269)
Probably I am missing some configuration or IAM access. Thank you in advance.
Related
I have an AppEngine application that is behind an IAP (identity-aware proxy), so it receives requests that are authenticated and include a JWT token. From the AppEngine application I want to make a request to the Google Sheets API. That also requires an authenticated connection, but given that I want that connection to be made under the same user that accessed the application via the IAP, does anyone know how to create a request from inside the AppEngine application that will forward the token to Google Sheets? Cannot find any information on the subject... I am using Java, so any Java pointers would be appreciated, but general/other language help is good too...
I will describe the 2 approach proposed in the comment
The first one, to reuse the IAP proxy token to access Google Sheet is impossible, and dangerous.
Impossible because you receive an identity token from IAP (at least the requester/browser send an identity token to IAP) and you need an access token to request Sheet APIs.
Dangerous because, if you are able to reuse the IAP token to request the Google Sheet, that means the user is authorized to access to the Google Sheet. And I'm sure that you build an app to prevent any direct access/modification to the Google Sheet.
The second one, is to use a technical account (typically a service account) and generate an access token to access the Sheet API.
This second approach is the best one (don't forget to correctly log the user request and the subsequent sheet API calls in your AppEngine app to have the end to end traceability). BUT, and it's for that you ask this question, it's impossible with the App Engine default service account.
In fact, to access to the Sheet API, you need to scope your access token with the Sheet API. Sadly, you can't do this with App Engine. You can do this with Cloud Run, Cloud Functions, Compute Engine (without the default service account, else you need an extra config to achieve this with the Compute Engine default service account). But not with App Engine.
So, you have 2 solutions:
Either you use another hosting platform (Cloud Run for example), but you loose the IAP capacity (for now)
You continue to use App Engine but you need to request an access token to another service account (it's not required to have a service account key file). You can use the Service Account Credential API for this. I wrote an article on this API
Note: later in 2021, App Engine should be able to accept custom service account, and thus the issue should be solved
If I create custom service account with limited permissions, can I set this new service account to be the Application Default Credential?
By default inside app engine there is environment variable GOOGLE_APPLICATION_CREDENTIALS, that is set to the (ADC), which is the default service account credentials. Can I modify this?
I am aware that I can set this environment variable to point to my custom service account credentials file locally, but I do not see a clear way to do this in app engine.
ADC is not a credential. ADC is a method of finding credentials.
Google Cloud Application Default Credentials
App Engine Standard uses one default service account per project. All App Engine Standard services in a project use the same service account. You can modify the default service account but you cannot change the default service account to use a different service account (as you can with Compute Engine).
If you decide to create a new service account for usage within your application, do not store the service account in your application. This means forget about GOOGLE_APPLICATION_CREDENTIALS. Instead, store the service account in Google Secret Manager (better) or Google Cloud Storage (OK with the right permissions). Load the service account JSON data during application startup.
Google Secret Manager
I created a custom tensorflow model and deployed to google cloud AI platform. To access the online prediction at (https://ml.googleapis.com/v1/projects/my-project/models/my-model/versions/my-version:predict),
I can use default auth token got with the following command:
access_token=$(gcloud auth application-default print-access-token)
or, according to: https://cloud.google.com/ml-engine/docs/python-client-library, I can create a service account key, download it, and use it to in an outh2 authentication protocol.
The problem arises when I want to access the online prediction from another API that I am deploying in Google Cloud Platform: in this case the command above doesn't work and I can't access to any file to sign my jwt for authentication. How can I create my jwt without accessing to any file?
You want to send and consume Cloud Pub/Sub messages from your App Engine application. The Cloud Pub/Sub API is currently disabled. You will use a service account to authenticate your application to the API. You want to make sure your application can use Cloud Pub/Sub. What should
you do?
A. Enable the Cloud Pub/Sub API in the API Library on the GCP Console.
B. Rely on the automatic enablement of the Cloud Pub/Sub API when the Service Account accesses it.
C. Use Deployment Manager to deploy your application. Rely on the automatic enablement of all APIs used by the application being deployed.
D. Grant the App Engine Default service account the role of Cloud Pub/Sub Admin. Have your application enable the API on the first connection to Cloud Pub/Sub.
I realized that if I build an app that uses pubsub. it works without manually enabled the pubsub api. But I couldn't find anything in the documentation about that. Which is the correct answer?
You want to send and consume Cloud Pub/Sub messages from your App
Engine application. The Cloud Pub/Sub API is currently disabled. You
will use a service account to authenticate your application to the
API. You want to make sure your application can use Cloud Pub/Sub.
What should you do?
Let's analyze each possible answer to determine the best answer.
A. Enable the Cloud Pub/Sub API in the API Library on the GCP Console.
This is a possible answer. The standard method is to enable services in the Google Cloud Console. You can also enable services with the Cloud SDK CLI gcloud services enable pubsub.googleapis.com
B. Rely on the automatic enablement of the Cloud Pub/Sub API when the
Service Account accesses it.
This is not a possible answer. Google Cloud Services are not automatically enabled when the service account accesses it. First, service accounts do not access APIs. Service accounts are used to obtain an OAuth Access Token (or Identity Token). These tokens are used to authorize APIs. Services are not automatically enabled with an API makes first access.
C. Use Deployment Manager to deploy your application. Rely on the
automatic enablement of all APIs used by the application being
deployed.
This is not a possible answer. Deployment Manager does not automatically enable services. You can use Deployment Manager Resource Types to enable services. You must create a virtual resource for each API that you want enabled.
D. Grant the App Engine Default service account the role of Cloud
Pub/Sub Admin. Have your application enable the API on the first
connection to Cloud Pub/Sub.
This is not a possible answer. Cloud Pub/Sub Admin does not have permissions to enable services. To enable services the service account (or User Account) will need roles/serviceusage.serviceUsageAdmin or another role with the permission serviceusage.services.enable.
Drumroll Please ....
Therefore the best answer is A in my opinion.
I created a service account and gave it Viewer access in IAM. Downloaded p12 key for the service account and used it to access Cloud Project using remote_api. Even though only Project Viewer access is granted, I can update data-store entities using the remote_api.
Is this a bug or expected behavior? If it is expected, how can I give read only access via remote_api to the Cloud Project?