I want to run a cron job on GAE that internally calls BigQuery.
I am currently able to run BigQuery but I need to log in with my credentials. But I would like to run the cron job for BigQuery without any login.
Any help would be highly appreciated.
I know it's not the java you're expecting. The secret is to use AppAssertionCredentials.
Here is the python sample:
from apiclient.discovery import build
from oauth2client.appengine import AppAssertionCredentials
import httplib2
from google.appengine.api import memcache
scope = 'https://www.googleapis.com/auth/bigquery'
credentials = AppAssertionCredentials(scope=scope)
http = credentials.authorize(httplib2.Http(memcache))
return build("bigquery", "v2", http=http)
I am successfully able to implement it in java.Before performing it we need to generate client id for service account.
private static final HttpTransport TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
private static Bigquery bigquery;
AppIdentityCredential credential = new AppIdentityCredential(Collections.singleton(BigqueryScopes.BIGQUERY));
bigquery = new Bigquery.Builder(TRANSPORT, JSON_FACTORY, credential)
.setApplicationName("BigQuery-Service-Accounts/0.1")
.setHttpRequestInitializer(credential).build();
Related
I am trying to invoke a Cloud Run service using Cloud Tasks as described in the docs here.
I have a running Cloud Run service. If I make the service publicly accessible, it behaves as expected.
I have created a cloud queue and I schedule the cloud task with a local script. This one is using my own account. The script looks like this
from google.cloud import tasks_v2
client = tasks_v2.CloudTasksClient()
project = 'my-project'
queue = 'my-queue'
location = 'europe-west1'
url = 'https://url_to_my_service'
parent = client.queue_path(project, location, queue)
task = {
'http_request': {
'http_method': 'GET',
'url': url,
'oidc_token': {
'service_account_email': 'my-service-account#my-project.iam.gserviceaccount.com'
}
}
}
response = client.create_task(parent, task)
print('Created task {}'.format(response.name))
I see the task appear in the queue, but it fails and retries immediately. The reason for this (by checking the logs) is that the Cloud Run service returns a 401 response.
My own user has the roles "Service Account Token Creator" and "Service Account User". It doesn't have the "Cloud Tasks Enqueuer" explicitly, but since I am able to create the task in the queue, I guess I have inherited the required permissions.
The service account "my-service-account#my-project.iam.gserviceaccount.com" (which I use in the task to get the OIDC token) has - amongst others - the following roles:
Cloud Tasks Enqueuer (Although I don't think it needs this one as I'm creating the task with my own account)
Cloud Tasks Task Runner
Cloud Tasks Viewer
Service Account Token Creator (I'm not sure whether this should be added to my own account - the one who schedules the task - or to the service account that should perform the call to Cloud Run)
Service Account User (same here)
Cloud Run Invoker
So I did a dirty trick: I created a key file for the service account, downloaded it locally and impersonated locally by adding an account to my gcloud config with the key file. Next, I run
curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://url_to_my_service
That works! (By the way, it also works when I switch back to my own account)
Final tests: if I remove the oidc_token from the task when creating the task, I get a 403 response from Cloud Run! Not a 401...
If I remove the "Cloud Run Invoker" role from the service account and try again locally with curl, I also get a 403 instead of a 401.
If I finally make the Cloud Run service publicly accessible, everything works.
So, it seems that the Cloud Task fails to generate a token for the service account to authenticate properly at the Cloud Run service.
What am I missing?
I had the same issue here was my fix:
Diagnosis: Generating OIDC tokens currently does not support custom domains in the audience parameter. I was using a custom domain for my cloud run service (https://my-service.my-domain.com) instead of the cloud run generated url (found in the cloud run service dashboard) that looks like this: https://XXXXXX.run.app
Masking behavior: In the task being enqueued to Cloud Tasks, If the audience field for the oidc_token is not explicitly set then the target url from the task is used to set the audience in the request for the OIDC token.
In my case this meant that enqueueing a task to be sent to the target https://my-service.my-domain.com/resource the audience for the generating the OIDC token was set to my custom domain https://my-service.my-domain.com/resource. Since custom domains are not supported when generating OIDC tokens, I was receiving 401 not authorized responses from the target service.
My fix: Explicitly populate the audience with the Cloud Run generated URL, so that a valid token is issued. In my client I was able to globally set the audience for all tasks targeting a given service with the base url: 'audience' : 'https://XXXXXX.run.app'. This generated a valid token. I did not need to change the url of the target resource itself. The resource stayed the same: 'url' : 'https://my-service.my-domain.com/resource'
More Reading:
I've run into this problem before when setting up service-to-service authentication: Google Cloud Run Authentication Service-to-Service
1.I created a private cloud run service using this code:
import os
from flask import Flask
from flask import request
app = Flask(__name__)
#app.route('/index', methods=['GET', 'POST'])
def hello_world():
target = os.environ.get('TARGET', 'World')
print(target)
return str(request.data)
if __name__ == "__main__":
app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))
2.I created a service account with --role=roles/run.invoker that I will associate with the cloud task
gcloud iam service-accounts create SERVICE-ACCOUNT_NAME \
--display-name "DISPLAYED-SERVICE-ACCOUNT_NAME"
gcloud iam service-accounts list
gcloud run services add-iam-policy-binding SERVICE \
--member=serviceAccount:SERVICE-ACCOUNT_NAME#PROJECT-ID.iam.gserviceaccount.com \
--role=roles/run.invoker
3.I created a queue
gcloud tasks queues create my-queue
4.I create a test.py
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import datetime
# Create a client.
client = tasks_v2.CloudTasksClient()
# TODO(developer): Uncomment these lines and replace with your values.
project = 'your-project'
queue = 'your-queue'
location = 'europe-west2' # app engine locations
url = 'https://helloworld/index'
payload = 'Hello from the Cloud Task'
# Construct the fully qualified queue name.
parent = client.queue_path(project, location, queue)
# Construct the request body.
task = {
'http_request': { # Specify the type of request.
'http_method': 'POST',
'url': url, # The full url path that the task will be sent to.
'oidc_token': {
'service_account_email': "your-service-account"
},
'headers' : {
'Content-Type': 'application/json',
}
}
}
# Convert "seconds from now" into an rfc3339 datetime string.
d = datetime.datetime.utcnow() + datetime.timedelta(seconds=60)
# Create Timestamp protobuf.
timestamp = timestamp_pb2.Timestamp()
timestamp.FromDatetime(d)
# Add the timestamp to the tasks.
task['schedule_time'] = timestamp
task['name'] = 'projects/your-project/locations/app-engine-loacation/queues/your-queue/tasks/your-task'
converted_payload = payload.encode()
# Add the payload to the request.
task['http_request']['body'] = converted_payload
# Use the client to build and send the task.
response = client.create_task(parent, task)
print('Created task {}'.format(response.name))
#return response
5.I run the code in Google Cloud Shell with my user account which has Owner role.
6.The response received has the form:
Created task projects/your-project/locations/app-engine-loacation/queues/your-queue/tasks/your-task
7.Check the logs, success
The next day I am no longer able to reproduce this issue. I can reproduce the 403 responses by removing the Cloud Run Invoker role, but I no longer get 401 responses with exactly the same code as yesterday.
I guess this was a temporary issue on Google's side?
Also, I noticed that it takes some time before updated policies are actually in place (1 to 2 minutes).
For those like me, struggling through documentation and stackoverflow when having continuous UNAUTHORIZED responses on Cloud Tasks HTTP requests:
As was written in thread, you better provide audience for oidcToken you send to CloudTasks. Ensure your requested url exactly equals to your resource.
For instance, if you have Cloud Function named my-awesome-cloud-function and your task request url is https://REGION-PROJECT-ID.cloudfunctions.net/my-awesome-cloud-function/api/v1/hello, you need to ensure, that you set function url itself.
{
serviceAccountEmail: SERVICE-ACCOUNT_NAME#PROJECT-ID.iam.gserviceaccount.com,
audience: https://REGION-PROJECT-ID.cloudfunctions.net/my-awesome-cloud-function
}
Otherwise seems full url is used and leads to an error.
I am using the newer version of PubSub - Publisher API
I have a P12 file and am building the credential like this:
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(transport)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(serviceAccount)
.setServiceAccountScopes(Arrays.asList("https://www.googleapis.com/auth/pubsub"))
.setServiceAccountPrivateKeyFromP12File(new File(keyFile))
.build();
How do I set the credentials on the Publisher?
Also, is there a way to get the static scope string "https://www.googleapis.com/auth/pubsub"
I found the question answered here in case anyone runs into this:
Basically,
Publisher
.defaultBuilder(topic)
.setChannelProvider(TopicAdminSettings
.defaultChannelProviderBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(yourCredentialsHere))
.build())
.build();
I am working on a product that is supposed to be installed in Google App Engine.
In this I am using Service account for authenticating Gmail API, Drive API, Calendar API etc.
Its working fine with downloaded P12 file as authentication. But as its product I don't want client to download and upload on app on every install.
Can there be a way to authenticate it without privatekey file or using that API without service account.
In below page its mentioned that there is System-managed key-pairs are managed automatically by Google. Can it be helpful? I did't find any example of it.
https://cloud.google.com/iam/reference/rest/v1/projects.serviceAccounts.keys
In below link it suggest that for Google Cloud Platform I should use Google Managed Key
https://cloud.google.com/iam/docs/understanding-service-accounts
Can this key used without downloaded file ?
Thanks
I could achieve it by IAM API
https://cloud.google.com/iam/reference/rest/v1/projects.serviceAccounts.keys
Below is Java code for it
AppIdentityCredential credential = new AppIdentityCredential(
Arrays.asList("https://www.googleapis.com/auth/cloud-platform"));
Iam iam = new Iam(httpTRANSPORT, jsonFACTORY, credential);
try {
Iam.Projects.ServiceAccounts.Keys.Create keyCreate = iam.projects().serviceAccounts().keys()
.create("projects/myProject/serviceAccounts/myProject#appspot.gserviceaccount.com", new CreateServiceAccountKeyRequest());
ServiceAccountKey key = keyCreate.execute();
} catch (IOException e) {
System.out.println(e.getMessage());
}
Any key can be used to generate GoogleCredential as below
InputStream stream = new ByteArrayInputStream(key.decodePrivateKeyData());
GoogleCredential credential = GoogleCredential.fromStream(stream);
I am trying to get an image in my App Engine backend and every time I try to get it I get the following error
com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable
{
"code": 503,
"errors": [
{
"domain": "global",
"message": "java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.",
"reason": "backendError"
}
],
"message": "java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information."
}
Now it was my understanding that when making a request from App Engine backend that the Application Default Credentials was sufficient enough to do it.
The Application Default Credentials provide a simple way to get
authorization credentials for use in calling Google APIs.
They are best suited for cases when the call needs to have the same
identity and authorization level for the application independent of
the user. This is the recommended approach to authorize calls to
Google Cloud APIs, particularly when you're building an application
that is deployed to Google App Engine or Google Compute Engine virtual
machines.
taken from here
This is how I am trying to get the image using the Java API
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
GoogleCredential credential = GoogleCredential.getApplicationDefault();
if(credential.createScopedRequired()){
credential = credential.createScoped(StorageScopes.all());
}
Storage.Builder storageBuilder = new Storage.Builder(httpTransport,new JacksonFactory(),credential);
Storage storage = storageBuilder.build();
Storage.Objects.Get getObject = storage.objects().get("myBucket", name);
ByteArrayOutputStream out = new ByteArrayOutputStream();
getObject.getMediaHttpDownloader().setDirectDownloadEnabled(false);
getObject.executeMediaAndDownloadTo(out);
byte[] oldImageData = out.toByteArray();
out.close();
ImagesService imagesService = ImagesServiceFactory.getImagesService();
Image oldImage = ImagesServiceFactory.makeImage(oldImageData);
Transform resize = ImagesServiceFactory.makeResize(width, height);
return imagesService.applyTransform(resize, oldImage);
am I just using the credentials wrong or can I not use the application default credentials?
If you want to access your Google Cloud Storage data from App Engine. You should be using the Google Cloud Storage Client Library
Github Project
Say you have a user who is authenticated via Google Accounts come in to your app at /run and then you need to spin off something in the default task queue for that user. Right now, that user's credentials are not carried over to the task servlet (e.g. /worker). Is there a way to pass them? Maybe by taking headers from the orig request and adding them to the Task Queue request?
If you're using Java Servlet, you can use this logic:
After successfully sign in (using Google+ Sign In and Endpoint), store user credentials in HttpSession as attributes (Put HttpServletRequest in your Endpoint API as parameter, and just get the session).
In your servlet logic, get the session, and retrieve the credential.
P.S. Don't forget to set <sessions-enabled>true</sessions-enabled> in your appengine-web.xml.
Put user credentials into datastore from your app, and read them from datastore in your task.
Example user model in Python with ndb and oauth2client:
from google.appengine.ext import ndb
from oauth2client import appengine
class User(ndb.Model):
# among other user props
credentials = appengine.CredentialsNDBProperty()