I have written an application that uses all the clients/sdks as officially documented.
credentials = GoogleCredentials \
.get_application_default() \
.create_scoped('https://www.googleapis.com/auth/drive')
drive = discovery.build(
'drive',
'v3',
http=self.credentials.authorize(Http())
)
drive.files() \
.get(fileId=file_id) \
.execute()
It works perfect in local with a Service Account generated from the panel, but when I deploy the application, the service account within AppEngine flexible environment runs into problems.
17:15:04.000 /env/lib/python3.4/site-packages/oauth2client/contrib/gce.py:99: UserWarning: You have requested explicit scopes to be used with a GCE service account.
17:15:04.000 Using this argument will have no effect on the actual scopes for tokens
17:15:04.000 requested. These scopes are set at VM instance creation time and
17:15:04.000 can't be overridden in the request.
17:15:04.000
17:15:04.000 warnings.warn(_SCOPES_WARNING)
17:15:04.000 INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/discovery/v1/apis/drive/v3/rest
17:15:04.000 INFO:oauth2client.client:Attempting refresh to obtain initial access_token
17:15:04.000 INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/drive/v3/files/0B0Kn....M1pBNFE?alt=json
17:15:04.000 ERROR:root:Failed to retrieve file 0B0K....M1pBNFE. Is it shared with me? project-id#appspot.gserviceaccount.com
17:15:04.000 Traceback (most recent call last):
17:15:04.000 File "/home/vmagent/app/script.py", line 45, in get
17:15:04.000 .execute()
17:15:04.000 File "/env/lib/python3.4/site-packages/oauth2client/util.py", line 135, in positional_wrapper
17:15:04.000 return wrapped(*args, **kwargs)
17:15:04.000 File "/env/lib/python3.4/site-packages/googleapiclient/http.py", line 760, in execute
17:15:04.000 raise HttpError(resp, content, uri=self.uri)
17:15:04.000 googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/drive/v3/files/0B0Kn....M1pBNFE?alt=json returned "Insufficient Permission">
I have checked the permissions and they are all set. The problem is probably due to the "Using this argument will have no effect..." message, that appears when trying to create the scoped credentials.
As you've mentioned in a prior comment, this is a known issue. As described by araf...#google.com, it seems that App Engine instances in the flexible environment assume the credentials of the uderlying GCE VM as the application default credentials.
As a workaround in the meantime, you can use a manually created service account exported as a JSON key stored in your app, as per Using OAuth 2.0 for Server to Server Applications.
For anyone affected by this issue or for whom the workaround is ineffective, please post any relevant information on said issue.
Related
I can use a service principal to get an access-token from https://graph.microsoft.com but when I try to get a token for https://graph.microsoft.com/.default I get the following error.
What is possible impact if my token was issued without this scope?
Get Token request returned http error: 400 and server response:
{
"error": "invalid_resource",
"error_description": "AADSTS500011: The resource principal named https://graph.microsoft.com/.default was not found in the tenant named 4c000000-0000-0000-0000-0000000000. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant.
Trace ID: 00-00-00-00000
Correlation ID: 00-00-00-000
Timestamp: 2020-08-06 00:17:31Z
"error_codes": [ 500011 ],
"timestamp": "2020-08-06 00:17:31Z",
"trace_id": "d301a1cb-8feb-44e0-8b04-e463dd8d5b00",
"correlation_id": "92947479-d924-49fd-8e29-1d7cbe70d289",
"error_uri": "https://login.microsoftonline.com/error?code=500011"
}
In case anybody else wonders what the /.default scope is all about you can check out this Microsoft doc:
The /.default scope is built in for every application that refers to the static list of permissions configured on the application registration.
So basically, the /.default scope infers the permissions from the according application.
I noticed that you use the v1.0 endpoint to get the access token. It generally uses Resource as the request parameter.
For v1.0 endpoints, there is generally no need to use the /.default scope,you need to add the /.default scope only when you use the v2.0 endpoint to get the token.
So you can use this token with confidence,this has no impact.
I am trying to invoke a Cloud Run service using Cloud Tasks as described in the docs here.
I have a running Cloud Run service. If I make the service publicly accessible, it behaves as expected.
I have created a cloud queue and I schedule the cloud task with a local script. This one is using my own account. The script looks like this
from google.cloud import tasks_v2
client = tasks_v2.CloudTasksClient()
project = 'my-project'
queue = 'my-queue'
location = 'europe-west1'
url = 'https://url_to_my_service'
parent = client.queue_path(project, location, queue)
task = {
'http_request': {
'http_method': 'GET',
'url': url,
'oidc_token': {
'service_account_email': 'my-service-account#my-project.iam.gserviceaccount.com'
}
}
}
response = client.create_task(parent, task)
print('Created task {}'.format(response.name))
I see the task appear in the queue, but it fails and retries immediately. The reason for this (by checking the logs) is that the Cloud Run service returns a 401 response.
My own user has the roles "Service Account Token Creator" and "Service Account User". It doesn't have the "Cloud Tasks Enqueuer" explicitly, but since I am able to create the task in the queue, I guess I have inherited the required permissions.
The service account "my-service-account#my-project.iam.gserviceaccount.com" (which I use in the task to get the OIDC token) has - amongst others - the following roles:
Cloud Tasks Enqueuer (Although I don't think it needs this one as I'm creating the task with my own account)
Cloud Tasks Task Runner
Cloud Tasks Viewer
Service Account Token Creator (I'm not sure whether this should be added to my own account - the one who schedules the task - or to the service account that should perform the call to Cloud Run)
Service Account User (same here)
Cloud Run Invoker
So I did a dirty trick: I created a key file for the service account, downloaded it locally and impersonated locally by adding an account to my gcloud config with the key file. Next, I run
curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://url_to_my_service
That works! (By the way, it also works when I switch back to my own account)
Final tests: if I remove the oidc_token from the task when creating the task, I get a 403 response from Cloud Run! Not a 401...
If I remove the "Cloud Run Invoker" role from the service account and try again locally with curl, I also get a 403 instead of a 401.
If I finally make the Cloud Run service publicly accessible, everything works.
So, it seems that the Cloud Task fails to generate a token for the service account to authenticate properly at the Cloud Run service.
What am I missing?
I had the same issue here was my fix:
Diagnosis: Generating OIDC tokens currently does not support custom domains in the audience parameter. I was using a custom domain for my cloud run service (https://my-service.my-domain.com) instead of the cloud run generated url (found in the cloud run service dashboard) that looks like this: https://XXXXXX.run.app
Masking behavior: In the task being enqueued to Cloud Tasks, If the audience field for the oidc_token is not explicitly set then the target url from the task is used to set the audience in the request for the OIDC token.
In my case this meant that enqueueing a task to be sent to the target https://my-service.my-domain.com/resource the audience for the generating the OIDC token was set to my custom domain https://my-service.my-domain.com/resource. Since custom domains are not supported when generating OIDC tokens, I was receiving 401 not authorized responses from the target service.
My fix: Explicitly populate the audience with the Cloud Run generated URL, so that a valid token is issued. In my client I was able to globally set the audience for all tasks targeting a given service with the base url: 'audience' : 'https://XXXXXX.run.app'. This generated a valid token. I did not need to change the url of the target resource itself. The resource stayed the same: 'url' : 'https://my-service.my-domain.com/resource'
More Reading:
I've run into this problem before when setting up service-to-service authentication: Google Cloud Run Authentication Service-to-Service
1.I created a private cloud run service using this code:
import os
from flask import Flask
from flask import request
app = Flask(__name__)
#app.route('/index', methods=['GET', 'POST'])
def hello_world():
target = os.environ.get('TARGET', 'World')
print(target)
return str(request.data)
if __name__ == "__main__":
app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))
2.I created a service account with --role=roles/run.invoker that I will associate with the cloud task
gcloud iam service-accounts create SERVICE-ACCOUNT_NAME \
--display-name "DISPLAYED-SERVICE-ACCOUNT_NAME"
gcloud iam service-accounts list
gcloud run services add-iam-policy-binding SERVICE \
--member=serviceAccount:SERVICE-ACCOUNT_NAME#PROJECT-ID.iam.gserviceaccount.com \
--role=roles/run.invoker
3.I created a queue
gcloud tasks queues create my-queue
4.I create a test.py
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import datetime
# Create a client.
client = tasks_v2.CloudTasksClient()
# TODO(developer): Uncomment these lines and replace with your values.
project = 'your-project'
queue = 'your-queue'
location = 'europe-west2' # app engine locations
url = 'https://helloworld/index'
payload = 'Hello from the Cloud Task'
# Construct the fully qualified queue name.
parent = client.queue_path(project, location, queue)
# Construct the request body.
task = {
'http_request': { # Specify the type of request.
'http_method': 'POST',
'url': url, # The full url path that the task will be sent to.
'oidc_token': {
'service_account_email': "your-service-account"
},
'headers' : {
'Content-Type': 'application/json',
}
}
}
# Convert "seconds from now" into an rfc3339 datetime string.
d = datetime.datetime.utcnow() + datetime.timedelta(seconds=60)
# Create Timestamp protobuf.
timestamp = timestamp_pb2.Timestamp()
timestamp.FromDatetime(d)
# Add the timestamp to the tasks.
task['schedule_time'] = timestamp
task['name'] = 'projects/your-project/locations/app-engine-loacation/queues/your-queue/tasks/your-task'
converted_payload = payload.encode()
# Add the payload to the request.
task['http_request']['body'] = converted_payload
# Use the client to build and send the task.
response = client.create_task(parent, task)
print('Created task {}'.format(response.name))
#return response
5.I run the code in Google Cloud Shell with my user account which has Owner role.
6.The response received has the form:
Created task projects/your-project/locations/app-engine-loacation/queues/your-queue/tasks/your-task
7.Check the logs, success
The next day I am no longer able to reproduce this issue. I can reproduce the 403 responses by removing the Cloud Run Invoker role, but I no longer get 401 responses with exactly the same code as yesterday.
I guess this was a temporary issue on Google's side?
Also, I noticed that it takes some time before updated policies are actually in place (1 to 2 minutes).
For those like me, struggling through documentation and stackoverflow when having continuous UNAUTHORIZED responses on Cloud Tasks HTTP requests:
As was written in thread, you better provide audience for oidcToken you send to CloudTasks. Ensure your requested url exactly equals to your resource.
For instance, if you have Cloud Function named my-awesome-cloud-function and your task request url is https://REGION-PROJECT-ID.cloudfunctions.net/my-awesome-cloud-function/api/v1/hello, you need to ensure, that you set function url itself.
{
serviceAccountEmail: SERVICE-ACCOUNT_NAME#PROJECT-ID.iam.gserviceaccount.com,
audience: https://REGION-PROJECT-ID.cloudfunctions.net/my-awesome-cloud-function
}
Otherwise seems full url is used and leads to an error.
I am using a daemon auth API and I am able to get /groups but if try and use /groups/GROUPID/planner/plans I get a UnknownError and the message says
401 - Unauthorized: Access is denied due to invalid credentials.
You do not have permission to view this directory or page using the credentials
that you supplied.
I have Group.Read.All, Directory.Read.All, Group.ReadWrite.All, Directory.ReadWrite.All permissions as per the documentation. And I am using the https://learn.microsoft.com/en-us/graph/auth-v2-service?context=graph%2Fapi%2F1.0&view=graph-rest-1.0 (Get access without a user
) steps to get my token. I have got my administrator to click the Grant access (they are all "Granted").
Only planner stuff seem to be the issue (I can get, create, delete groups, and everything else) I am using v1.0 of the API and I tried beta both didn't work.
I checked my access token on jwt and it has
"roles": [
"Group.Read.All",
"Directory.ReadWrite.All",
"Group.ReadWrite.All",
"Directory.Read.All"
],
Which I assume means they are all there.
You are using client credentials flow which uses application permission. But GET /groups/{group-id}/planner/plans api doesn't support application permission. It needs delegated permissions. See the difference here.
I'm getting below error when i'm trying to access google pubsub via python library.
HttpError 403 when requesting
https://monitoring.googleapis.com/v3/projects/xxxx/timeSeries?filter=metric.type%3D%22pubsub.googleapis.com%2Fsubscription%2Fnum_undelivered_messages%22+AND+resource.label.subscription_id%3Dtest&interval.endTime=2018-08-28T13%3A11%3A58.256545Z&alt=json&interval.startTime=2018-08-28T13%3A10%3A58.256533Z returned "The caller does not have permission"
Below is the scenario which i'm trying:
I'm trying to fetch "number of undelivered messages" from the subscriber.
I'm using python library to access the google cloud.
I'm using to secret json file to access the google.
I created a service account called "monitoring" and gave monitoring admin, pubsub admin roles, monitoring read roles.
I have given below scopes for the authorization.
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/pubsub
https://www.googleapis.com/auth/monitoring
https://www.googleapis.com/auth/monitoring.read
By this script I can get topics list but when I try to fetch monitoring details it is throwing error.
Can somebody help me what I'm missing here?
I had the same access issues. Your account needs access to the role "Monitoring Viewer".
UPDATE: I've received notice from Microsoft that this problem is a bug in the Graph API. They're working on a solution.
I'm using the new v2.0 OAuth flow to authenticate my app for use with Microsoft Graph to make it able to list any users files, download and upload files in any users OneDrive and set permissions to files. This without the user being logged in, that is running it as a service account/daemon.
I've set up a new "Converged application" in the new Application Registration Portal. I've set all necessary scopes/application permission, including Files.ReadWrite.All. (I actually checked all possible boxes...). In the Microsoft Graph docs this should be the only scope necessary when calling the endpoints I'm interested in:
/v1.0/users/{userID}/drive
/v1.0/users/{userID}/drive/items/{ItemID}/children
/v1.0/users/{userID}/drive/items/{ItemID}/content
/v1.0/users/{userID}/drive/items/{ItemID}/invite
/v1.0/users/{userID}/drive/items/{ItemID}/createLink
Then I've followed the documentation for the Client Credentials flow, including giving Admin Consent to the app for use in my company tenant.
I'm successfully receiving an access token. After receiving the access token I've double checked at jwt.io that the token actually contains all scopes (incl. Files.ReadWrite.All).
I'm able to use this access token to get any user's drive and list any users files (the first two endpoints listed above). I've also tried to get thumbnails of any users files which works fine. But as soon as I try to download a file, add permissions to a file or create a Sharing Link (the last three endpoints listed above), I receive an 401 Unauthorized error. From this, I assume the scope Files.Read.All works fine, but the scope Files.ReadWrite.All is not working.
As to what I can understand from the Scopes documentation, the scopes I'm trying to use should work. It the "App-only permissions requiring administrator's consent" section, it describes Files.ReadWrite.All as:
Allows the app to read, create, update and delete all files in all site collections without a signed in user.
I've hit a wall. Are there limitations to the new v2.0 OAuth token and/or Microsoft Graph regarding App-Only access that I'm missing?
Closing the loop for those who stumble on this question. There was an issue with Files.ReadWrite.All in App-Only scenarios when it came to uploading or changing permissions of a file.
The issue with downloading is unrelated. Authorization errors when downloading a file stem from passing an Authorization header in the download request. The `/content/ endpoint returns a URL that can be used to download the file. This is a pre-authorized URL that exists for a short period of time. Passing an Authorization header in that request results in an error since it doesn't expect to receive such a header, nor can it determine which credentials it should use (super-oversimplification but this the general idea).