I'm trying to access a pull queue from google compute with the compute OAuth token using python
from oauth2client import gce
from apiclient.discovery import build
import httplib2
credentials = gce.AppAssertionCredentials('')
http = httplib2.Http()
http=credentials.authorize(http)
credentials.refresh(http)
service = build('taskqueue', 'v1beta2', http=http)
tq=service.taskqueues()
tq.get(project=MY_APPENGINE_PROJECT, taskqueue=PULL_QUEUE_NAME, getStats=True).execute()
I keep getting HttpError 403 "you are not allowed to make this api call"
please help, what configure have I missing?
thanks,
Shay
UPDATE: Thanks to #Shay for asking this question, the issue he encountered is no longer an issue, as we have allowed aliases to work (when relevant) in the Task Queue API.
For posterity here is the original answer below:
Two of the most common mistakes I have seen are:
Forgetting to include the s~ in your App Engine Project. For example, if your application ID is my-awesome-app, then you are calling
tq.get(project='my-awesome-app', taskqueue=PULL_QUEUE_NAME...
when you should be calling
tq.get(project='s~my-awesome-app', taskqueue=PULL_QUEUE_NAME...
Forgetting to add the Compute service account to the task queue ACL in queue.yaml. To do this, you need to get the service account associated with your project and add it to the acl:
queue:
- name: pull-queue
mode: pull
acl:
- writer_email: 123845678986#project.gserviceaccount.com # can do all
and of course this would mean PULL_QUEUE_NAME = 'pull-queue' here. Also note, 123845678986#project.gserviceaccount.com should be replaced with the service account for your Compute Engine instance.
Related
I am trying to invoke a Cloud Run service using Cloud Tasks as described in the docs here.
I have a running Cloud Run service. If I make the service publicly accessible, it behaves as expected.
I have created a cloud queue and I schedule the cloud task with a local script. This one is using my own account. The script looks like this
from google.cloud import tasks_v2
client = tasks_v2.CloudTasksClient()
project = 'my-project'
queue = 'my-queue'
location = 'europe-west1'
url = 'https://url_to_my_service'
parent = client.queue_path(project, location, queue)
task = {
'http_request': {
'http_method': 'GET',
'url': url,
'oidc_token': {
'service_account_email': 'my-service-account#my-project.iam.gserviceaccount.com'
}
}
}
response = client.create_task(parent, task)
print('Created task {}'.format(response.name))
I see the task appear in the queue, but it fails and retries immediately. The reason for this (by checking the logs) is that the Cloud Run service returns a 401 response.
My own user has the roles "Service Account Token Creator" and "Service Account User". It doesn't have the "Cloud Tasks Enqueuer" explicitly, but since I am able to create the task in the queue, I guess I have inherited the required permissions.
The service account "my-service-account#my-project.iam.gserviceaccount.com" (which I use in the task to get the OIDC token) has - amongst others - the following roles:
Cloud Tasks Enqueuer (Although I don't think it needs this one as I'm creating the task with my own account)
Cloud Tasks Task Runner
Cloud Tasks Viewer
Service Account Token Creator (I'm not sure whether this should be added to my own account - the one who schedules the task - or to the service account that should perform the call to Cloud Run)
Service Account User (same here)
Cloud Run Invoker
So I did a dirty trick: I created a key file for the service account, downloaded it locally and impersonated locally by adding an account to my gcloud config with the key file. Next, I run
curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://url_to_my_service
That works! (By the way, it also works when I switch back to my own account)
Final tests: if I remove the oidc_token from the task when creating the task, I get a 403 response from Cloud Run! Not a 401...
If I remove the "Cloud Run Invoker" role from the service account and try again locally with curl, I also get a 403 instead of a 401.
If I finally make the Cloud Run service publicly accessible, everything works.
So, it seems that the Cloud Task fails to generate a token for the service account to authenticate properly at the Cloud Run service.
What am I missing?
I had the same issue here was my fix:
Diagnosis: Generating OIDC tokens currently does not support custom domains in the audience parameter. I was using a custom domain for my cloud run service (https://my-service.my-domain.com) instead of the cloud run generated url (found in the cloud run service dashboard) that looks like this: https://XXXXXX.run.app
Masking behavior: In the task being enqueued to Cloud Tasks, If the audience field for the oidc_token is not explicitly set then the target url from the task is used to set the audience in the request for the OIDC token.
In my case this meant that enqueueing a task to be sent to the target https://my-service.my-domain.com/resource the audience for the generating the OIDC token was set to my custom domain https://my-service.my-domain.com/resource. Since custom domains are not supported when generating OIDC tokens, I was receiving 401 not authorized responses from the target service.
My fix: Explicitly populate the audience with the Cloud Run generated URL, so that a valid token is issued. In my client I was able to globally set the audience for all tasks targeting a given service with the base url: 'audience' : 'https://XXXXXX.run.app'. This generated a valid token. I did not need to change the url of the target resource itself. The resource stayed the same: 'url' : 'https://my-service.my-domain.com/resource'
More Reading:
I've run into this problem before when setting up service-to-service authentication: Google Cloud Run Authentication Service-to-Service
1.I created a private cloud run service using this code:
import os
from flask import Flask
from flask import request
app = Flask(__name__)
#app.route('/index', methods=['GET', 'POST'])
def hello_world():
target = os.environ.get('TARGET', 'World')
print(target)
return str(request.data)
if __name__ == "__main__":
app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))
2.I created a service account with --role=roles/run.invoker that I will associate with the cloud task
gcloud iam service-accounts create SERVICE-ACCOUNT_NAME \
--display-name "DISPLAYED-SERVICE-ACCOUNT_NAME"
gcloud iam service-accounts list
gcloud run services add-iam-policy-binding SERVICE \
--member=serviceAccount:SERVICE-ACCOUNT_NAME#PROJECT-ID.iam.gserviceaccount.com \
--role=roles/run.invoker
3.I created a queue
gcloud tasks queues create my-queue
4.I create a test.py
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import datetime
# Create a client.
client = tasks_v2.CloudTasksClient()
# TODO(developer): Uncomment these lines and replace with your values.
project = 'your-project'
queue = 'your-queue'
location = 'europe-west2' # app engine locations
url = 'https://helloworld/index'
payload = 'Hello from the Cloud Task'
# Construct the fully qualified queue name.
parent = client.queue_path(project, location, queue)
# Construct the request body.
task = {
'http_request': { # Specify the type of request.
'http_method': 'POST',
'url': url, # The full url path that the task will be sent to.
'oidc_token': {
'service_account_email': "your-service-account"
},
'headers' : {
'Content-Type': 'application/json',
}
}
}
# Convert "seconds from now" into an rfc3339 datetime string.
d = datetime.datetime.utcnow() + datetime.timedelta(seconds=60)
# Create Timestamp protobuf.
timestamp = timestamp_pb2.Timestamp()
timestamp.FromDatetime(d)
# Add the timestamp to the tasks.
task['schedule_time'] = timestamp
task['name'] = 'projects/your-project/locations/app-engine-loacation/queues/your-queue/tasks/your-task'
converted_payload = payload.encode()
# Add the payload to the request.
task['http_request']['body'] = converted_payload
# Use the client to build and send the task.
response = client.create_task(parent, task)
print('Created task {}'.format(response.name))
#return response
5.I run the code in Google Cloud Shell with my user account which has Owner role.
6.The response received has the form:
Created task projects/your-project/locations/app-engine-loacation/queues/your-queue/tasks/your-task
7.Check the logs, success
The next day I am no longer able to reproduce this issue. I can reproduce the 403 responses by removing the Cloud Run Invoker role, but I no longer get 401 responses with exactly the same code as yesterday.
I guess this was a temporary issue on Google's side?
Also, I noticed that it takes some time before updated policies are actually in place (1 to 2 minutes).
For those like me, struggling through documentation and stackoverflow when having continuous UNAUTHORIZED responses on Cloud Tasks HTTP requests:
As was written in thread, you better provide audience for oidcToken you send to CloudTasks. Ensure your requested url exactly equals to your resource.
For instance, if you have Cloud Function named my-awesome-cloud-function and your task request url is https://REGION-PROJECT-ID.cloudfunctions.net/my-awesome-cloud-function/api/v1/hello, you need to ensure, that you set function url itself.
{
serviceAccountEmail: SERVICE-ACCOUNT_NAME#PROJECT-ID.iam.gserviceaccount.com,
audience: https://REGION-PROJECT-ID.cloudfunctions.net/my-awesome-cloud-function
}
Otherwise seems full url is used and leads to an error.
I have a private HTTP Google Cloud Function which I'd like to call from an AppEngine app in another project.
Ideally, the AppEngine Service Account would have roles/cloudfunctions.invoker on my Cloud Function, I'd turn off all other invokers, and I wouldn't have to worry about auth at all inside of the CF. I'm struggling to get the AppEngine identity passed along.
Google's docs show how to do this from one Cloud Function to another, but AppEngine instead uses its own identity library to simplify getting access tokens. AppEngine docs outline:
Identity for other AppEngine apps in the same project
Identity for Google APIs
Something seemingly unrelated: verifying a payload's signature
Any way to include the AppEngine identity such that Google's native Cloud Function invoker role will the request through?
For this situation you will need to do the authentication programmatically by yourself.
First you need to add the app engine service account to the Cloud Functions permission.
After that, you need to follow the steps for this situation. Basically you will need to create a JWT, to authorize it and then to include the JWT in your request.
Here you can find a code example for creating and authorising a JWT.
I have reproduced your situation in python. I used the code from the link I have sent to you, and then after I had my JWT alright, I made a request like this :
#app.route('/')
def index():
data = {'headers': request.headers,
'service_name': os.environ.get('GAE_SERVICE', '(running locally)'),
'environment': os.environ}
return render_template('index.html', data=data)
#app.route('/request')
def send_request():
import requests
receiving_function_url = 'YOUR-CLOUD-FUNCT-URL'
r=requests.get("http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token?audience="+receiving_function_url,
headers={'Metadata-Flavor': 'Google'})
response = make_iap_request('YOUR-CLOUD-FUNCTION-URL', 'YOUR-CLOUD-FUNCTION-URL')
print(response)
return response
if __name__ == '__main__':
app.run('127.0.0.1', port=8080, debug=True)
The dependencies you need, in requirements.txt:
flask
PyJWT==1.7.1
cryptography==2.7
google-auth==1.6.3
gunicorn==19.9.0
requests==2.22.0
requests_toolbelt==0.9.1
In this repository you can find more code examples on how to do IAP(Identity Aware Proxy) requests.
I'm trying to wrap my head around Google App Engine and more specifically at the Tasks.
My question is about security, if I define a queue like :
- url: /queues/long-task
script: urlhandlers.QueueLongTask.app
login: admin
Will I be sure that the /queues/long-task can only be accessed by admin AND task system ? I was not able to find a reference about this in the Google documentation.
Thank you in advance
You are correct, login: admin takes care of it.
Here you can find more info on the documentation:
https://cloud.google.com/appengine/docs/python/taskqueue/overview-push#Python_Securing_URLs_for_tasks
You can also use the headers like X-AppEngine-QueueName if you want to do specific things only when this is called from a task:
"These headers are set internally by Google App Engine. If your request handler finds any of these headers, it can trust that the request is a Task Queue request. If any of the above headers are present in an external user request to your app, they are stripped."
I have an appengine app that needs to access a single, hard-coded spreadsheet on Google Drive.
Up until now I have been achieving this as follows:
SpreadsheetService service = new SpreadsheetService("myapp");
service.setUserCredentials("myusername#gmail.com", "myhardcodedpassword");
When I tried this today with a new user, I got InvalidCredentialsException even though the username and password were definitely correct. I got an email in my inbox saying suspicions sign-ins had been prevented, and there seems to be no way to enable them again.
I am also aware that hardcoding passwords in source is bad practice.
However, I have read very widely online for how to enable OAuth/OAuth2 for this, and have ended up wasting hours and hours piecing fragments of information from blogs, stackoverflow answers etc, to no avail.
Ideally the solution would involve an initial process to generate a long-lived access token, which could then be hard-coded in to the app.
I want a definitive list of steps for how to achieve this?
EDIT: As Google have redesigned the API Console, the details of the steps below have changed - see comments
OK here goes, step by step
Go to Google Cloud Console and register your project (aka application)
You need to note the Client ID, and Client Secret
Go to the OAuth Playground, click the gear icon and choose and enter your own credentials
You will be reminded that you need to go back to the Cloud COnsole and add the Oauth Playground as a valid callback url. So do that.
Do Step 1, choosing the spreadsheet scope and click authorize
Choose your Google account if prompted and grant auth when prompted
Do Step 2, Click 'Exchange auth code for tokens'
You will see an input box containing a refresh token
The refresh token is the equivalent of your long lived username/password, so this is what you'll hard code (or store someplace secure your app can retrieve it).
When you need to access Google Spreadsheets, you will call
POST https://accounts.google.com/o/oauth2/token
content-type: application/x-www-form-urlencoded
client_secret=************&grant_type=refresh_token&refresh_token=1%2xxxxxxxxxx&client_id=999999999999.apps.googleusercontent.com
which will return you an access token
{
"access_token": "ya29.yyyyyyyyyyyyyyyyyy",
"token_type": "Bearer",
"expires_in": 3600
}
Put the access token into an http header for whenever you access the spreadsheet API
Authorization: Bearer ya29.yyyyyyyyyyyyyyyyy
And you're done
Pinoyyid has indeed provided wonderful help. I wanted to follow up with Python code that will allow access to a Personal (web-accessible) Google Drive.
This runs fine within the Google App Engine (as part of a web app) and also standalone on the desktop (assuming the Google App Engine SDK is installed on your machine [available from: https://developers.google.com/appengine/downloads]).
In my case I added https://www.googleapis.com/auth/drive scope during Pinyyid's process because I wanted access to all my Google Drive files.
After following Pinoyyid's instructions to get your refresh token etc., this little Python script will get a listing of all the files on your Google drive:
import httplib2
import datetime
from oauth2client.client import OAuth2Credentials
from apiclient.discovery import build
API_KEY = 'AIz...' # from "Key for Server Applications" from "Public API Access" section of Google Developers Console
access_token = "ya29..." # from Piinoyyid's instructions
refresh_token = "1/V..." # from Piinoyyid's instructions
client_id = '654....apps.googleusercontent.com' # from "Client ID for web application" from "OAuth" section of Google Developers Console
client_secret = '6Cl...' # from "Client ID for web application" from "OAuth" section of Google Developers Console
token_expiry = datetime.datetime.utcnow() - datetime.timedelta(days=1)
token_uri = 'https://accounts.google.com/o/oauth2/token'
user_agent = 'python urllib (I reckon)'
def main():
service = createDrive()
dirlist = service.files().list(maxResults=30)
print 'dirlist', dirlist
result = dirlist.execute()
print 'result', result
def createDrive():
credentials = OAuth2Credentials(access_token, client_id, client_secret, refresh_token, token_expiry, token_uri, user_agent)
http = httplib2.Http()
http = credentials.authorize(http)
return build('drive', 'v2', http=http, developerKey=API_KEY)
main()
I'm grateful to all who have provided the steps along the way to solving this.
I am implementing Cloud Endpoints with a Python app that uses custom authentication (GAE Sessions) instead of Google Accounts. I need to authenticate the requests coming from the Javascript client, so I would like to have access to the cookie information.
Reading this other question leads me to believe that it is possible, but perhaps not documented. I'm not familiar with the Java side of App Engine, so I'm not quite sure how to translate that snippet into Python. Here is an example of one of my methods:
class EndpointsAPI(remote.Service):
#endpoints.method(Query_In, Donations_Out, path='get/donations',
http_method='GET', name='get.donations')
def get_donations(self, req):
#Authenticate request via cookie
where Query_In and Donations_Out are both ProtoRPC messages (messages.Message). The parameter req in the function is just an instance of Query_In and I didn't find any properties related to HTTP data, however I could be wrong.
First, I would encourage you to try to use OAuth 2.0 from your client as is done in the Tic Tac Toe sample.
Cookies are sent to the server in the Cookie Header and these values are typically set in the WSGI environment with the keys 'HTTP_...' where ... corresponds to the header name:
http = {key: value for key, value in os.environ.iteritems()
if key.lower().startswith('http')}
For cookies, os.getenv('HTTP_COOKIE') will give you the header value you seek. Unfortunately, this doesn't get passed along through Google's API Infrastructure by default.
UPDATE: This has been enabled for Python applications as of version 1.8.0. To send cookies through, specify the following:
from google.appengine.ext.endpoints import api_config
AUTH_CONFIG = api_config.ApiAuth(allow_cookie_auth=True)
#endpoints.api(name='myapi', version='v1', auth=AUTH_CONFIG, ...)
class MyApi(remote.service):
...
This is a (not necessarily comprehensive list) of headers that make it through:
HTTP_AUTHORIZATION
HTTP_REFERER
HTTP_X_APPENGINE_COUNTRY
HTTP_X_APPENGINE_CITYLATLONG
HTTP_ORIGIN
HTTP_ACCEPT_CHARSET
HTTP_ORIGINALMETHOD
HTTP_X_APPENGINE_REGION
HTTP_X_ORIGIN
HTTP_X_REFERER
HTTP_X_JAVASCRIPT_USER_AGENT
HTTP_METHOD
HTTP_HOST
HTTP_CONTENT_TYPE
HTTP_CONTENT_LENGTH
HTTP_X_APPENGINE_PEER
HTTP_ACCEPT
HTTP_USER_AGENT
HTTP_X_APPENGINE_CITY
HTTP_X_CLIENTDETAILS
HTTP_ACCEPT_LANGUAGE
For the Java people who land here. You need to add the following annotation in order to use cookies in endpoints:
#Api(auth = #ApiAuth(allowCookieAuth = AnnotationBoolean.TRUE))
source
(Without that it will work on the local dev server but not on the real GAE instance.)