I am hitting the REST API for cloud tasks queues via roughly:
import { cloudtasks_v2 } from "googleapis/build/src/apis/cloudtasks/v2";
const cloudTasksClient = new cloudtasks_v2.Cloudtasks({ auth: authClient });
cloudTasksClient.projects.locations.list({
name: `projects/${projectId}`
});
and I am getting this error:
App Engine targets require "appengine.applications.get" IAM permission (https://cloud.google.com/appengine/docs/admin-api/access-control) and the API_CLOUD_PLATFORM scope (https://www.googleapis.com/auth/cloud-platform).
the REST API does not state that the permission is needed for that call
https://cloud.google.com/tasks/docs/reference/rest/v2/projects.locations.queues/list
Is there any way to only get the queues that don't require app engine permissions, or to circumvent this error?
I do not want to give permissions to app engine
I tried on a project with app engine tasks and it works, but on a project with app engine tasks it throws an error and I can't determine a way to work around that.
Related
I'm using Google App Engine to host a couple of services (a NextJS SSR service and a backend API built on Express). I've setup my dispatch.yaml file to route /api/* requests to my API service and all other requests get routed to the default (NextJS) service.
dispatch:
- url: '*/api/*'
service: api
The problem: I've also turned on Identity-Aware Proxy for App Engine. When I try to make a GET request from my NextJS service to my API (server-side, via getServerSideProps) it triggers the IAP sign-in page again instead of hitting my API. I've tried out a few ideas to resolve this:
Forwarding all cookies in the API request
Setting the X-Requested-With header as mentioned here
Giving IAP-secured Web App User permissions to my App Engine default service account
But nothing seems to work. I've confirmed that turning off IAP for App Engine allows everything to function as expected. Any requests to the API from the frontend also work as expected. Is there a solution I'm missing or a workaround for this?
You need to perform a service to service call. That's no so simple and you have not really example for that. Anyway I tested (in Go) and it worked.
Firstly, based your development on the Cloud Run Service to Service documentation page.
You will have this piece of code in NodeJS sorry, I'm not a NodeJS developer and far least a NexJS developer, you will have to adapt
// Make sure to `npm install --save request-promise` or add the dependency to your package.json
const request = require('request-promise');
const receivingServiceURL = ...
// Set up metadata server request
// See https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature
const metadataServerTokenURL = 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=';
const tokenRequestOptions = {
uri: metadataServerTokenURL + receivingServiceURL,
headers: {
'Metadata-Flavor': 'Google'
}
};
// Fetch the token, then provide the token in the request to the receiving service
request(tokenRequestOptions)
.then((token) => {
return request(receivingServiceURL).auth(null, null, true, token)
})
.then((response) => {
res.status(200).send(response);
})
.catch((error) => {
res.status(400).send(error);
});
This example won't work because you need the correct audience. Here, the variable is receivingServiceURL. It's correct for Cloud Run (and Cloud Functions) but not for App Engine behind IAP. You need to use the Client ID of the OAuth2 credential named IAP-App-Engine-app
Ok, hard to understand what I'm talking about. So, go to the console, API & Services -> Creentials. From there, you have a OAuth2 Client ID section. copy the Client ID column of the line IAP-App-Engine-app, like that
Final point, be sure that your App Engine default service account has the authorization to access to IAP. And add it as IAP-secured Web App User. The service account has this format <PROJECT_ID>#appspot.gserviceaccount.com
Not really clear also. So, go to the IAP page (Security -> Identity Aware Proxy), click on the check box in front of App Engine and go the right side of the page, in the permission panel
In the same time, I can explain how to deactivate IAP on a specific service (as proposed by NoCommandLine). Just a remark: deactivate security when you have trouble with it is never a good idea!!
Technically, you can't deactive IAP on a service. But you can grant allUsers as IAP-secured Web App User on a specific service (instead of clicking on the checkbox of App Engine, click on the checkbox of a specific service). And like that, even with IAP you authorized all users to access to your service. it's an activation without checks in fact.
I am following the documentation here https://cloud.google.com/appengine/docs/managing-costs to disable App Engine using cloud functions when a spending limit is reached.
I want to disable App Engine for all projects associated with the same billing account, which the docs say should work in the following note
Note: The source code assumes that the function you are creating and the app you want to disable are in the same Google Cloud project. If the function and the app are in separate projects, change the source code so APP_NAME identifies the project that contains the app you want to disable.
But I get this error
Error: function terminated. Recommended action: inspect logs for
termination reason. Details: <HttpError 403 when requesting
https://appengine.googleapis.com/v1/apps/other-project?alt=json
returned "The caller does not have permission">
I also verified that I am using the App Engine default service account with Admin role per the docs
Select a service account that has the App Engine Admin role. The App Engine default service account has this role by default.
I modified the example for testing which is basically this
import base64
import json
import os
from googleapiclient import discovery
APP_NAME = os.getenv('GCP_PROJECT')
def limit_use_appengine(data, context):
pubsub_data = base64.b64decode(data['data']).decode('utf-8')
pubsub_json = json.loads(pubsub_data)
cost_amount = pubsub_json['costAmount']
budget_amount = pubsub_json['budgetAmount']
if cost_amount <= budget_amount:
print(f'No action necessary. (Current cost: {cost_amount})')
appengine = discovery.build(
'appengine',
'v1',
cache_discovery=False
)
apps = appengine.apps()
# for testing
display_status(apps, APP_NAME) # works fine
display_status(apps, "billing-account-project-name") # also works fine
display_status(apps, "other-project") # permission error
return
appengine = discovery.build(
'appengine',
'v1',
cache_discovery=False
)
apps = appengine.apps()
check_app(apps, "APP_NAME")
check_app(apps, "other-project")
def check_app(apps, appName):
# Get the target app's serving status
target_app = apps.get(appsId=appName).execute()
current_status = target_app['servingStatus']
# Disable target app, if necessary
if current_status == 'SERVING':
print(f'Attempting to disable app {appName}...')
body = {'servingStatus': 'USER_DISABLED'}
apps.patch(appsId=appName, updateMask='serving_status', body=body).execute()
def display_status(apps, appName):
target_app = apps.get(appsId=appName).execute()
current_status = target_app['servingStatus']
print(f'Serving status for {appName} is {current_status}')
What am I missing?
Service Accounts either inherit permissions or are assigned permission to the project. A billing account does not affect a service account's permissions.
Solution: assign permission in the project for the service account where you are getting the permissions error. If you are using Organizations, assign the permission higher up so that the service account inherits permissions to projects.
I currently have App Engine up and running, protected by IAP, and my eventual aim is to have this be triggered by an Apps Script project. I've tested the code without IAP and it works fine. However, I'm running into difficulties successfully authorizing access to it when IAP is enabled.
I've added myself as an IAP-secured Web App User (as well as Policy Admin) to the App Engine, but whenever I try triggering it from a GSheets Apps Script where I'm the owner and it's associated with the correct GCP project (using this great explanation as a guide) I get the following:
"Invalid IAP credentials: JWT audience doesn't match this application ('aud' claim (1084708005908-bk66leo0dnkrjsh276f0rgeoq8ns87qu.apps.googleusercontent.com) doesn't match expected value (1084708005908-oqkn6pcj03c2pmdufkh0l7mh37f79po2.apps.googleusercontent.com))"
I've tried adding/removing various permissions to my account, as well creating a new Apps Script and re-adding to the project, but to no avail. I run into the same issue when triggering from CLI, so I'm fairly sure it's an issue with authentication, however this is my Apps Script code in case it helps:
function test() {
const options = {
headers: {'Authorization': 'Bearer ' + ScriptApp.getIdentityToken()},
muteHttpExceptions: true
}
var result = UrlFetchApp.fetch('https://APP-ENGINE-URL.appspot.com', options);
Logger.log(result);
}
And the manifest file:
{
"timeZone": "Europe/London",
"dependencies": {
},
"exceptionLogging": "STACKDRIVER",
"runtimeVersion": "V8",
"oauthScopes": ["openid", "https://www.googleapis.com/auth/script.external_request"]
}
Any help on this is super appreciated! Never posted here before, but pretty desperate and couldn't find anyone with this exact problem on SO.
Consideration
The problem with your solution is that you are using the identity of an auto-generated OAuth Client for Apps Script. This clients are not suitable for this kind of authentication, here a complete list of supported OAuth clients.
Solution
In order to complete your authentication you will need an extra step. You will have to create another OAuth Client and build an identity token with its credentials.
To make things easier I would recommend to use this Apps Script library: https://github.com/gsuitedevs/apps-script-oauth2
The inital Set-up is covered in the linked documentation.
Important: When creating the OAuth Client take note of the ClientID and the Client-secret. Plus, you will need to add an Authorized Redirect URI. This is standard when using the OAuth2 GAS library and it has this form: https://script.google.com/macros/d/{Your Apps Script ID}/usercallback
Now you have all the necessary information to build your identity token. In the Github repository there is a boilerplate sample that will cover the first coding steps with the OAuth2 GAS library.
Here is the link.
Copy this code to your Apps Script project and follow the instructions in the comments. You will need to add an extra OAuth scope: "https://www.googleapis.com/auth/userinfo.email".
Once you set all the constants with your OAuth clients information you should run the run() function from your Apps Script editor. This will log a URL you have to open in your browser to authorize your App. Once you authorized the App run again the run() function and you will successfully access your IAP protected application.
References
OAuth2 GAS library
IAP programmatic authentication
I am trying to work with the Google Cloud Tasks API
In python2.7 app engine standard you had this amazing library (deferred) that allowed you to easily assign workers to multiple tasks that could be completed asynchronisly.
So in a webapp2 handler I could do this:
create_csv_file(data):
#do a bunch of work ...
return
MyHandler(webapp2.Handler):
def get(self)
data = myDB.query()
deferred.defer(create_csv_file, data)
Now I am working on the new Google App Engine Python 3 runtime and the deferred library is not available for GAE Py3.
Is the google cloud tasks the correct solution/replacement?
This is where I am at now... I've scoured the internet looking for answer but my Google powers have failed me. I've found come examples but they are not very good and they appear as though you should be creating creating /adding tasks from gcloud console or locally but no examples of adding tasks from a front end api endpoint.
ExportCSVFileHandler(Resource):
def get(self):
create_task()
return
CSVTaskHandler(Resource):
def(post):
#do a lot of work creating a csv file
return
create_task():
client = tasks.CloudTasksClient(credentials='mycreds')
project = 'my-project_id'
location = 'us-east4'
queue_name = 'csv-worker'
parent = client.location_path(project, location)
the_queue = {
'name': client.queue_path(project, location, queue_name),
'rate_limits': {
'max_dispatches_per_second': 1
},
'app_engine_routing_override': {
'version': 'v2',
'service': 'task-module'
}
}
queues = [the_queue]
task = {
'app_engine_http_request': {
'http_method': 'GET',
'relative_uri': '/create-csv',
'app_engine_routing': {
'service': 'worker'
},
'body': str(20).encode()
}
}
# Use the client to build and send the task.
response = client.create_task(parent, task)
print('Created task {}'.format(response.name))
# [END taskqueues_using_yaml]
return response
Yes, Cloud Tasks is the replacement for App Engine Taskqueues. The API can be called from anywhere, ie locally, from App Engine, from external services, and even gcloud. The samples show you how to do this locally, but you can easily replace your old taskqueue code with the new Cloud Tasks library.
Unfortunately, there is no deferred library for Cloud Tasks. There are multiple ways around this. Create separate endpoints for task handlers and use the App Engine routing to send the task to the right endpoint, or add metadata to the task body in order for your handler to appropriate process the task request.
We're having trouble publishing messages to a Google Cloud PubSub topic on Google AppEngine. Using the Application Default credentials works perfect locally. But once it's deployed on Google AppEngine it gives the following error:
<HttpError 403 when requesting https://pubsub.googleapis.com/v1/projects/our-project-id/topics/our-topic:publish?alt=json returned "The request cannot be identified with a project. Please pass a valid API key with the request.">
I would assume that it's will use the service account of app engine to access the PubSub API. Here is the code we used to create the credentials.
credentials = GoogleCredentials.get_application_default()
if credentials.create_scoped_required():
credentials = credentials.create_scoped(['https://www.googleapis.com/auth/pubsub'])
http = httplib2.Http()
credentials.authorize(http)
pubsub_service = build('pubsub', 'v1', http=http)
The error is thrown when publishing the actual message to PubSub.
pubsub_service.projects().topics().publish(topic="projects/out-project-id/topics/out-topic", body = { 'messages' : [ { 'data': base64.b64encode(request.get_data()) }]}).execute()
Not that the same flow works doing API call's to "BigQuery", so it's not a general Google API problem. It seems to be specific to PubSub...
It's a rare case of the service account without project id embedded in it. We fixed your service account and you should be good to go now. Sorry for the trouble.