We're having trouble publishing messages to a Google Cloud PubSub topic on Google AppEngine. Using the Application Default credentials works perfect locally. But once it's deployed on Google AppEngine it gives the following error:
<HttpError 403 when requesting https://pubsub.googleapis.com/v1/projects/our-project-id/topics/our-topic:publish?alt=json returned "The request cannot be identified with a project. Please pass a valid API key with the request.">
I would assume that it's will use the service account of app engine to access the PubSub API. Here is the code we used to create the credentials.
credentials = GoogleCredentials.get_application_default()
if credentials.create_scoped_required():
credentials = credentials.create_scoped(['https://www.googleapis.com/auth/pubsub'])
http = httplib2.Http()
credentials.authorize(http)
pubsub_service = build('pubsub', 'v1', http=http)
The error is thrown when publishing the actual message to PubSub.
pubsub_service.projects().topics().publish(topic="projects/out-project-id/topics/out-topic", body = { 'messages' : [ { 'data': base64.b64encode(request.get_data()) }]}).execute()
Not that the same flow works doing API call's to "BigQuery", so it's not a general Google API problem. It seems to be specific to PubSub...
It's a rare case of the service account without project id embedded in it. We fixed your service account and you should be good to go now. Sorry for the trouble.
Related
I'm using Google App Engine to host a couple of services (a NextJS SSR service and a backend API built on Express). I've setup my dispatch.yaml file to route /api/* requests to my API service and all other requests get routed to the default (NextJS) service.
dispatch:
- url: '*/api/*'
service: api
The problem: I've also turned on Identity-Aware Proxy for App Engine. When I try to make a GET request from my NextJS service to my API (server-side, via getServerSideProps) it triggers the IAP sign-in page again instead of hitting my API. I've tried out a few ideas to resolve this:
Forwarding all cookies in the API request
Setting the X-Requested-With header as mentioned here
Giving IAP-secured Web App User permissions to my App Engine default service account
But nothing seems to work. I've confirmed that turning off IAP for App Engine allows everything to function as expected. Any requests to the API from the frontend also work as expected. Is there a solution I'm missing or a workaround for this?
You need to perform a service to service call. That's no so simple and you have not really example for that. Anyway I tested (in Go) and it worked.
Firstly, based your development on the Cloud Run Service to Service documentation page.
You will have this piece of code in NodeJS sorry, I'm not a NodeJS developer and far least a NexJS developer, you will have to adapt
// Make sure to `npm install --save request-promise` or add the dependency to your package.json
const request = require('request-promise');
const receivingServiceURL = ...
// Set up metadata server request
// See https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature
const metadataServerTokenURL = 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=';
const tokenRequestOptions = {
uri: metadataServerTokenURL + receivingServiceURL,
headers: {
'Metadata-Flavor': 'Google'
}
};
// Fetch the token, then provide the token in the request to the receiving service
request(tokenRequestOptions)
.then((token) => {
return request(receivingServiceURL).auth(null, null, true, token)
})
.then((response) => {
res.status(200).send(response);
})
.catch((error) => {
res.status(400).send(error);
});
This example won't work because you need the correct audience. Here, the variable is receivingServiceURL. It's correct for Cloud Run (and Cloud Functions) but not for App Engine behind IAP. You need to use the Client ID of the OAuth2 credential named IAP-App-Engine-app
Ok, hard to understand what I'm talking about. So, go to the console, API & Services -> Creentials. From there, you have a OAuth2 Client ID section. copy the Client ID column of the line IAP-App-Engine-app, like that
Final point, be sure that your App Engine default service account has the authorization to access to IAP. And add it as IAP-secured Web App User. The service account has this format <PROJECT_ID>#appspot.gserviceaccount.com
Not really clear also. So, go to the IAP page (Security -> Identity Aware Proxy), click on the check box in front of App Engine and go the right side of the page, in the permission panel
In the same time, I can explain how to deactivate IAP on a specific service (as proposed by NoCommandLine). Just a remark: deactivate security when you have trouble with it is never a good idea!!
Technically, you can't deactive IAP on a service. But you can grant allUsers as IAP-secured Web App User on a specific service (instead of clicking on the checkbox of App Engine, click on the checkbox of a specific service). And like that, even with IAP you authorized all users to access to your service. it's an activation without checks in fact.
Below are my requirements.
Develop a flask app.
Use collections in the firebase in the app.
Deploy this app on Google App Engine using a standard service account
What I have done.
Created a service account
Downloaded the corresponding credentials json; I am calling it as key.json
written a main.py
cred = credentials.Certificate('key.json')
default_app = initialize_app(cred)
db = firestore.client()
user_ref = db.collection_group('Users')
#app.route('/', methods=['GET'])
def home():
return "<h1>Welcome to my first app</h1>"
#app.route('/users', methods=['GET'])
def getUsers():
try:
result = [user.to_dict() for user in user_ref .stream()]
return jsonify(result), 200
except Exception as e:
result = { "message:"failed"}
return jsonify(result), 500
I have tested this locally and also on deployed on Google App Engine.
In both the cases, key.json was in the same directory as the code.
I have verified that if this key.json is modified to store wrong data, then /users endpoint won't work and gives me a 500 error.
So far so good. I want to know if this is even the right approach.
I want the key.json authentication to applied even for the root / endpoint.
i.e., if the user supplies a valid key.json, only then the Welcome to my first app should be displayed.
Else, Unauthorized user message needs to be displayed.
As mentioned by #Gaefan and #DishantMakwana, as well as in this documentation:
An API key only identifies the application and doesn't require user authentication. It is sufficient for accessing public data.
So in order to authenticate/authorize your users you should reconsider your strategy. I would recommend you to follow the instructions in the Authenticating as an end user Documentation.
I have found that we can use Google Cloud Endpoints for API management. Works as a charm.
I'm trying to setup a google group for marketing purposes, in which when certain users sign up to my application, I send their email to this google group with the following code
# google_admin_apis.py
def add_member(member):
if not member.email:
return False
try:
service = build('admin', 'directory_v1')
except DefaultCredentialsError: # For developers
return False
group_key = 'mygroup#mydomain.com'
body = {
"email": member.email
}
members = service.members()
request = members.insert(groupKey=group_key, body=body)
response = request.execute()
return True
My application is hosted on Google App Engine, so by default ADC will use the default service account when run on the server. I have tried to run this code locally by using gcloud auth application-default-account login and logging in with my G Suite admin account, and also my personal account (both are owners of the GCP project). After this failed, I did some research and realised that to enable OAuth2 to access my G Suite User data (I'm not really accessing anything by inserting a user?!?) I had to 'enable domain wide delegation' on the default service account, so I did this, I then downloaded the service account JSON and attempted to manually authorise with $GOOGLE_APPLICATION_CREDENTIALS, but was still getting a 403. I then went one step further and followed these instructions. Giving my Client ID access to https://www.googleapis.com/auth/admin.directory.group and group.member.
After all this, I still get a 403 error.
With the application-default-credentials I get:
<HttpError 403 when requesting
https://www.googleapis.com/admin/directory/v1/groups/groupKey/members?alt=json
returned "Insufficient Permission">
When using the app engine default service account through .json with either activate-service-account or through the GOOGLE_APPLICATION_CREDENTIALS, I get:
<HttpError 403 when requesting
https://www.googleapis.com/admin/directory/v1/groups/groupKey/members?alt=json
returned "Not Authorized to access this resource/api">
(groupKey intentially censored)
In short, I have an app-engine default service account with domain wide delegation and have given it's client ID access to both roles required for the Directory API's member.insert() function, yet I am still not allowed to call the API as above.
Any help would be greatly appreciated.
I followed this tutorial https://developers.google.com/admin-sdk/directory/v1/quickstart/python to run a similar function locally using Google's google_auth_oauthlib to set up OAuth2 credentials
service = build('admin', 'directory_v1', credentials=creds)
I am trying to build a Gmail service which will read a user's emails, once their IT admin has authenticated the App on the Apps marketplace. From the documentation, it seemed service accounts would be the right fit, for which I tried both:
scope = "https://www.googleapis.com/auth/gmail.readonly"
project_number = "c****io"
authorization_token, _ = app_identity.get_access_token(scope)
logging.info("Using token %s to represent identity %s",
authorization_token, app_identity.get_service_account_name())
#authorization_token = "OAuth code pasted from playground"
response = urlfetch.fetch(
"https://www.googleapis.com/gmail/v1/users/me/messages",
method=urlfetch.GET,
headers = {"Content-Type": "application/json",
"Authorization": "OAuth " + authorization_token})
and
credentials = AppAssertionCredentials(scope=scope)
http = credentials.authorize(httplib2.Http(memcache))
service = build(serviceName='gmail', version='v1', http=http)
listReply = gmail_service.users().messages().list(userId='me', q = '').execute()
I then started dev_appserver.py as per Unable to access BigQuery from local App Engine development server
However, I get an HTTP error code 500 "Backend Error". Same code, but when I paste the access_token from the OAuth playground, it works fine (HTTP 200). I'm on my local machine in case that makes any difference. Wondering if I'm missing anything? I'm trying to find all emails for all users of a particular domain where their IT admin has installed my Google Marketplace App.
Thanks for the help!
To do this type of impersonation, you should create a JWT and set the "sub" field to the email address of the user whose mailbox you want to access. Developer documentation: Using OAuth 2.0 for Server to Server Applications: Additional claims.
The python code to construct the credentials will look something like
credentials = SignedJwtAssertionCredentials(
"<service account email>#developer.gserviceaccount.com",
file("secret-privatekey.pem", "rb").read(),
scope=["https://www.googleapis.com/auth/gmail.readonly"],
sub="<user to impersonate>#your-domain.com"
)
I am using Google App Engine Identity to access Google Drive API of a Google Apps user.
I have inserted GAE URL in Manage API client access in Google Apps, with the right scope.
AppIdentityCredential credential = new AppIdentityCredential(Arrays.asList(DriveScopes.DRIVE_FILE));
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
Drive service = new Drive.Builder(httpTransport, jsonFactory, credential).build();
I get this error:
com.google.appengine.repackaged.org.apache.http.impl.client.DefaultRequestDirector handleResponse
WARNING: Authentication error: Unable to respond to any of these challenges: {authsub=WWW-Authenticate: AuthSub realm="https://www.google.com/accounts/AuthSubRequest" allowed-scopes="https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/docs,https://www.googleapis.com/auth/drive.file,https://www.googleapis.com/auth/drive.readonly,https://www.googleapis.com/auth/drive.metadata.readonly"}
An error occurred: com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 OK
{
"code" : 401,
"errors" : [ {
"domain" : "global",
"location" : "Authorization",
"locationType" : "header",
"message" : "Invalid Credentials",
"reason" : "authError"
} ],
"message" : "Invalid Credentials"
}
Is it necessary to enable GAE application in API Client access in Google Apps? If not, how I authorise GAE app to gain acess to an API Scope?
Is possible to test GAE Identity in Local / Eclipse?
Summarizing, what is wrong or lack?
You can not debug AppEngine Credentials locally. The only way to debug is to use different credentials (AppEngine for production and Key-based service account locally). It will require to write different code for local and production environment.
You need to add your appengine service accounts to Team. If you use custom Google Apps domain for your project than you will need to create email group in your custom domain and add this group. All service accounts should be added to this group.
Consider adding the scopes it tries to authenticate against.
I had the same error in my code and ended up solving in by adding the drive.file scope.
Two important steps
Get the Application Default Credentials Service account JSON file.
Set GOOGLE_APPLICATION_CREDENTIALS environment variable with the path of the JSON file. Example: GOOGLE_APPLICATION_CREDENTIALS=c:/myFolder/my.jsonfile
FYI:
https://developers.google.com/identity/protocols/application-default-credentials