How can I call a Google Cloud Function from Google App Engine? - google-app-engine

I have an App Engine project.
I also have a Google Cloud Function.
And I want to call that Google Cloud Function from the App Engine project. I just can't seem to get that to work.
Yes, if I make the function full public (i.e. set the Cloud Function to 'allow all traffic' and create a rule for 'allUsers' to allow calling the function) it works. But if I limit either of the two settings, it stops working immediately and I get 403's.
The App and Function are in the same project, so I would at least assume that setting the Function to 'allow internal traffic only' should work just fine, provided that I have a rule for 'allUsers' to allow calling the function.
How does that work? How does one generally call a (non-public) Google Cloud Function from Google App Engine?

You need an auth header for the ping to the function url. It should look like:
headers = {
....
'Authorization': 'Bearer some-long-hash-token'
}
Here is how to get the token:
import requests
token_response = requests.get(
'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=' +
'https://[your zone]-[your app name].cloudfunctions.net/[your function name]',
headers={'Metadata-Flavor': 'Google'})
return token_response.content.decode("utf-8")
'Allow internal traffic only' does not work as expected. My App Engine app is in the same project as the Functions, and it does not work. I had to turn on 'Allow all traffic', and use the header method.
Example:
def get_access_token():
import requests
token_response = requests.get(
'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=' +
'https://us-central1-my_app.cloudfunctions.net/my_function',
headers={'Metadata-Flavor': 'Google'})
return token_response.content.decode("utf-8")
def test():
url_string = f"https://us-central1-my_app.cloudfunctions.net/my_function?message=it%20worked"
access_token = get_access_token()
print(
requests.get(url_string, headers={'Authorization': f"Bearer {access_token}"}
)

As mentioned in the docs, Allow internal traffic only mentions the following:
Only requests from VPC networks in the same project or VPC Service Controls perimeter are allowed. All other requests are rejected.
Please note that since App Engine Standard is a serverless product, it is not part of the VPC and then the requests made from this product are not considered "Internal" calls, actually the calls are made from the Public IPs of the instances and for this reason you get an HTTP 403 error message.
Also using a VPC Serverless Connector won't work since this more a bridge to reach resources in the VPC (like VMs or Memorystore instances) but not a Cloud Function because this is also a Serverless product and it does not have an IP in the VPC.
I think here are three options:
Using App Engine Flex:
Since App Engine Flex uses VM instances, these instances will be part of the VPC and you'll reach the Function even when setting the "Allow internal traffic only" option.
Use a VM as a proxy:
You can create a VPC Serverless Connector and assign it to the app in App Engine. Then you can create a VM and reach the function using the VM as a proxy. This is not the best option because of the costs but at the end is an option.
The last option considers that the function can use the Allow All Traffic option:
You can set some security on the Cloud Function to only allow a particular Service Account and you can use this sample code to authenticate.
EDITED:
A good sample of the code for this option was shared by #gaefan in the other answer.

#GAEfan is correct.
As an addition: I used the official Google Auth library to give me the necessary headers.
const {GoogleAuth} = require('google-auth-library');
// Instead of specifying the type of client you'd like to use (JWT, OAuth2, etc)
// this library will automatically choose the right client based on the environment.
const googleCloudFunctionURL = 'https://europe-west1-project.cloudfunctions.net/function';
(async function() {
const auth = new GoogleAuth();
let googleCloudFunctionClient = await auth.getIdTokenClient(googleCloudFunctionURL);
console.log(await googleCloudFunctionClient.getRequestHeaders(googleCloudFunctionURL));
})();

Related

Service to service requests on App Engine with IAP

I'm using Google App Engine to host a couple of services (a NextJS SSR service and a backend API built on Express). I've setup my dispatch.yaml file to route /api/* requests to my API service and all other requests get routed to the default (NextJS) service.
dispatch:
- url: '*/api/*'
service: api
The problem: I've also turned on Identity-Aware Proxy for App Engine. When I try to make a GET request from my NextJS service to my API (server-side, via getServerSideProps) it triggers the IAP sign-in page again instead of hitting my API. I've tried out a few ideas to resolve this:
Forwarding all cookies in the API request
Setting the X-Requested-With header as mentioned here
Giving IAP-secured Web App User permissions to my App Engine default service account
But nothing seems to work. I've confirmed that turning off IAP for App Engine allows everything to function as expected. Any requests to the API from the frontend also work as expected. Is there a solution I'm missing or a workaround for this?
You need to perform a service to service call. That's no so simple and you have not really example for that. Anyway I tested (in Go) and it worked.
Firstly, based your development on the Cloud Run Service to Service documentation page.
You will have this piece of code in NodeJS sorry, I'm not a NodeJS developer and far least a NexJS developer, you will have to adapt
// Make sure to `npm install --save request-promise` or add the dependency to your package.json
const request = require('request-promise');
const receivingServiceURL = ...
// Set up metadata server request
// See https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature
const metadataServerTokenURL = 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=';
const tokenRequestOptions = {
uri: metadataServerTokenURL + receivingServiceURL,
headers: {
'Metadata-Flavor': 'Google'
}
};
// Fetch the token, then provide the token in the request to the receiving service
request(tokenRequestOptions)
.then((token) => {
return request(receivingServiceURL).auth(null, null, true, token)
})
.then((response) => {
res.status(200).send(response);
})
.catch((error) => {
res.status(400).send(error);
});
This example won't work because you need the correct audience. Here, the variable is receivingServiceURL. It's correct for Cloud Run (and Cloud Functions) but not for App Engine behind IAP. You need to use the Client ID of the OAuth2 credential named IAP-App-Engine-app
Ok, hard to understand what I'm talking about. So, go to the console, API & Services -> Creentials. From there, you have a OAuth2 Client ID section. copy the Client ID column of the line IAP-App-Engine-app, like that
Final point, be sure that your App Engine default service account has the authorization to access to IAP. And add it as IAP-secured Web App User. The service account has this format <PROJECT_ID>#appspot.gserviceaccount.com
Not really clear also. So, go to the IAP page (Security -> Identity Aware Proxy), click on the check box in front of App Engine and go the right side of the page, in the permission panel
In the same time, I can explain how to deactivate IAP on a specific service (as proposed by NoCommandLine). Just a remark: deactivate security when you have trouble with it is never a good idea!!
Technically, you can't deactive IAP on a service. But you can grant allUsers as IAP-secured Web App User on a specific service (instead of clicking on the checkbox of App Engine, click on the checkbox of a specific service). And like that, even with IAP you authorized all users to access to your service. it's an activation without checks in fact.

Authentication using Google Service Account in a flask app and deploying on Google App Engine

Below are my requirements.
Develop a flask app.
Use collections in the firebase in the app.
Deploy this app on Google App Engine using a standard service account
What I have done.
Created a service account
Downloaded the corresponding credentials json; I am calling it as key.json
written a main.py
cred = credentials.Certificate('key.json')
default_app = initialize_app(cred)
db = firestore.client()
user_ref = db.collection_group('Users')
#app.route('/', methods=['GET'])
def home():
return "<h1>Welcome to my first app</h1>"
#app.route('/users', methods=['GET'])
def getUsers():
try:
result = [user.to_dict() for user in user_ref .stream()]
return jsonify(result), 200
except Exception as e:
result = { "message:"failed"}
return jsonify(result), 500
I have tested this locally and also on deployed on Google App Engine.
In both the cases, key.json was in the same directory as the code.
I have verified that if this key.json is modified to store wrong data, then /users endpoint won't work and gives me a 500 error.
So far so good. I want to know if this is even the right approach.
I want the key.json authentication to applied even for the root / endpoint.
i.e., if the user supplies a valid key.json, only then the Welcome to my first app should be displayed.
Else, Unauthorized user message needs to be displayed.
As mentioned by #Gaefan and #DishantMakwana, as well as in this documentation:
An API key only identifies the application and doesn't require user authentication. It is sufficient for accessing public data.
So in order to authenticate/authorize your users you should reconsider your strategy. I would recommend you to follow the instructions in the Authenticating as an end user Documentation.
I have found that we can use Google Cloud Endpoints for API management. Works as a charm.

Service account identity from AppEngine to Cloud Function

I have a private HTTP Google Cloud Function which I'd like to call from an AppEngine app in another project.
Ideally, the AppEngine Service Account would have roles/cloudfunctions.invoker on my Cloud Function, I'd turn off all other invokers, and I wouldn't have to worry about auth at all inside of the CF. I'm struggling to get the AppEngine identity passed along.
Google's docs show how to do this from one Cloud Function to another, but AppEngine instead uses its own identity library to simplify getting access tokens. AppEngine docs outline:
Identity for other AppEngine apps in the same project
Identity for Google APIs
Something seemingly unrelated: verifying a payload's signature
Any way to include the AppEngine identity such that Google's native Cloud Function invoker role will the request through?
For this situation you will need to do the authentication programmatically by yourself.
First you need to add the app engine service account to the Cloud Functions permission.
After that, you need to follow the steps for this situation. Basically you will need to create a JWT, to authorize it and then to include the JWT in your request.
Here you can find a code example for creating and authorising a JWT.
I have reproduced your situation in python. I used the code from the link I have sent to you, and then after I had my JWT alright, I made a request like this :
#app.route('/')
def index():
data = {'headers': request.headers,
'service_name': os.environ.get('GAE_SERVICE', '(running locally)'),
'environment': os.environ}
return render_template('index.html', data=data)
#app.route('/request')
def send_request():
import requests
receiving_function_url = 'YOUR-CLOUD-FUNCT-URL'
r=requests.get("http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token?audience="+receiving_function_url,
headers={'Metadata-Flavor': 'Google'})
response = make_iap_request('YOUR-CLOUD-FUNCTION-URL', 'YOUR-CLOUD-FUNCTION-URL')
print(response)
return response
if __name__ == '__main__':
app.run('127.0.0.1', port=8080, debug=True)
The dependencies you need, in requirements.txt:
flask
PyJWT==1.7.1
cryptography==2.7
google-auth==1.6.3
gunicorn==19.9.0
requests==2.22.0
requests_toolbelt==0.9.1
In this repository you can find more code examples on how to do IAP(Identity Aware Proxy) requests.

What service account permissions should I set for secure flexible app engine -> cloud function communication

I am having problems identifying which service account I need to give certain roles to.
I have a NodeJS app running on my flexible app engine environment.
I have a single hello-world python3.7 HTTP cloud function.
I want to do a GET request from my app engine to my cloud function.
When the allUser member is given the Cloud Function Invoker role on the hello-world cloud function everything works fine.
But now I want to secure my cloud function endpoint so that only my flexible app engine can reach it.
I remove the allUser member and as expected I get a 403 when the app engine tries to call.
Now I add the #appspot.gserviceaccount.com and #gae-api-prod.google.com.iam.gserviceaccount.com members to the hello-world cloud function and give them Cloud Function Invoker roles.
I would expect the flexible app engine to now be able to call the hello-world cloud function seeing as I gave it the Cloud Function Invoker role.
But I keep getting a 403 error.
What service account is app engine flexible using to do these calls to the cloud function API?
The are some settings to made in order to connect cloud functions wit a service account:
Enable required APIs
Enable Service account
Act as User Service Account
The default service account creates a cloud function and sometimes doesn't have all the privileges.
You can find more info Here:
https://cloud.google.com/functions/docs/securing/
John Hanley was correct,
When using GCP libraries to perform actions (like google-cloud-firestore for example) the executing function will use the underlying service account permissions to do those actions.
When doing manual HTTP requests to cloud function URLs, you will have to fetch a token from the metadata server to properly authenticate your request.
def generate_token() -> str:
"""Generate a Google-signed OAuth ID token"""
token_request_url: str = f'http://metadata/computeMetadata/v1/instance/service-
accounts/default/identity?audience={TARGET_URL}'
token_request_headers: dict = {'Metadata-Flavor': 'Google'}
token_response = requests.get(token_request_url, headers=token_request_headers)
return token_response.content.decode("utf-8")
def do_request():
token: str = generate_token()
headers: dict = {
'Content-type': 'application/json',
'Accept': 'application/json',
'Authorization': f'Bearer {token}'
}
requests.post(url=TARGET_URL, json=data, headers=headers)

How do I protect my API that was built using Google Cloud Endpoints?

The API is a backend to a mobile app. I don't need user authentication. I simply need a way to secure access to this API. Currently, my backend is exposed.
The documentation seems to only talk about user authentication and authorization, which is not what I need here. I just need to ensure only my mobile app can talk to this backend and no one else.
Yes, you can do that: use authentication to secure your endpoints without doing user authentication.
I have found that this way of doing it is not well documented, and I haven't actually done it myself, but I intend to so I paid attention when I saw it being discussed on some of the IO13 videos (I think that's where I saw it):
Here's my understanding of what's involved:
Create a Google API project (though this doesn't really involve their API's, other than authentication itself).
Create OATH client ID's that are tied to your app via its package name and the SHA1 fingerprint of the certificate that you will sign the app with.
You will add these client ID's to the list of acceptable ID's for your endpoints. You will add the User parameter to your endpoints, but it will be null since no user is specified.
#ApiMethod(
name = "sendInfo",
clientIds = { Config.WEB_CLIENT_ID, Config.MY_APP_CLIENT_ID, Config.MY_DEBUG_CLIENT_ID },
audiences = { Config.WEB_CLIENT_ID }
// Yes, you specify a 'web' ID even if this isn't a Web client.
)
public void sendInfo(User user, Info greeting) {
There is some decent documentation about the above, here:
https://developers.google.com/appengine/docs/java/endpoints/auth
Your client app will specify these client ID's when formulating the endpoint service call. All the OATH details will get taken care of behind the scenes on your client device such that your client ID's are translated into authentication tokens.
HttpTransport transport = AndroidHttp.newCompatibleTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleAccountCredential credential = GoogleAccountCredential.usingAudience( ctx, Config.WEB_CLIENT_ID );
//credential.setSelectedAccountName( user ); // not specify a user
Myendpoint.Builder builder = new Myendpoint.Builder( transport, jsonFactory, credential );
This client code is just my best guess - sorry. If anyone else has a reference for exactly what the client code should look like then I too would be interested.
I'm sorry to say that Google doesn't provide a solution for your problem (which is my problem too).
You can use their API key mechanism (see https://developers.google.com/console/help/new/#usingkeys), but there is a huge hole in this strategy courtesy of Google's own API explorer (see https://developers.google.com/apis-explorer/#p/), which is a great development tool to test API's, but exposes all Cloud Endpoint API's, not just Google's services API's. This means anyone with the name of your project can browse and call your API at their leisure since the API explorer circumvents the API key security.
I found a workaround (based on bossylobster's great response to this post: Simple Access API (Developer Key) with Google Cloud Endpoint (Python) ), which is to pass a request field that is not part of the message request definition in your client API, and then read it in your API server. If you don't find the undocumented field, you raise an unauthorized exception. This will plug the hole created by the API explorer.
In iOS (which I'm using for my app), you add a property to each request class (the ones created by Google's API generator tool) like so:
#property (copy) NSString *hiddenProperty;
and set its value to a key that you choose. In your server code (python in my case) you check for its existence and barf if you don't see it or its not set to the value that your server and client will agree on:
mykey,keytype = request.get_unrecognized_field_info('hiddenProperty')
if mykey != 'my_supersecret_key':
raise endpoints.UnauthorizedException('No, you dont!')
Hope this puts you on the right track
The documentation is only for the client. What I need is documentation
on how to provide Service Account functionality on the server side.
This could mean a couple of different things, but I'd like to address what I think the question is asking. If you only want your service account to access your service, then you can just add the service account's clientId to your #Api/#ApiMethod annotations, build a GoogleCredential, and invoke your service as you normally would. Specifically...
In the google developer's console, create a new service account. This will create a .p12 file which is automatically downloaded. This is used by the client in the documentation you linked to. If you can't keep the .p12 secure, then this isn't much more secure than a password. I'm guessing that's why this isn't explicitly laid out in the Cloud Endpoints documentation.
You add the CLIENT ID displayed in the google developer's console to the clientIds in your #Api or #ApiMethod annotation
import com.google.appengine.api.users.User
#ApiMethod(name = "doIt", scopes = { Constants.EMAIL_SCOPE },
clientIds = { "12345678901-12acg1ez8lf51spfl06lznd1dsasdfj.apps.googleusercontent.com" })
public void doIt(User user){ //by convention, add User parameter to existing params
// if no client id is passed or the oauth2 token doesn't
// match your clientId then user will be null and the dev server
// will print a warning message like this:
// WARNING: getCurrentUser: clientId 1234654321.apps.googleusercontent.com not allowed
//..
}
You build a client the same way you would with the unsecured version, the only difference being you create a GoogleCredential object to pass to your service's MyService.Builder.
HttpTransport httpTransport = new NetHttpTransport(); // or build AndroidHttpClient on Android however you wish
JsonFactory jsonFactory = new JacksonFactory();
// assuming you put the .p12 for your service acccount
// (from the developer's console) on the classpath;
// when you deploy you'll have to figure out where you are really
// going to put this and load it in the appropriate manner
URL url = getClass().class.getResource("/YOURAPP-b12345677654.p12");
File p12file = new File(url.toURI());
GoogleCredential.Builder credentialBuilder = new GoogleCredential.Builder();
credentialBuilder.setTransport(httpTransport);
credentialBuilder.setJsonFactory(jsonFactory);
//NOTE: use service account EMAIL (not client id)
credentialBuilder.setServiceAccountId("12345678901-12acg1ez8lf51spfl06lznd1dsasdfj#developer.gserviceaccount.com"); credentialBuilder.setServiceAccountScopes(Collections.singleton("https://www.googleapis.com/auth/userinfo.email"));
credentialBuilder.setServiceAccountPrivateKeyFromP12File(p12file);
GoogleCredential credential = credentialBuilder.build();
Now invoke your generated client the same way
you would the unsecured version, except the builder takes
our google credential from above as the last argument
MyService.Builder builder = new MyService.Builder(httpTransport, jsonFactory, credential);
builder.setApplicationName("APP NAME");
builder.setRootUrl("http://localhost:8080/_ah/api");
final MyService service = builder.build();
// invoke service same as unsecured version

Resources