I have a Google App Engine web app that runs the majority of my site. However, for certain functions, I need a linux machine. I would like my Google App Engine app to automatically spin-up a Google Compute Instance on certain events.
I understand that you can add Google Compute instances using the Compute Engine REST API. However, in order to access the Google Compute REST API, you need to get an access token using the OAuth2 authentication process.
How can I programmatically get an access token from within Google App Engine?
It seems that all of the authentication methods require a window to appear so you can type in your username and password, which is impractical from within Google App Engine.
Here is a complete example of using service accounts and App Engine cron tasks to stop instances after they've been running for a while:
(opposite of starting instances, but the authorization code will be the same)
https://github.com/GoogleCloudPlatform/compute-appengine-timeout-python
AppAssertionCredentials handles the access token using this code:
# Obtain App Engine AppAssertion credentials and authorize HTTP connection.
# https://developers.google.com/appengine/docs/python/appidentity/overview
credentials = AppAssertionCredentials(
scope='https://www.googleapis.com/auth/compute')
HTTP = credentials.authorize(httplib2.Http(memcache))
# Build object for the 'v1beta15' version of the GCE API.
# https://developers.google.com/compute/docs/reference/v1beta13/
compute = build('compute', 'v1beta15', http=HTTP)
You should be able to use the service account associated with your project to authenticate to the Compute Engine API and launch VMs.
Documentation on service accounts suggests that the following python code should fetch a service account token.
import httplib2
import discovery
from oauth2client.appengine import AppAssertionCredentials
...
credentials = AppAssertionCredentials(
scope='https://www.googleapis.com/auth/compute')
auth_http = credentials.authorize(httplib2.Http())
compute_service = discovery.build('compute', 'v1beta15', http=auth_http)
I'd thought that the Google I/O demo from this year where they built a video-sharing site was going to be available, but I don't see it on GitHub yet. There are a number of demos that use AppEngine to control GCE, but most of them seem to use the user's project and credentials, rather the app's own credentials.
Obviously, you probably don't want to spin up a VM on direct user input unless you've got a very large budget or some form of rate limiting in place, but it's quite helpful to spin up a VM now and then when you've got a lot of computation to do. (Transcoding, etc.)
Related
I created an API using Python + FastAPI and deployed it to Google App Engine and I would like to measure the cost for each request made.
I saw there is a header "x-appengine-estimated-cpm-us-dollars" that show up when logged in with the owner account on GAE, but I didn't see it when accessed the API using the browser "https://example.uc.r.appspot.com/api"
Any idea how to can I see this header or a way to get an estimated cost for each request made?
Note: the deployed script is an API, not a website with auth like the one mentioned here (Usage of X-AppEngine-Estimated-CPM-US-Dollars in AppEngine)
According to the documentation:
If you access dynamic pages on your site while signed in using an administrator account, App Engine includes per-request statistics in the response headers
And then shows the description for this particular header, therefore, this is not something that is available for APIs hosted in AppEngine.
You could alternatively use the Cloud Billing API to gather some information, although not exactly the same.
Our production app (python 2.7 standard environment) running on Google App Engine suddenly lost permissions to write (create objects) on Google Cloud Storage, without any change in the code on our side.
The code is able to create new buckets, but not new objects within them.
It seems that the default app engine service account is not granted the permission.
Needless to say, the service account has the Storage Object Creator role, as well as the Editor role on the project level.
Strangely, the exact same code running on the test environment project, continues to work perfectly.
We are using the api client library to obtain credentials, like so:
from oauth2client.appengine import AppAssertionCredentials
from apiclient.discovery import build as discovery_build
credentials = AppAssertionCredentials(scope=scope)
http = credentials.authorize(httplib2.Http())
service = discovery_build('storage', 'v1', http=http)
And then using the service to make the api call.
All calls to create objects are suddenly failing with the message:
"Anonymous caller does not have storage.objects.create access to /"
Any ideas what could suddenly have gone wrong ??
This turned up to be an issue with Google Cloud Storage (GCS).
A payed support ticket was opened, after approximately 90 hours, a rollback was made by Google GCS engineers which solved the issue, however, the root cause of the issue was not found or reported.
Very troubling that a production app can be affected this way for such a long time and eventually there is no explanation.
In Google App Engine, I have 3 services, 1 for front end, 2 for back end.
Is there a way to block http calls to my backend services for accounts not from my company's domain (and the service account of the front end), but allow everyone http access to my front end service?
I know there is the firewall option, but this is restricted to IP addresses, I would prefer user based
If it matters all services are python3
There's currently no option to filter traffic to specific App Engine services within a single application/project:
App Engine Firewall filters by source IP ranges but can only be set for the whole app, not per service.
Identity-Aware Proxy can filter access by user account as you'd prefer but also applies to the whole app. Also, it only supports user account and can't be used with service accounts.
One option you may have would be to split your app in 2 different projects. Keep the front-end in one project open to the world and restrict access to the backend services in your other project via firewall rules.
I have seen the following being used in task queues in GAE. Maybe it would help.
If u were using python 2, in standard environment, i think u could have used login handler element in app.yaml file.
You could have added following lines to your app.yaml file:
handlers:
- url: /.*
script: worker.app
login: admin
This prevents other users from accessing this service.
But the same login handler is not available for python3, according to Google Docs.
Just found following in Google Docs:
If a task performs sensitive operations (such as modifying data), you might want to secure the handler URL to prevent a malicious external user from calling it directly. You can prevent users from accessing task URLs by restricting access to App Engine administrators. Task requests themselves are issued by App Engine and can always target a restricted URL.
You can restrict a URL by adding the login: admin element to the handler configuration in your app.yaml file.
You can also call your backend services through cloud tasks or task queues (both are almost the same i guess), in case this only work for cloud tasks.
Find the code usage here:
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/6f5f3bcb81779679a24e0964a6c57c0c7deabfac/appengine/standard/taskqueue/counter
Find details about handler here.
https://cloud.google.com/appengine/docs/standard/python/config/appref#handlers_element
Find details about Cloud task and queue here:
https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/creating-handlers
My goal is to run a google app engine application with the minimal amount of access to resources it needs. In my case the application will access the datastorage in the project (this is the golang example tutorial using the source code git checkout origin/part4-usingdatastore from https://github.com/GoogleCloudPlatform/appengine-guestbook-go.git)
I did the following
Create a new project, foobarproject3
Created a new app in the project (using golang)
In the project IAM/IAM noticed the
foobarproject3#appspot.gserviceaccount.com, assumed this is the service account, so changed it's role to just BigQuery User. Notice that no Datastorage roles are configured (The UI forces me to provide access to something so I chose BQ)
Followed the tutorial instructions for the using datastore golang app (guestbook application)and deployed the app.
Opened the link to my app: https://foobarproject3.appspot.com/ It failed (this is great, this is what I expected, since the service account does not give the app permissions to read/write datastorage)
Refreshed https://foobarproject3.appspot.com/ and it started to work
There is something basic that I'm not understanding about service account from app engine. Isn't the app engine using these service account to access project resources? Why is the app getting access to datastorage when the service account does not have a policy that would allow access to datastorage?
"My goal is to run a google app engine application with the minimal amount of access to resources it needs."
This is dicey to unpack without more context. What is it that you're trying to achieve that goes beyond App Engine's default behavior?
My experience is that if one starts changing roles without understanding the basics, things go sideways (or South, or West, or Pear-shaped, depending on where you are). So I suspect you shot yourself in the foot in your third bullet.
When you access your app from the browser you are using your own user credentials, not the app's service account. And your user credentials might be exactly the app owner/admin ones, if you created the app using those credentials. See, for example, app.yaml handler login: admin option not effective on standard env python GAE app?
Make sure you log out from the app, or try accessing the app from an incognito browser window or by using a dumb(er) utility to prevent accidental/undesired credential leaking.
The app's service account is for your app to identify itself when it's interacting with other services/apps. From Understanding Service Accounts:
A service account is a special type of Google account that belongs to
your application or a virtual machine (VM), instead of to an
individual end user. Your application assumes the identity of the
service account to call Google APIs, so that the users aren't
directly involved. A service account can have zero or more pairs of
service account keys, which are used to authenticate to Google.
I have a Golang app running on App Engine and I would like to call the Analytics API to process some metrics. Both on App Engine and on the development server.
This page describes the overall procedure using a service account for server-to-server communication and getting an access token using OAuth2. I would like to avoid doing this myself so I found Google APIs for Go and the following that describes how to do it for App Engine.
I have added the App Engine app's service account email to Google Analytics for read access.
I have played around with queries using the Query Explorer and that works great. I have even used the API Query URI from here that includes the proper access token and that works fine. That works fine with cURL or calling it through urlfetch on App Engine.
What I need is to get hold of an access token automatically on App Engine and also while using the development server.
In here is an example for URL shortener which a have modified to use the Analytics API instead.
client := http.Client{
Transport: &oauth2.Transport{
Source: google.AppEngineTokenSource(c, analytics.AnalyticsScope),
Base: &urlfetch.Transport{Context: c},
},
}
svc, err := analytics.New(&client)
if err != nil {
return nil, err
}
data, err := svc.Data.Ga.Get("...")
The response in the development server is 401 Invalid Credentials, authError.
Is this supposed to work like this or am I missing something?
UPDATE
So, the problem is half solved. It works fine now using the development server when I pass in the service account's email and .pem file. However, that very same service account that is in App Engine does not work with the above code!
googleapi: Error 403: User does not have any Google Analytics account., insufficientPermissions
User should be the service account that I already successfully use on the development server, so I don't really understand this problem. It's the exact same code on App Engine and on the development server as well.
Google Analytics is set up to have read access and I use the read-only API. And it already works on the development server so that should be fine.
What might be different that is the issue?
It is possible to provide the development server with the necessary information at start like here.
$ dev_appserver.py --appidentity_email_address email --appidentity_private_key_path file.pem
And for App Engine you need to not use this Service Account but the one with the app name:
app_name#appspot.gserviceaccount.com