For using the google pub sub we need key.json file which serves as service account.
For deployed code we can pass the service_account: name#service-name.iam.gserviceaccount.com in app.yaml which works if everything is in google cloud.
As it is not recommend to download the service account file.
Is there a way where we can just pass the key name in local environment file along with individual credential who has access to service account in run time or any other way where we can run google cloud services in local enviornment without downloading the service key file?
Google makes this really hard to find, but gcloud auth application-default login might do what you need.
From that page:
Obtains user access credentials via a web flow and puts them in the
well-known location for Application Default Credentials (ADC).
This command is useful when you are developing code that would
normally use a service account but need to run the code in a local
development environment where it's easier to provide user credentials.
The credentials will apply to all API calls that make use of the
Application Default Credentials client library. Do not set the
GOOGLE_APPLICATION_CREDENTIALS environment variable if you want to use
the credentials generated by this command in your local development.
This command tries to find a quota project from gcloud's context and
write it to ADC so that Google client libraries can use it for billing
and quota. Alternatively, you can use the --client-id-file flag. In
this case, the project owning the client ID will be used for billing
and quota. You can create the client ID file at
https://console.cloud.google.com/apis/credentials.
Related
I have a Python script stored in a Compute Engine instance. I also have a web application deployed on the Google App Engine.
What I would like to achieve is let users enter some parameters on the web application interface and have it execute the script in the Compute Engine instance with the entered parameters.
My question is: how can I access the Compute Engine instance from App Engine and execute the script with the parameters that users passed in?
I think there are several factors here to take into account:
Security implications: Having a web site backend accessing a different host to run a command with client-defined parameters upon a request can easily introduce lots of potential exploits that aren't worth dealing with.
Sanitizing the parameter or parameters from the Python script would be a must, which you might be able to do this with shlex.quote().
Running the script in the VM instance via SSH from App Engine:
With the Google Cloud Client Library for Python you may be able to connect to a given GCE instance and run a command by setting up OS Login and granting roles/compute.osLogin for this instance to the Service Account that runs your App Engine app as described in this guide (with this example).
Otherwise, you may try creating a system account for this purpose inside the instance and allow its login in its /etc/ssh/ssh_config and use a new RSA key added to this user's ~/ssh/.authorized_keys file with a generic SSH client library like Paramiko connecting to its external IP address, assuming it has one.
In both cases this is going to introduce very high latencies for the requests as SSH sessions usually take 2+ seconds to get created.
As an acceptable-latency (and potentially more secure) alternative, you might be able to have a simple HTTPS service in the VM (you can probably check for a correct snakeoil certificate in your Python code if needed) and set up a webhook with a long hash-like URL path (and optionally a non-default port) handled by, for instance, a simple PHP script that runs the end script with exec() after passing the parameter variable in its $_POST[] superglobal through excapeshellarg() to (re-)sanitize it.
Pub/Sub is really easy to use from my local work station. I set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path to my .json authentication object.
But what if you need to interact with multiple different Pub/Sub projects? This seems like an odd way to do authentication. Shouldn't there be a way to pass the .json object in from the Java code?
How can I use the client libraries without setting the system's environment variable?
Thanks
You can grant access for PubSub in different project using single Service Account and set it as env variable. See PubSub Access Control for more details.
A service account JSON key file allows to identify a service account on GCP. This service account is the equivalent of a user account but for the app (without user, only machine).
Thereby, if your app need to interact with several topic from different projects, you simply have to grant the service account email with the correct role on each topics/projects.
Another important thing. The service account key file are useful for app outside GCP. Else it's bad. Let me explain this. The service account key file is a secret file that you have to keep and store securely. And a file, it's easy to copy it, to send it by email and even to commit it into public git repository... In addition, it's recommended to rotate this key at least every 90 days for security reasons.
So, for all these reasons and the difficulty in security that represent the service account key files, I don't recommend you to use them.
With your local environment, use your user account (simply use the gcloud SDK and perform a gcloud auth application-default auth). Your aren't a machine (I hope!!)
With GCP component, use the component "identity" (load a service account when you deploy a service, create a VM,..., and grant the correct role on this service account without generating JSON key
With external app (other cloud provider, on premise, Apigee, CI/CD pipeline,...), generate a JSON file on your service account, you can't avoid them in this case.
I'm trying to implement a simple design in google cloud using app engine standard and flexible with datastore. App1 lives in GAE standard environment. When a user interacts with this app, it writes some data to datastore and queues a task. The target of the queued task is App2 that lives in app engine flexible environment (the task can take a longer time to complete than standard environment allows). The idea is for App2 to read the data from datastore, perform the task using the data, once complete it should write a report entity to datastore. I've attached a simple diagram.
In App1 I've set up a Service Account named flexKey with Owner permissions, downloaded the json file.
when I run App2 locally I first export the path to the credentials json file as an environment variable:
GOOGLE_APPLICATION_CREDENTIALS: "path/to/flexKey.json"
then launch the app with mvn jetty:run-exploded and everything works fine, App2 is able to authenticate with live datastore (not local emulation), and read the data written by App1. When I unset the environment variable, I get an 'Unauthenticated' error (expected)
To use the same service account when App2 is deployed I've added the following in app.yaml for App2 to set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the service account json file flexKey.json (this is the path to the file on the deployed instance):
env: flex
env_variables:
GOOGLE_APPLICATION_CREDENTIALS: "/var/lib/jetty/webapps/root/WEB-INF/classes/flexKey.json"
runtime: java
However, when I deploy App2 to app engine flexible environment, there is an error authenticating with datastore when trying to do the read query (this works fine when querying datastore with the same credentials from a locally running instance of App2):
com.google.cloud.datastore.DatastoreException: Missing or insufficient permissions.
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:129)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:114)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:182)
at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:178)
at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:174)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:173)
.....
the code is PERMISSION_DENIED
If I leave out the environment variable GOOGLE_APPLICATION_CREDENTIALS from the app.yaml file for App2, then I get an 'Unauthenticated' error, so I think it's reading the file so I'm not sure what the issue is
I'm using Objectify v6
I'm not able to see why the same credentials (created with Owner role) work fine when querying the datastore from a locally running instance of the app but don't work when datastore is queried from the deployed version of the same app (deployed to flexible environment). setting the path to the credentials file via an environment variable in app.yaml is the method recommended in the documentation unless I am mistaken.
Is the GOOGLE_APPLICATION_CREDENTIALS environment variable not properly set in app.yaml?
Is there something conceptually problematic about my design?
All help appreciated.
You have a blocking issue in your design: it is not possible for one application to enqueue tasks into a push queue targeted at a service from another application. From the <target> (push queues) rows in the Syntax tables for both queue.yaml and queue.xml references:
The string is prepended to the domain name of your app when
constructing the HTTP request for a task. For example, if your app ID
is my-app and you set the target to my-version.my-service, the
URL hostname will be set to
my-version.my-service.my-app.appspot.com.
If you want to use the task queue then you have to make the 2 services part of the same application. As a (positive) side effect you don't have to worry about setting up the authentication for datastore access anymore - both services can directly access the app's datastore.
My goal is to run a google app engine application with the minimal amount of access to resources it needs. In my case the application will access the datastorage in the project (this is the golang example tutorial using the source code git checkout origin/part4-usingdatastore from https://github.com/GoogleCloudPlatform/appengine-guestbook-go.git)
I did the following
Create a new project, foobarproject3
Created a new app in the project (using golang)
In the project IAM/IAM noticed the
foobarproject3#appspot.gserviceaccount.com, assumed this is the service account, so changed it's role to just BigQuery User. Notice that no Datastorage roles are configured (The UI forces me to provide access to something so I chose BQ)
Followed the tutorial instructions for the using datastore golang app (guestbook application)and deployed the app.
Opened the link to my app: https://foobarproject3.appspot.com/ It failed (this is great, this is what I expected, since the service account does not give the app permissions to read/write datastorage)
Refreshed https://foobarproject3.appspot.com/ and it started to work
There is something basic that I'm not understanding about service account from app engine. Isn't the app engine using these service account to access project resources? Why is the app getting access to datastorage when the service account does not have a policy that would allow access to datastorage?
"My goal is to run a google app engine application with the minimal amount of access to resources it needs."
This is dicey to unpack without more context. What is it that you're trying to achieve that goes beyond App Engine's default behavior?
My experience is that if one starts changing roles without understanding the basics, things go sideways (or South, or West, or Pear-shaped, depending on where you are). So I suspect you shot yourself in the foot in your third bullet.
When you access your app from the browser you are using your own user credentials, not the app's service account. And your user credentials might be exactly the app owner/admin ones, if you created the app using those credentials. See, for example, app.yaml handler login: admin option not effective on standard env python GAE app?
Make sure you log out from the app, or try accessing the app from an incognito browser window or by using a dumb(er) utility to prevent accidental/undesired credential leaking.
The app's service account is for your app to identify itself when it's interacting with other services/apps. From Understanding Service Accounts:
A service account is a special type of Google account that belongs to
your application or a virtual machine (VM), instead of to an
individual end user. Your application assumes the identity of the
service account to call Google APIs, so that the users aren't
directly involved. A service account can have zero or more pairs of
service account keys, which are used to authenticate to Google.
I studied and could successfully replicate the quickstart.py example on https://developers.google.com/drive/web/quickstart/quickstart-python to upload a file to my google drive using command line.
However, I wish to write an app that does the same, but through a cron job i.e. uploads a file everyday at 8am say, without the need to authenticate each time. Is there sample code/examples that I can look at to implement the oauth steps without the command line intervention?
Thanks!
You can use your App Engine app's built-in Service Account to authorize requests to the Google Drive API.
https://cloud.google.com/appengine/docs/python/appidentity/
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
Your app will need to have an embedded Refresh Token, or some way of fetching it from a secure server. The Refresh Token acts a bit like a stored username/password, albeit with constrained access. Therefore you need to consider the security implications. For example, since it's uploading, it will only need drive.file scope, so your corpus of Drive files remain inaccessible.
If you're happy with the security implications, then the steps you need are described How do I authorise an app (web or installed) without user intervention? (canonical ?)