My application needs a bunch of secrets to run: database credentials, API credentials, etc. It's running in Google App Engine Standard Java 11. I need these secrets as environment variables or as arguments to my application, so that my framework can pick them up and establish the connections accordingly. My particular framework is Spring Boot, but I believe Django, Rails and many others use the same methods.
What's the best way of doing this?
One of the answers I get to this question is to use Google Cloud Key Management, which looks promising, but I can't figure out how to turn those values into environment variables in App Engine. Is it possible? I've read Setting Up Authentication for Server to Server Production Applications, but I don't see any indication there about how to turn the secrets into environment variables in App Engine (am I missing it?).
The other alternatives I've seen include hard-coding them in app.yaml or another file that is never committed and lives in my machine, which means I'm the only one who can deploy... I can't even deploy from another machine. This is problematic for me.
Another potential solution I've seen is to delegate the problem to Google Cloud Build, so that it fetches a value/file from CKM and pushes it to App Engine (1, 2). I'm not using GCB and I doubt I will, since it's so basic.
I really wish App Engine had a environment variables page like Heroku does.
[Update] (as of Feb 2020) GCP's Secret Manager is in beta, see:
https://cloud.google.com/secret-manager/docs/overview
For Java-specific implementation, see:
https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#secretmanager-access-secret-version-java
Your specific solution would depend how your app is set up, but you should be able to access the secret(s) and create environment variables with the values or otherwise pass them to your app.
You can use GCP IAM to create a service accounts to manage access or add a role like Secret Manager Secret Accessor to an existing member/service (e.g., in this case, I added that permision to the App Engine default service account).
I tried it out with Node.js on GAE standard, and it seems to work well; I didn't do any performance tests but it should be fine, particularly if you primarily need the secrets on app start or as part of a build process.
For local (non-GCP) development/testing, you can create a service account with appropriate secret manager permissions and get the json service key. You then set an environment variable named GOOGLE_APPLICATION_CREDENTIALS to the path of the file, e.g.:
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/local_service_key.json
and the app running in that shell session should pick up the permissions without any additional auth code.
See: https://cloud.google.com/docs/authentication/getting-started
(You would want to exclude the key file from version control.)
At this date, App Engine Standard Standard does not have a Google provided solution for storing application secrets.
[UPDATE]
I noticed your comment on another answer that you require environment variables to be valid before you have application control. In that case, you have no options for App Engine today. I would deploy to a different service (Kubernetes) better suited for your system goals that can provided managed secrets.
[END UPDATE]
You have two choices for secrets for App Engine Standard:
Store the secrets as environment variables in app.yaml
Store the secrets someplace else.
For both options, you can add a layer of security by encrypting them. However, adding encryption adds another secret (decryption key) that you must somehow provide to your app. The chicken-or-egg situation.
App Engine Standard uses a Service Account. This service account can be used as an identity to control access to other resources. Examples of other resources are KMS and Cloud Storage. This means that you can securely access KMS or Cloud Storage without adding another secret to App Engine.
Let's assume that your company wants all application secrets encrypted. We can use the App Engine Service Account as the identity authorized to access KMS for a single key.
Note: The following examples use Windows syntax. Replace the line continuation ^ with \ for Linux/macOS.
Create the KMS Keyring. Keyrings cannot be deleted, so this is a one-time operation.
set GCP_KMS_KEYRING=app-keyring
set GCP_KMS_KEYNAME=app-keyname
gcloud kms keyrings create %GCP_KMS_KEYRING% --location global
Create the KMS Key.
gcloud kms keys create %GCP_KMS_KEYNAME% ^
--location global ^
--keyring %GCP_KMS_KEYRING% ^
--purpose encryption
Add the service account to the KMS policy for the keyring and key that we created.
This will allow App Engine to decrypt data without requiring secrets for KMS. The service account identity provides access control. No roles are required for KMS. You will need to provide the KMS Keyring and Keyname which can be included in app.yaml.
set GCP_SA=<replace with the app engine service acccount email adddress>
set GCP_KMS_ROLE=roles/cloudkms.cryptoKeyDecrypter
gcloud kms keys add-iam-policy-binding %GCP_KMS_KEYNAME% ^
--location global ^
--keyring %GCP_KMS_KEYRING% ^
--member serviceAccount:%GCP_SA% ^
--role %GCP_KMS_ROLE%
For this example, let's assume that you need to access a MySQL database. We will store the credentials in a JSON file and encrypt it. The file is named config.json.
{
"DB_HOST": "127.0.0.1",
"DB_PORT": "3306",
"DB_USER": "Roberts",
"DB_PASS": "Keep-This-Secret"
}
Encrypt config.json using Cloud KMS and store the encrypted results in config.enc:
call gcloud kms encrypt ^
--location=global ^
--keyring %GCP_KMS_KEYRING% ^
--key=%GCP_KMS_KEYNAME% ^
--plaintext-file=config.json ^
--ciphertext-file=config.enc
The encrypted file can be stored in Cloud Storage. Since it is encrypted, you could store the file with your build files, but I do not recommend that.
The final piece is to write the code in Java that is part of your program that uses KMS to decrypt the file config.enc using KMS. Google has a number of examples of KMS decryption:
Java KMS Decrypt
Java Samples
You can pass secrets as env variables at build time. This example retrieves a Stripe API key and updates app.yaml within Cloud Build, ensuring the local file is not accidentally checked in to source control
First make sure the CloudBuild service account has IAM role Secret Manager Secret Accessor
an app.dev.yaml file with a place holder for the env variable
runtime: python39
env: standard
instance_class: F4
automatic_scaling:
max_instances: 1
env_variables:
STRIPE_API_KEY: STRIPE_API_VAR
etc
etc
Cloudbuild.yaml to retrieve the secret and insert at build time
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args:
- -c
- |
echo 'my api key from secret manager is '$$STRIPE_API_VAR
sed -i "s|STRIPE_API_VAR|$$STRIPE_API_VAR|g" app.dev.yaml
cat app.dev.yaml # you can now see the secret value inserted as the env variable
gcloud app deploy --appyaml=app.dev.yaml # deploy with the updated app.yaml, the local copy of the file is not changed
secretEnv: ['STRIPE_API_VAR']
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/stripe-api-key/versions/latest
env: 'STRIPE_API_VAR'
Berglas looks interesting.
Another option is to put the secrets in app.yaml file(s) (you can have more than one) and encrypt it before committing it to version control.
There are many tools to encrypt secrets before putting them in version control, like https://github.com/StackExchange/blackbox
Pros:
Very versatile
I find it simple to understand compared to other options
Easy to get started
Cons:
You can't really remove access for a person (since the file could always be copied) so you have rotate secrets sometimes
Can be hard to keep the unencrypted files out of the repo. Ones you get used to it, and have ignore files and/or scripts, it's usually OK.
For secret management, I'm personally fan of Berglas project. It's based on KMS and, in addition, manage DEK and KEK
It's today write in Go and it's not compliant with Java. I wrote a python library for some colleagues. I can write a Java package if you plan to use it. It's not very hard.
Let me know
Related
For using the google pub sub we need key.json file which serves as service account.
For deployed code we can pass the service_account: name#service-name.iam.gserviceaccount.com in app.yaml which works if everything is in google cloud.
As it is not recommend to download the service account file.
Is there a way where we can just pass the key name in local environment file along with individual credential who has access to service account in run time or any other way where we can run google cloud services in local enviornment without downloading the service key file?
Google makes this really hard to find, but gcloud auth application-default login might do what you need.
From that page:
Obtains user access credentials via a web flow and puts them in the
well-known location for Application Default Credentials (ADC).
This command is useful when you are developing code that would
normally use a service account but need to run the code in a local
development environment where it's easier to provide user credentials.
The credentials will apply to all API calls that make use of the
Application Default Credentials client library. Do not set the
GOOGLE_APPLICATION_CREDENTIALS environment variable if you want to use
the credentials generated by this command in your local development.
This command tries to find a quota project from gcloud's context and
write it to ADC so that Google client libraries can use it for billing
and quota. Alternatively, you can use the --client-id-file flag. In
this case, the project owning the client ID will be used for billing
and quota. You can create the client ID file at
https://console.cloud.google.com/apis/credentials.
I have an IdentityServer4 app running on App Engine. As I prepare for production, I have a lot of confusion using .AddSigningCredential() in this scenario. In app engine, I don't have a file system to cert store in a traditional sense, and it seems like I can only pass in a path to a cert/key or the contents itself.
As I'm hosting on Google Cloud, I would like to use KMS. The API for KMS gives me the ability to sign something, but Identity Server doesn't give me a way to use that, and I don't see a way for KMS to give Identity server what it wants.
The only workaround I came up with is to generate a key pair, save it as a secret, and pass that in to .AddSigningCredential(). The downside is that now I have to manage this key/secret manually. There must be another way.
Pub/Sub is really easy to use from my local work station. I set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path to my .json authentication object.
But what if you need to interact with multiple different Pub/Sub projects? This seems like an odd way to do authentication. Shouldn't there be a way to pass the .json object in from the Java code?
How can I use the client libraries without setting the system's environment variable?
Thanks
You can grant access for PubSub in different project using single Service Account and set it as env variable. See PubSub Access Control for more details.
A service account JSON key file allows to identify a service account on GCP. This service account is the equivalent of a user account but for the app (without user, only machine).
Thereby, if your app need to interact with several topic from different projects, you simply have to grant the service account email with the correct role on each topics/projects.
Another important thing. The service account key file are useful for app outside GCP. Else it's bad. Let me explain this. The service account key file is a secret file that you have to keep and store securely. And a file, it's easy to copy it, to send it by email and even to commit it into public git repository... In addition, it's recommended to rotate this key at least every 90 days for security reasons.
So, for all these reasons and the difficulty in security that represent the service account key files, I don't recommend you to use them.
With your local environment, use your user account (simply use the gcloud SDK and perform a gcloud auth application-default auth). Your aren't a machine (I hope!!)
With GCP component, use the component "identity" (load a service account when you deploy a service, create a VM,..., and grant the correct role on this service account without generating JSON key
With external app (other cloud provider, on premise, Apigee, CI/CD pipeline,...), generate a JSON file on your service account, you can't avoid them in this case.
Thanks for your help in advance!
I'm trying to create a simple but secure way to inject secrets from a commandline call to gcloud app deploy app.yml.
I'm relying on secret information stored in environment variables in my app's runtime, which I can set using the following app.yml:
runtime: nodejs
env: flex
env_variables:
SECRET_KEY: "passed-in-value"
For example, I'd like to be able to do something like
SECRETKEY=${LOCAL_SECRET_VALUE} gcloud app deploy app.yml
or use a cli argument if there is one, but I don't see any here.
At the end of the day, I just need a simple way to inject secrets so I can deploy to my testing environment from my local machine, or to production from a github action. I could just write the app.yml dynamically from my ci script and inject local environment variables, but it seems like there must be a more canonical way.
I can set environment variables with app.yml using the following syntax.
I would like a solution which works with both standard and flex app engine if possible.
Google Cloud has silently released in Beta Secret Manager. The APIs are available but the client libraries aren't yet. The service will be announced soon and will help you to manage your secrets.
The principle is simple: In your yaml files and in your GIT, you simply save a reference to the secret, for example mySecret#2 where 2 is the version number, and mySecret the name of your secret.
Then, perform a simple call to the API to get access to the secret
https://secretmanager.googleapis.com/v1beta1/projects/<myProject>/secrets/<MySecret>/versions/<Version>:access
The call must be secured with a Bearer access token in the header. Be sure that the App Engine service account has the required roles (default service account: YOUR_PROJECT_ID#appspot.gserviceaccount.com), at least roles/
secretmanager.secretAccessor
Note: I don't know how the library will be implemented if you request the secret without giving an explicit version. Is the 1st version which will be taken? the lastest?
For now, there is a lot of manual overhead (get the token, build the URL, manage errors,...). If you can wait a couple of weeks, the life will be easier with client libraries!
I have the same problem as the one mentioned here: Securely storing environment variables in GAE with app.yaml - namely:
"I need to store API keys and other sensitive information in app.yaml as environment variables for deployment on GAE. The issue with this is that if I push app.yaml to GitHub, this information becomes public (not good)."
Additionally I'm looking to check the following boxes:
Prevent vendor lock-in (as much as possible) & ability to take my dockerfile elsewhere.
Ease of deployment with GitHub. I want a push to the master which triggers a build.
Minimal setup, or a suitable effort and workflow for a solo-dev or small team.
My research yielded the following:
Securely storing environment variables in GAE with app.yaml
How to set environment variables/app secrets in Google App Engine
GAE : How to deploy various environments with secrets?
appengine and OS environment variables
How to pass environment variables to the app.yaml using cloud build
A lot of good information from GAE : How to deploy various environments with secrets?
where the author listed the three workarounds and their reason to not be used:
Use Google KMS - allows us to put encrypted secrets directly into the
project, but it requires us to put custom code in our apps to decrypt
them. It creates a different environment management between local,
staging and production. It increases the risk of bugs due to the
complexity.
Store secrets in Google Datastore - I tried it, I created a helper
that searches env vars in proccess.ENV, then in cache and ultimately
in Datastore. But like KMS, it increases complexity a lot.
Store secrets in a JSON file and put in on Google Cloud Storage : again, it requires to load env variables through an helper that
checks env vars, then loads the file etc...
However the best solution for me came from How to pass environment variables to the app.yaml using cloud build
It allows me to have the following deployment flow using GAE flexible environment for nodejs:
A merge to my Github master branch triggers a cloud build
My first step in my cloudbuild.yaml sources my app.yaml file using the gsutil builder, since app.yaml is not in source control
My app.yaml points to my dockerfile for my runtime and has my env variables
This checks all my boxes and was a fairly easy solution but, this definitely doesn't seem to be a popular solution, so am I missing something here?
Most importantly are there any security concerns?
I am amazed at how you did your research, you actually collected all the possible ways to do achieve it.
As you mentioned there are many ways to pass the variables to the application but I believe that the solution you propose ( storing the variables in Google Cloud Storage and retrieving them with Google Cloud Build ) is optimal for your purposes. It doesn't require much code and it's elegant, I hope this post helps people to be aware of this solution. Regarding your security concerns, this solution includes a high degree of security as you can set the file in the bucket to only be accessible from Google Cloud Build and the owner of the project.
Another solution I've employed, is to store the env variables in the Cloud Build trigger substitution variables directly and use a custom Cloud Builder envsubt to render a templated app.yaml.
I could not find documentation on how the substitution variables are stored in the Cloud Build trigger (any reference here would be helpful). However, I think most data in Google Cloud is encrypted at rest and encrypted on use and transfer. The main drawback is that the values are show in plain text, so sensitive information like API keys are not obscured, and any one who has access to the trigger can see the sensitive information.
One benefit is that this keeps the templated app.yaml close to the code you'll be using it with, and can be reviewed in the same pull request. Also you don't need to use another service, like Google Storage.
Steps:
Add the envsubst Cloud builder to your project, see instructions here.
Create a templated app.yaml file, e.g.
runtime: <your runtime>
service: ${GAE_SERVICE}
env_variables:
MY_VAR: ${MY_VAR}
MY_VAR_2: ${MY_VAR_2}
Add an app.yaml template rendering step in cloudbuild.yaml
steps:
- id: "render-app-yaml"
name: "gcr.io/${PROJECT_ID}/envsubst"
env:
- "GAE_SERVICE=${_GAE_SERVICE}"
- "MY_VAR=${_MY_VAR}"
- "MY_VAR_2=${_MY_VAR_2}"
args: ["app.yaml"]
Add the substitution variables in the Cloud Build trigger, e.g. _GAE_SERVICE, _MY_VAR, and _MY_VAR_2. Note: user-defined variables in the trigger are prefixed with a _.
When I was doing my research, I couldn't find any solution like this one either. Any feedback is welcome.