I have an IdentityServer4 app running on App Engine. As I prepare for production, I have a lot of confusion using .AddSigningCredential() in this scenario. In app engine, I don't have a file system to cert store in a traditional sense, and it seems like I can only pass in a path to a cert/key or the contents itself.
As I'm hosting on Google Cloud, I would like to use KMS. The API for KMS gives me the ability to sign something, but Identity Server doesn't give me a way to use that, and I don't see a way for KMS to give Identity server what it wants.
The only workaround I came up with is to generate a key pair, save it as a secret, and pass that in to .AddSigningCredential(). The downside is that now I have to manage this key/secret manually. There must be another way.
Related
My application needs a bunch of secrets to run: database credentials, API credentials, etc. It's running in Google App Engine Standard Java 11. I need these secrets as environment variables or as arguments to my application, so that my framework can pick them up and establish the connections accordingly. My particular framework is Spring Boot, but I believe Django, Rails and many others use the same methods.
What's the best way of doing this?
One of the answers I get to this question is to use Google Cloud Key Management, which looks promising, but I can't figure out how to turn those values into environment variables in App Engine. Is it possible? I've read Setting Up Authentication for Server to Server Production Applications, but I don't see any indication there about how to turn the secrets into environment variables in App Engine (am I missing it?).
The other alternatives I've seen include hard-coding them in app.yaml or another file that is never committed and lives in my machine, which means I'm the only one who can deploy... I can't even deploy from another machine. This is problematic for me.
Another potential solution I've seen is to delegate the problem to Google Cloud Build, so that it fetches a value/file from CKM and pushes it to App Engine (1, 2). I'm not using GCB and I doubt I will, since it's so basic.
I really wish App Engine had a environment variables page like Heroku does.
[Update] (as of Feb 2020) GCP's Secret Manager is in beta, see:
https://cloud.google.com/secret-manager/docs/overview
For Java-specific implementation, see:
https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#secretmanager-access-secret-version-java
Your specific solution would depend how your app is set up, but you should be able to access the secret(s) and create environment variables with the values or otherwise pass them to your app.
You can use GCP IAM to create a service accounts to manage access or add a role like Secret Manager Secret Accessor to an existing member/service (e.g., in this case, I added that permision to the App Engine default service account).
I tried it out with Node.js on GAE standard, and it seems to work well; I didn't do any performance tests but it should be fine, particularly if you primarily need the secrets on app start or as part of a build process.
For local (non-GCP) development/testing, you can create a service account with appropriate secret manager permissions and get the json service key. You then set an environment variable named GOOGLE_APPLICATION_CREDENTIALS to the path of the file, e.g.:
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/local_service_key.json
and the app running in that shell session should pick up the permissions without any additional auth code.
See: https://cloud.google.com/docs/authentication/getting-started
(You would want to exclude the key file from version control.)
At this date, App Engine Standard Standard does not have a Google provided solution for storing application secrets.
[UPDATE]
I noticed your comment on another answer that you require environment variables to be valid before you have application control. In that case, you have no options for App Engine today. I would deploy to a different service (Kubernetes) better suited for your system goals that can provided managed secrets.
[END UPDATE]
You have two choices for secrets for App Engine Standard:
Store the secrets as environment variables in app.yaml
Store the secrets someplace else.
For both options, you can add a layer of security by encrypting them. However, adding encryption adds another secret (decryption key) that you must somehow provide to your app. The chicken-or-egg situation.
App Engine Standard uses a Service Account. This service account can be used as an identity to control access to other resources. Examples of other resources are KMS and Cloud Storage. This means that you can securely access KMS or Cloud Storage without adding another secret to App Engine.
Let's assume that your company wants all application secrets encrypted. We can use the App Engine Service Account as the identity authorized to access KMS for a single key.
Note: The following examples use Windows syntax. Replace the line continuation ^ with \ for Linux/macOS.
Create the KMS Keyring. Keyrings cannot be deleted, so this is a one-time operation.
set GCP_KMS_KEYRING=app-keyring
set GCP_KMS_KEYNAME=app-keyname
gcloud kms keyrings create %GCP_KMS_KEYRING% --location global
Create the KMS Key.
gcloud kms keys create %GCP_KMS_KEYNAME% ^
--location global ^
--keyring %GCP_KMS_KEYRING% ^
--purpose encryption
Add the service account to the KMS policy for the keyring and key that we created.
This will allow App Engine to decrypt data without requiring secrets for KMS. The service account identity provides access control. No roles are required for KMS. You will need to provide the KMS Keyring and Keyname which can be included in app.yaml.
set GCP_SA=<replace with the app engine service acccount email adddress>
set GCP_KMS_ROLE=roles/cloudkms.cryptoKeyDecrypter
gcloud kms keys add-iam-policy-binding %GCP_KMS_KEYNAME% ^
--location global ^
--keyring %GCP_KMS_KEYRING% ^
--member serviceAccount:%GCP_SA% ^
--role %GCP_KMS_ROLE%
For this example, let's assume that you need to access a MySQL database. We will store the credentials in a JSON file and encrypt it. The file is named config.json.
{
"DB_HOST": "127.0.0.1",
"DB_PORT": "3306",
"DB_USER": "Roberts",
"DB_PASS": "Keep-This-Secret"
}
Encrypt config.json using Cloud KMS and store the encrypted results in config.enc:
call gcloud kms encrypt ^
--location=global ^
--keyring %GCP_KMS_KEYRING% ^
--key=%GCP_KMS_KEYNAME% ^
--plaintext-file=config.json ^
--ciphertext-file=config.enc
The encrypted file can be stored in Cloud Storage. Since it is encrypted, you could store the file with your build files, but I do not recommend that.
The final piece is to write the code in Java that is part of your program that uses KMS to decrypt the file config.enc using KMS. Google has a number of examples of KMS decryption:
Java KMS Decrypt
Java Samples
You can pass secrets as env variables at build time. This example retrieves a Stripe API key and updates app.yaml within Cloud Build, ensuring the local file is not accidentally checked in to source control
First make sure the CloudBuild service account has IAM role Secret Manager Secret Accessor
an app.dev.yaml file with a place holder for the env variable
runtime: python39
env: standard
instance_class: F4
automatic_scaling:
max_instances: 1
env_variables:
STRIPE_API_KEY: STRIPE_API_VAR
etc
etc
Cloudbuild.yaml to retrieve the secret and insert at build time
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args:
- -c
- |
echo 'my api key from secret manager is '$$STRIPE_API_VAR
sed -i "s|STRIPE_API_VAR|$$STRIPE_API_VAR|g" app.dev.yaml
cat app.dev.yaml # you can now see the secret value inserted as the env variable
gcloud app deploy --appyaml=app.dev.yaml # deploy with the updated app.yaml, the local copy of the file is not changed
secretEnv: ['STRIPE_API_VAR']
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/stripe-api-key/versions/latest
env: 'STRIPE_API_VAR'
Berglas looks interesting.
Another option is to put the secrets in app.yaml file(s) (you can have more than one) and encrypt it before committing it to version control.
There are many tools to encrypt secrets before putting them in version control, like https://github.com/StackExchange/blackbox
Pros:
Very versatile
I find it simple to understand compared to other options
Easy to get started
Cons:
You can't really remove access for a person (since the file could always be copied) so you have rotate secrets sometimes
Can be hard to keep the unencrypted files out of the repo. Ones you get used to it, and have ignore files and/or scripts, it's usually OK.
For secret management, I'm personally fan of Berglas project. It's based on KMS and, in addition, manage DEK and KEK
It's today write in Go and it's not compliant with Java. I wrote a python library for some colleagues. I can write a Java package if you plan to use it. It's not very hard.
Let me know
I am using different third party API keys in my reactjs-firestore project. But I can't find a way to secure them in firebase hosting. How can I hide these API keys in firebase hosting?
For example, in Netlify hosting services they provide environment variables feature which can be used to secure the API keys.
that is I can just store the API keys in the variables in netlify and it will be retrieved from there which will be secured.
But in firebase how do I do this?
I can't seem to find a similar setting wherein I can store the keys as environment variables in the hosting services.
if there is no such feature is there another way to secure these API keys?
and for the firebase API keys,
I have already read some answers and understood that firebase API keys will not be hidden.
is there at least some way to secure these firebase API keys to just one secured URL at least? (I know that writing security rules is the best approach but am trying to find other options as well).
I can't seem to find a way to secure firebase project API key usage to one secured URL.
I have tried to find ways to secure the API key but I haven't been successful.
below is how I retrieve data in reactjs code
axios.post(`https://data.retrieval.com/1/data?key=API_KEY`, data)
I am trying to hide the API_KEY in the production code
I want to secure third party API keys in my hosted website.
and also restrict my firebase project API key to just one secure URL.
am not able to do this now.
any suggestions or solutions?
Thank you for trying to help.
and thank you for your time
If you're using the API key in client-side code, there is always the chance that a malicious user can find the key and abuse it. The only way to protect against this is to not use the API key in client-side code, or to have a backend system that can protect access based on something else (such as Firebase's server-side security rules).
Since your backend system likely doesn't have such a security model, you'll typically have to wrap their API in your own middleware that you host in a trusted environment such a server you control, or Cloud Functions. That's then where you ensure all access to the API is authorized, for example by setting up your own security system.
Not sure if this help, but my Firebase Cloud Function use this.
Create your secret by
firebase functions:config:set secret.API_KEY="THE API KEY"
Access your secret by using functions.config().secret.API_KEY
Note: This should only use for server use case, not in the client code. For server I meant Firebase Cloud Function or your backend.
The safe way I've found to store your third-party keys is using the Google Secrets Manager. It is now baked into the Firebase Functions SDK and works very well. You can find the information here, under the section titled "Store and access sensitive configuration information".
Two things worth mentioning:
There is a small bug in the syntax example, they forgot to add the https before onCall.
You'll need to give the service account which runs the cloud function when deployed access to the secrets. Here are the official docs on how to do that. If you are deploying through Firebase, you'll want to look for the service account whose address is [project-name]#appspot.gserviceaccount.com. If you have any doubts about which service account is running the Cloud Function, look under the Details tab in the Cloud Functions section of Google Cloud Platform and it will show you that information. Also, under the Variables tab, you can see what secrets your Cloud Function has access to.
This process makes it really easy to manage third-party keys as you can manage them at your project level and not have to worry about them being stored else where or needing to manage .env files. It also works with the Firebase Emulators and uses the credentials of the user running the emulators for access.
We are using a Google Compute Engine instance to generate convert datastore tables into large downloadable csv files. We want this instance to log errors to an app engine front end instance by calling:
/api/log
But we want to be sure that this call came from the Google Compute Engine instance which has the authority to write to the log. How can we check that an api call came from a Google Compute Engine instance?
It all depends on how secure you want it to be. The simplest solution is to include a parameter in your POST request that only your backend and front-end instances would recognize - any random sequence of characters will do the trick. The next level is to use a secret key to encrypt the contents of the request - there are many implementations depending on the language that you use.
This approach is more flexible too, if you decide, for example, to move your backend from App Engine to Compute Engine.
I have a google app engine application which needs to be given a public-private key pair. I don't want to check this into source control because it will be accessible by too many people. Since this is GAE I can't use the build system to write the keys to the file system of the server.
Is there a known best practice for this?
My first thought was does Jenkins provide a way to manage keys securely? I know I can just copy the keys to a location on the jenkins server and copy them into the build but this project will be used by third party teams so I need to provide a UI based solution in jenkins. I did not find any relevant plugin but I would like to make sure there isn't a better way before writing my own.
There are of course several approaches to this. I believe certificates are a concern of admins, not developers.
What we do is have custom admin pages where we upload certificates to blobstore under separate namespace. Then we have an internal "service" (just a simple factory) so that other pieces of code can retrieve certs.
If you are happy to use a cloud based Jenkins, we (CloudBees) have an oauth based solution at appengine.cloudbees.com
You could roll your own. It is not excessively tricky. You will need to
Register with google's api console to get a client key and secret and define the endpoints that your app will show up as
Write some way if feeding those credentials to your Jenkins build. I would recommend using the credentials plugin
Either write a build wrapper that exposes the refresh token to your build (python sdk deployment) or exposes the access token (java sdk... Got to love that the two sdks do the same thing in different ways)
Or use our free service ;-)
I want to create application with consists of Desktop application and google cloud storage. So, each my client should have separate cloud storage. Does google provide such thing?
More info.
Because I do not know what can offer google app engine i wrote this question.
I need some database hosting for my desktop application. In future I think I will switch to GWT and app engine. I want to sell my application so each my client can't access my other client databases. I was thinking that would be safer if each client will have data in a separate database so I can't do some mistakes in code.
You can separate data in the datastore using namespaces on google app engine:
https://developers.google.com/appengine/docs/java/multitenancy/multitenancy
It's up to you to decide how to implement the namespaces. You can separate them out by your user authentication system.
You can create a folder per client and restrict the folder access to the user (works only with Google Accounts) or your can do the same with buckets, create a bucket per user (which might be an overhead if you have a lot of users).
For database AppEngine datastore has the ability to separate the data by namespaces. this doen't require any user account and its your responsibility to select with which namespace to work with per request.
You can use GAE namespace capability as pointed above by #dragonx without Google authentication.
Use a client name as a namespace identifier (needs to be unique) . How you fetch this client name is upto you. It can be stored in GAE itself if you wish or can be deciphered from the url used specific to a client.
Do have a look at the GAE multitenancy link https://developers.google.com/appengine/docs/java/multitenancy/multitenancy
The example here can be easily adapted to use any string identifier per client.