I've noticed that the developer console doesn't seem to expose anywhere where I can configure static environment variables.
Is the expectation on GAE that I will bundle those variables as part of the deployment from my build server? If so, is there any documentation on GAE/Google Cloud that covers why or details the philosophy?
Years have passed, and still it doesn't.
My workaround is to compile app.yaml during deployment process (locally or with CI).
For example, I have a template file app.tml.yaml file
runtime: python37
handlers:
- url: /static
static_dir: app/static/
- url: /.*
script: auto
env_variables:
DJANGO_GC_DATABASE_PASSWORD: ${DJANGO_GC_DATABASE_PASSWORD}
Then I call envsubst right before deployment
envsubst < ./app.tml.yaml > app.yaml and after that gcloud app deploy as usual. After the deployment is done app.yaml with sensitive data is deleted. Variables are read from local .env file or are set in CI system.
There also other approaches I found listed in this post: https://dev.to/mungell/google-cloud-app-engine-environment-variables-5990 but for me they are not convienient or generic enough.
Environment variables can be defined in your application's app.yaml
An example for a python/php/(maybe go?) app. Java uses a different format.
env_variables:
MY_ENV_VAR: 'some value here'
https://cloud.google.com/appengine/docs/python/config/appconfig#Python_app_yaml_Defining_environment_variables
You can set these values during your CI process as well if you need to by programmatically appending them to your app.yaml before deploying.
One solution is apparently https://cloud.google.com/secret-manager/docs, but I opted for the solution offered here:
Securely storing environment variables in GAE with app.yaml
First, put the environment variables in an env_variables.yaml, e.g.,
env_variables:
SECRET: 'my_secret'
Then, include this env_variables.yaml in the app.yaml
includes:
- env_variables.yaml
Finally, add the env_variables.yaml to .gitignore, so that the secret variables won't exist in the repository.
Further,
I commit env_variables.sample.yaml, with instructions and placeholder values which the next dev can fill in
in dev, I parse env_variables.yaml and add the vars to process.env so I have a single source of truth for those vars…
if (process.env.NODE_ENV === "development") {
try {
const fs = require("fs");
const yaml = require("js-yaml");
let fileContents = fs.readFileSync("./env_variables.yaml", "utf8");
let {env_variables} = yaml.load(fileContents);
console.log({ env_variables });
Object.keys(env_variables).forEach((v) => {
process.env[v] = env_variables[v];
});
} catch (error) {
res.status(500).end("500: Problem getting env vars");
return;
}
}
I'm adding my solution here as the quoted question specifies python, and this question is generic.
As with other PAAS solutions (eg Heroku, Netlify), if a user has access to the App Engine console, they can see the secrets (in this case by browsing the source files in the console).
Related
I'm trying to provide a .well-known folder under my Google App Engine Application I'm using the standard environment and the python27 runtime.
with a web-app-origin-association.json file to try the Progressive Web Apps as URL Handlers
origin trial from chrome.
I've added the following code to my app.yaml file under handlers:
# .well-known Ordner
- url: /.well-known/(.*)
static_files: well-known/\1
upload: well-known/.*
The folder in my project is named well-known without a dot cause I've read that there are problems when using a folder Name with a dot at the start of the foldername.
But the url https://example.com/.well-known/web-app-origin-associate.json isn't available instead it works without the dot:
What do I have to change in order to make it work under https://example.com/.well-known/web-app-origin-association.json?
You can use the workaround documented at "Make skip_files rule explicit and tweak to allow .well-known/* to upload":
^(.*/)?\.(?!well-known(?:/|$)).*$
You many want to migrate to Python 3 as described in the guide:
Starting on January 1, 2020, the Python community will no longer
update, fix bugs, or patch security issues for Python 2.7. We
recommend that you update apps that are still running in the Python 2
runtime of the App Engine standard environment to the Python 3 runtime
as soon as possible.
The best way i found out about is to just do it like that:
- url: /\.well-known
static_dir: .well-known
secure: always
and use the python39 runtime.
We have a web application (frontend) using React created from Create React App. The app is running on Google Cloud Platform App Engine Standard. Our web application is code splitted. Each page is then loaded on user navigation.
It's working really well. The issue we have is for example user A is on the app home page. We deploy a fix that change the chunk file name. The user A then try to access another page and then got the error Loading chunk * failed. The url to get the file is now returning a 404 because the file has been replaced by some new chunk files.
It's a frequent problem as I can see during my research but I didn't find a solution that apply for Google App Engine.
Here's an article that explain the problem / solution: https://mitchgavan.com/code-splitting-react-safely/
I would like to use the solution "Solution 1: Keep old files on your server after a deployment" but I can't see how to do this using GCP ...
Here's the app.yaml file
service: frontend
runtime: nodejs14
env: standard
instance_class: F1
handlers:
- url: /(.*\..+)$
static_files: build/\1
upload: build/(.*\..+)$
- url: /.*
static_files: build/index.html
upload: build/index.html
We have the following dispatch file (* for masked url)
dispatch:
- url: "*"
service: frontend
- url: "www.*"
service: frontend
Haven't tried this before but see if it makes sense/works.
We have a blog article about downloading your source code from GAE. It contains an explanation of where your source is stored when you deploy (a staging bucket), how long it stays there and how you can modify how long it stays before Google automatically deletes it.
When you deploy to GAE, gcloud only deploys files that have changed (it doesn't deploy those that haven't). Since you now have 'new' files because new hashes were generated, the older files no longer exist on your local machine. I do not know if Google will automatically delete those files from the staging location in bullet 1 above.
My proposal here is that you follow the steps in the blog article (from bullet 1) and alter (change) how long the files are retained in your staging bucket. Another option is to check the retention policy tab and see if you can change the rule so the files don't get deleted. If you're able to alter how long the files remain or the retention policy, it just might solve your problem. Let me know if this works
I know you can declare env_variables in your app.yaml as described in the app.yaml documentation. However, is it possible to include environment variables from your local environment into app.yaml when deploying.
As an example of what I'm trying to accomplish
# in app.yaml
runtime: python27
api_version:1
threadsafe: true
service: {{ $AN_ENVIRONMENT_VARIABLE }}
Yes, you can use includes: to specify an array of files to be included. And in the included file, you can specify env_variables: just like you do in app.yaml.
Example: app.yaml:
runtime: go
api_version: go1
env_variables:
FIST_VAR: myFirstVar
includes:
- credentials.yaml
credentials.yaml:
env_variables:
SECOND_VAR: mySecondVar
No, no such templating support exists for the app.yaml configuration files.
Side note: the app.yaml file is not only used to extract deployment instructions information, it's also used to configure the operation of the respective service on GAE. Making the service name configurable in such manner doesn't make a lot of sense unless the services being deployed are identical in every aspect (other than their name) - highly unlikely.
One possible approach for environment-specific deployment would be to have different version control branches for the app code, one for each environment, each having the desired app.yaml content.
Another one would be to wrap the deployment command in a script and perform the enviroment substitutions inside that script.
As for passing credentials info to the app a clean, straight-forward solution is not yet available. But approaches exist:
GAE: best practices for storing secret keys?
How to set environment variables/app secrets in Google App Engine
Google app engine: Best practice for hiding Rails secret keys?
How to handle sensitive configuration information when deploying app-engine applications?
if I have my circle-ci deplying to google with e.g. 'gcloud app deploy app.yaml' is there a simple way I can pass a dev / prod variable and have it replaced in my app.yml file below ?
Sure, I could have multiple app.dev.yml, app.dev.yml files etc but theres a lot of duplication involved unless I can just override some by using some sort of base app.yml
runtime: nodejs
env: flex
service: email
env_variables:
PUBSUB_TOPIC: dev-email-integration-email
PUBSUB_VERIFICATION_TOKEN: <your-verification-token>
Sadly there really isn't a good way to do this. I'd actually suggest not using app.yaml for this purpose, and instead using a json file and nconf. That's how we do it in all of our nodejs samples.
Hope this helps!
For example, I stumbled upon this link https://cloud.google.com/deployment-manager/configuration/create-configuration-file which shows below in its examples (not particularly for env).
resources:
- name: {{ env['name'] }}
type: compute.v1.instance
properties:
You can only set environment variables in app engine through the app.yaml and they can be accessed by your application instance.
You can define environment variables in app.yaml to make them available to the app:
env_variables:
MY_VAR: 'my value'
access with os.environ
import os
os.environ.get('MY_VAR', 'default value')
The app.yaml file must be valid YAML and must follow the app.yaml syntax.
The example given isn't YAML. It's a jinja template file used to create a configuration file.
There might be simpler ways but here's a long shot: you can generate a valid app.yaml with the right environment variables (with their values) from a script.
template.jinja - template to generate app.yaml
generate_app_yaml.py - python script to generate app.yaml