I am porting a golang service from App Engine standard environment to the flexible environment, and have a question about accessing app.yaml during development/testing.
In my app.yaml I have a section where I set environment variables, that I later access in the code via os.GetEnv(...):
env_variables:
FORGE_CLIENT_ID: 'my-client-id'
FORGE_CLIENT_SECRET: 'my-client-secret'
In the App Engine standard environment, this runs fine, as I am using the app engine development server dev-server.py, which I believe takes care of reading the app.yaml file and making these environment variables available.
However, in the flexible environment during development, the service is started simply with go run *.go, and the app doesn't seem to pick up any information in app.yaml, meaning I get an error that my environment variables are not set.
I understand that app.yaml is used during deployment in the flexible environment, but I don't understand how or if it is used during development. How do I access these environment variables from my development server?
Thanks.
For Google App Engine flexible, the dev server don't use "app.yaml", as the configuration is set locally. A workaround for this would be to set the environment variables before running the dev server. You can check this question for setting those variables just for that command.
You will need to if/else based on whatever denotes dev vs live, then you can simply parse the yaml file to obtain and set these variables. For instance, one of my services (still running on standard mind you) is still using the old SDK where the version is required in the yaml. I parse it out like so:
import "github.com/ghodss/yaml" // or your choice
type AppVersion struct {
Version string `json:"version"`
}
func VersionID() (string, error) {
dat, err := ioutil.ReadFile("app.yaml")
if err != nil {
return "", err
}
a := &AppVersion{}
err = yaml.Unmarshal(dat, a)
if err != nil {
return "", err
}
return a.Version, nil
}
Obviously this is then stored in a global so I don't have to open the file each time, but it wouldn't really matter if it's only parsed on local dev. Your case would probably best be served by os.SetEnv This is just one possible solution that works for me.
Related
Whilst developing I want to handle some things slight differently than I will when I eventually upload to the Google servers.
Is there a quick test that I can do to find out if I'm in the SDK or live?
See: https://cloud.google.com/appengine/docs/python/how-requests-are-handled#Python_The_environment
The following environment variables are part of the CGI standard, with special behavior in App Engine:
SERVER_SOFTWARE:
In the development web server, this value is "Development/X.Y" where "X.Y" is the version of the runtime.
When running on App Engine, this value is "Google App Engine/X.Y.Z".
In app.yaml, you can add IS_APP_ENGINE environment variable
env_variables:
IS_APP_ENGINE: 1
and in your Python code check if it has been set
if os.environ.get("IS_APP_ENGINE"):
print("The app is being run in App Engine")
else:
print("The app is being run locally")
Based on the same trick, I use this function in my code:
def isLocal():
return os.environ["SERVER_NAME"] in ("localhost", "www.lexample.com")
I have customized my /etc/hosts file in order to be able to access the local version by prepending a "l" to my domain name, that way it is really easy to pass from local to production.
Example:
production url is www.example.com
development url is www.lexample.com
I just check the httplib (which is a wrapper around appengine fetch)
def _is_gae():
import httplib
return 'appengine' in str(httplib.HTTP)
A more general solution
A more general solution, which does not imply to be on a Google server, detects if the code is running on your local machine.
I am using the code below regardless the hosting server:
import socket
if socket.gethostname() == "your local computer name":
DEBUG = True
ALLOWED_HOSTS = ["127.0.0.1", "localhost", ]
...
else:
DEBUG = False
ALLOWED_HOSTS = [".your_site.com",]
...
If you use macOS you could write a more generic code:
if socket.gethostname().endswith(".local"): # True in your local computer
...
Django developers must put this sample code in the file settings.py of the project.
EDIT:
According to Jeff O'Neill in macOS High Sierra socket.gethostname() returns a string ending in ".lan".
The current suggestion from Google Cloud documentation is:
if os.getenv('GAE_ENV', '').startswith('standard'):
# Production in the standard environment
else:
# Local execution.
Update on October 2020:
I tried using os.environ["SERVER_SOFTWARE"] and os.environ["APPENGINE_RUNTIME"] but both didn't work so I just logged all keys from the results from os.environ.
In these keys, there was GAE_RUNTIME which I used to check if I was in the local environment or cloud environment.
The exact key might change or you could add your own in app.yaml but the point is, log os.environ, perhaps by adding to a list in a test webpage, and use its results to check your environment.
Currently I'm getting access to the API though the GOOGLE_APPLICATION_CREDENTIALS environment variable. I created and downloaded a service account key as a .json file and set the environment variable to that file. Due to the requirements of the project I'm working on, I won't be able to do this. I tried using the credentials variable in the constructor for the datastore client object, but I wasn't able to get that to work. How should I be going about this?
I'm running Windows 10, but any solution should be (relatively) OS agnostic. I'm writing in python 3.6.
You can use from_service_account_json():
from google.cloud import datastore
client = datastore.Client.from_service_account_json('/path/to/credentials.json')
var = client.get(client.key('MyKind', '<key value>'))
print(var)
see: https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually
I want to set global variable, for example:
var DEBUG_MODE bool
func init() {
// set DEBUG_MODE true if localhost (not appspot.com or other domain)
}
How to do this?
Easiest is to use use appengine.IsDevAppServer() to tell if your app is running in development mode (using the AppEngine SDK) or live (in production):
func IsDevAppServer() bool
IsDevAppServer reports whether the App Engine app is running in the development App Server.
Alternatively you can also use appengine.ServerSoftware() which contains this information along with your App version, merged into one string:
func ServerSoftware() string
ServerSoftware returns the App Engine release version. In production, it looks like "Google App Engine/X.Y.Z". In the development appserver, it looks like "Development/X.Y".
So for example what you want to do:
var DEBUG_MODE bool
func init() {
DEBUG_MODE = appengine.IsDevAppServer()
}
Or in one line:
var DEBUG_MODE = appengine.IsDevAppServer()
Although note that you could just call appengine.IsDevAppServer() whenever you would refer to DEBUG_MODE. Also the name DEBUG_MODE does not conform to Go naming conventions, it should either be DebugMode if it needs to be exported (because you want to access it from other packages too), or it should be debugMode if it doesn't need to be exported.
See this related question (possible duplicate?): How to set variables based on project id?
I currently have an App Engine Go app with 2 projects: myapp-prod and myapp-staging.
I'd like to be able to set the value of certain variables depending if the app is running in prod or staging.
Is there a way for the app to detect which environment it is running in?
Thanks
You can use the appengine.AppID() function to get the name/id of your application:
// AppID returns the application ID for the current application.
// The string will be a plain application ID (e.g. "appid"), with a
// domain prefix for custom domain deployments (e.g. "example.com:appid").
func AppID(c Context) string
And you can use appengine.IsDevAppServer() to tell if your app is running in development mode (using the AppEngine SDK) or live (in production):
// IsDevAppServer reports whether the App Engine app is running in the
// development App Server.
func IsDevAppServer() bool
Alternatively you can also use appengine.ServerSoftware() which contains both of the information above, merged into one string:
// ServerSoftware returns the App Engine release version.
// In production, it looks like "Google App Engine/X.Y.Z".
// In the development appserver, it looks like "Development/X.Y".
func ServerSoftware() string
Use an environment variable describing whether your app is on production or staging. Add to app.yml,
env_variables:
ENVIRONMENT: 'production'
In your code,
import "os"
if v := os.Getenv("ENVIRONMENT"); v == "production" {
// You're in production
}
To access a remote datastore locally using the original dev_appserver I would set --default_partition=s as mentioned here
In March 2013 Google made devappserver2 the default development server, and it does not support --default_partition resulting in the original, dreaded:
BadRequestError: app s~appname cannot access app dev~appname's data
It appears like the first few requests are served correctly with
os.environ["APPLICATION_ID"] == 's~appname'
Then a subsequent request results in a call to /_ah/warmup and then
os.environ["APPLICATION_ID"] == 'dev~appname'
The docs specifically mention related topics but appear geared to dev_appserver here
Warning! Do not get the App ID from the environment variable. The development server simulates the production App Engine service. One way in which it does this is to prepend a string (dev~) to the APPLICATION_ID environment variable, which is similar to the string prepended in production for applications using the High Replication Datastore. You can modify this behavior with the --default_partition flag, choosing a value of "" to match the master-slave option in production. Google recommends always getting the application ID using the get_application_id() method, and never using the APPLICATION_ID environment variable.
You can do the following dirty little trick:
from google.appengine.datastore.entity_pb import Reference
DEV = os.environ['SERVER_SOFTWARE'].startswith('Development')
def myApp(*args):
return os.environ['APPLICATION_ID'].replace("dev~", "s~")
if DEV:
Reference.app = myApp