Environment variable for App Engine Flexible [duplicate] - google-app-engine

Whilst developing I want to handle some things slight differently than I will when I eventually upload to the Google servers.
Is there a quick test that I can do to find out if I'm in the SDK or live?

See: https://cloud.google.com/appengine/docs/python/how-requests-are-handled#Python_The_environment
The following environment variables are part of the CGI standard, with special behavior in App Engine:
SERVER_SOFTWARE:
In the development web server, this value is "Development/X.Y" where "X.Y" is the version of the runtime.
When running on App Engine, this value is "Google App Engine/X.Y.Z".

In app.yaml, you can add IS_APP_ENGINE environment variable
env_variables:
IS_APP_ENGINE: 1
and in your Python code check if it has been set
if os.environ.get("IS_APP_ENGINE"):
print("The app is being run in App Engine")
else:
print("The app is being run locally")

Based on the same trick, I use this function in my code:
def isLocal():
return os.environ["SERVER_NAME"] in ("localhost", "www.lexample.com")
I have customized my /etc/hosts file in order to be able to access the local version by prepending a "l" to my domain name, that way it is really easy to pass from local to production.
Example:
production url is www.example.com
development url is www.lexample.com

I just check the httplib (which is a wrapper around appengine fetch)
def _is_gae():
import httplib
return 'appengine' in str(httplib.HTTP)

A more general solution
A more general solution, which does not imply to be on a Google server, detects if the code is running on your local machine.
I am using the code below regardless the hosting server:
import socket
if socket.gethostname() == "your local computer name":
DEBUG = True
ALLOWED_HOSTS = ["127.0.0.1", "localhost", ]
...
else:
DEBUG = False
ALLOWED_HOSTS = [".your_site.com",]
...
If you use macOS you could write a more generic code:
if socket.gethostname().endswith(".local"): # True in your local computer
...
Django developers must put this sample code in the file settings.py of the project.
EDIT:
According to Jeff O'Neill in macOS High Sierra socket.gethostname() returns a string ending in ".lan".

The current suggestion from Google Cloud documentation is:
if os.getenv('GAE_ENV', '').startswith('standard'):
# Production in the standard environment
else:
# Local execution.

Update on October 2020:
I tried using os.environ["SERVER_SOFTWARE"] and os.environ["APPENGINE_RUNTIME"] but both didn't work so I just logged all keys from the results from os.environ.
In these keys, there was GAE_RUNTIME which I used to check if I was in the local environment or cloud environment.
The exact key might change or you could add your own in app.yaml but the point is, log os.environ, perhaps by adding to a list in a test webpage, and use its results to check your environment.

Related

How to access Google Datastore API with credentials without using environment variables?

Currently I'm getting access to the API though the GOOGLE_APPLICATION_CREDENTIALS environment variable. I created and downloaded a service account key as a .json file and set the environment variable to that file. Due to the requirements of the project I'm working on, I won't be able to do this. I tried using the credentials variable in the constructor for the datastore client object, but I wasn't able to get that to work. How should I be going about this?
I'm running Windows 10, but any solution should be (relatively) OS agnostic. I'm writing in python 3.6.
You can use from_service_account_json():
from google.cloud import datastore
client = datastore.Client.from_service_account_json('/path/to/credentials.json')
var = client.get(client.key('MyKind', '<key value>'))
print(var)
see: https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually

App Engine: Copy live Datastore to local dev Datastore (that still works)

This used to be possible by downloading with the bulkloader and uploading to the local dev server. However, the bulkloader download has been non-functional for several months now, due to not supporting oauth2.
A few places recommend downloading from a cloud storage backup, and uploading to the local datastore through either bulkloader or by directly parsing the backup. However, neither of these appear functional anymore. The bulkloader method throws:
OperationalError: unable to open database file
And the RecordsReader class, which is used to read the backup files, reaches end of file when trying to read the first record, resulting in no records being read.
Does there exist a current, functional, method for copying the live datastore to the local dev datastore?
RecordsReader is functioning perfectly on unix. I've tried this https://gist.github.com/jehna/3b258f5287fcc181aacf one day ago and it worked amazing.
You should add to the imports your Kinds implementation and run it in the datastore interactive shell.
for example:
from myproject.kinds_implementations import MyKind
I've removed the
for pp in dir(a):
try:
ppp = getattr(a, "_" + pp)
if isinstance(ppp, db.Key):
ppp._Key__reference.set_app(appname)
ppp
except AttributeError:
""" It's okay """
And it worked well. In my case the backup downloaded in multiple directories so I've modified the access to the directories. for some thing like that:
for directory in mypath:
full_directory_path = join(mypath, directory)
for sub_dir in listdir(directory_full_path):
full_sub_dir_path = join(full_directory_path, sub_dir)
onlyfiles = [ f for f in listdir(full_sub_dir_path) if isfile(join(mypath,f)) ]
for file in onlyfiles:
If you're working on windows you're welcome to follow my question about RecordsReader on windows, hopefully someone will answer there Google datastore backup to local dev_appserver
edit:
Working great on windows if you change the file open permissions from 'r' to 'rb'
The bulkloader is still functional on Python with OAuth2, albeit with some caveats. In downloading from the live app, there is an issue with refreshing of the OAuth2 token so the total download time is limited to 3600 seconds, or 3600+3600 if you manually use a refresh token with --oauth2_refresh_token.
When uploading to the development server app, OAuth2 will fail with a 401, so it's necessary to edit google.appengine.ext.remote_api.handler and stub out 'CheckIsAdmin' to always return True as a workaround:
def CheckIsAdmin(self):
return True
user_is_authorized = False
...
I upvoted the above answer however, as it looks like a more robust solution at this point.

how to set environment variables on google appengine?

I'm trying to set and use an environment variable on google app engine. My app.yaml file looks as below. However when I use os.Getenv("mytoken") I get an empty string instead the actual value I set. Is it a GAE bug?
api_version: go1
handlers:
- url: /.*
script: _go_app
env_variables:
mytoken: '88786d9b9a0359824'
The feature is now documented here. However, as I have tested just now, it doesn't work on AppEngine. It does work on local server, so don't be fooled..
Edit: It works on Google AppEngine. My previous failure is due to a mistake. Consider this:
import "os"
var consumer = OAuth1Consumer{
secret: os.Getenv("secret")
}
It does not work if you declare as global variable.
Unfortunately, the GAE Go runtime does not support environment variable setting in app.yaml -- see for example How to set GAE environment-specific environment variables? and https://groups.google.com/forum/#!msg/google-appengine-go/qzMbZapLyAU/eKOZzZO14qQJ .
The functionality is supported in PHP, per https://cloud.google.com/appengine/docs/php/config/appconfig#PHP_app_yaml_Defining_environment_variables ; Java, per https://cloud.google.com/appengine/docs/java/config/appconfig#Java_appengine_web_xml_System_properties_and_environment_variables ; and Python, per https://cloud.google.com/appengine/docs/python/config/appconfig#Python_app_yaml_Defining_environment_variables .
In the Go runtime for App Engine, however -- see https://cloud.google.com/appengine/docs/go/config/appconfig -- there is simply no equivalent functionality.
I would recommend opening a feature request at https://code.google.com/p/googleappengine/issues/list?can=2&q=language=Go&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log (I don't see any equivalent feature request already in the list of 27 open FRs, or else, of course, I would recommend just "starring" the existing FR to register your interest in it).
I can't find the docs for that field anywhere. I only see it in the python config, not the Go config. It's likely unavailable for go if you're basing this off of the python docs here

devappserver2, remote_api, and --default_partition

To access a remote datastore locally using the original dev_appserver I would set --default_partition=s as mentioned here
In March 2013 Google made devappserver2 the default development server, and it does not support --default_partition resulting in the original, dreaded:
BadRequestError: app s~appname cannot access app dev~appname's data
It appears like the first few requests are served correctly with
os.environ["APPLICATION_ID"] == 's~appname'
Then a subsequent request results in a call to /_ah/warmup and then
os.environ["APPLICATION_ID"] == 'dev~appname'
The docs specifically mention related topics but appear geared to dev_appserver here
Warning! Do not get the App ID from the environment variable. The development server simulates the production App Engine service. One way in which it does this is to prepend a string (dev~) to the APPLICATION_ID environment variable, which is similar to the string prepended in production for applications using the High Replication Datastore. You can modify this behavior with the --default_partition flag, choosing a value of "" to match the master-slave option in production. Google recommends always getting the application ID using the get_application_id() method, and never using the APPLICATION_ID environment variable.
You can do the following dirty little trick:
from google.appengine.datastore.entity_pb import Reference
DEV = os.environ['SERVER_SOFTWARE'].startswith('Development')
def myApp(*args):
return os.environ['APPLICATION_ID'].replace("dev~", "s~")
if DEV:
Reference.app = myApp

google app engine python uploading application first time

i'm trying to upload my app engine project for the very first time and i have no clue why it is not working. the error from my terminal is:
[me][~/Desktop]$ appcfg.py update ProjectDir/
Application: tacticalagentz; version: 1
Host: appengine.google.com
Starting update of app: tacticalagentz, version: 1
Scanning files on local disk.
Error 404: --- begin server output ---
This application does not exist (app_id=u'tacticalagentz').
--- end server output ---
i'm using python 2.6.5 and ubuntu 10.04.
not sure if this is relevant, but i just created a google app engine account today. and i also just created the application today (like a couple of hours ago). this is really frustrating because i just want to upload what i have so far (as a demo). in my app.yaml this is my first line:
application: tacticalagentz
Furthermore, i checked on my admin console, and i CLEARLY see the app id right there, and it matches letter for letter with the app id in my app.yaml
could someone please enlighten me and tell me what i am doing wrong? or is it something beyond my comprehension (like indexing issue with Google that they need time to index my app id) ?
thank you very much in advance
apparently adding the "--no_cookies" parameter will work
appcfg.py update --no_cookies ProjectDir/
the way i was able to find my answer was by uploading my app from my Mac OS X (thank god i have linux mac and windows). AppEngine on Mac OS X comes with a GUI interface, and it worked for uploading. so then i found the command they used in the console, which included "--no_cookies". perhaps if you run into similar issues in the future, this is one approach to getting the answer
App Engine for Java have the same problem. The problem is about account login.
If you are using Eclipse, use Sign In button.
If u are using command-line, use "-e" option, like this:
appcfg.sh -e your#email.com update yoursite/
I had the same problem. When I changed the name of the app I used in the launcher to match the one in the app engine, It worked without any problem. The way I figured out, it was the name mismatch which caused the problem. You can see the name of your registered app in the admin console of app engine.(https://appengine.google.com/)
Here's what fixed it for me:
i had an instance of dev_appserver.py myProjDirectory/ on a different terminal.
i guess the scripts are somehow linked and aren't thread safe
An alternate option that worked for me is to just "Clear Deployment Credential" from the Control option of the GUI. When the app was deployed after this, it opened a google page to allow GAE to access the user profile and then deployment was successful.
The key bit is
This application does not exist (app_id=u'tacticalagentz').
which is telling you that appspot.com doesn't know of an application by that name. The admin console (https://appengine.google.com/) shows your applications. Check there. You might have made an inadvertent typo when you registered the app.

Resources