To access a remote datastore locally using the original dev_appserver I would set --default_partition=s as mentioned here
In March 2013 Google made devappserver2 the default development server, and it does not support --default_partition resulting in the original, dreaded:
BadRequestError: app s~appname cannot access app dev~appname's data
It appears like the first few requests are served correctly with
os.environ["APPLICATION_ID"] == 's~appname'
Then a subsequent request results in a call to /_ah/warmup and then
os.environ["APPLICATION_ID"] == 'dev~appname'
The docs specifically mention related topics but appear geared to dev_appserver here
Warning! Do not get the App ID from the environment variable. The development server simulates the production App Engine service. One way in which it does this is to prepend a string (dev~) to the APPLICATION_ID environment variable, which is similar to the string prepended in production for applications using the High Replication Datastore. You can modify this behavior with the --default_partition flag, choosing a value of "" to match the master-slave option in production. Google recommends always getting the application ID using the get_application_id() method, and never using the APPLICATION_ID environment variable.
You can do the following dirty little trick:
from google.appengine.datastore.entity_pb import Reference
DEV = os.environ['SERVER_SOFTWARE'].startswith('Development')
def myApp(*args):
return os.environ['APPLICATION_ID'].replace("dev~", "s~")
if DEV:
Reference.app = myApp
Related
Whilst developing I want to handle some things slight differently than I will when I eventually upload to the Google servers.
Is there a quick test that I can do to find out if I'm in the SDK or live?
See: https://cloud.google.com/appengine/docs/python/how-requests-are-handled#Python_The_environment
The following environment variables are part of the CGI standard, with special behavior in App Engine:
SERVER_SOFTWARE:
In the development web server, this value is "Development/X.Y" where "X.Y" is the version of the runtime.
When running on App Engine, this value is "Google App Engine/X.Y.Z".
In app.yaml, you can add IS_APP_ENGINE environment variable
env_variables:
IS_APP_ENGINE: 1
and in your Python code check if it has been set
if os.environ.get("IS_APP_ENGINE"):
print("The app is being run in App Engine")
else:
print("The app is being run locally")
Based on the same trick, I use this function in my code:
def isLocal():
return os.environ["SERVER_NAME"] in ("localhost", "www.lexample.com")
I have customized my /etc/hosts file in order to be able to access the local version by prepending a "l" to my domain name, that way it is really easy to pass from local to production.
Example:
production url is www.example.com
development url is www.lexample.com
I just check the httplib (which is a wrapper around appengine fetch)
def _is_gae():
import httplib
return 'appengine' in str(httplib.HTTP)
A more general solution
A more general solution, which does not imply to be on a Google server, detects if the code is running on your local machine.
I am using the code below regardless the hosting server:
import socket
if socket.gethostname() == "your local computer name":
DEBUG = True
ALLOWED_HOSTS = ["127.0.0.1", "localhost", ]
...
else:
DEBUG = False
ALLOWED_HOSTS = [".your_site.com",]
...
If you use macOS you could write a more generic code:
if socket.gethostname().endswith(".local"): # True in your local computer
...
Django developers must put this sample code in the file settings.py of the project.
EDIT:
According to Jeff O'Neill in macOS High Sierra socket.gethostname() returns a string ending in ".lan".
The current suggestion from Google Cloud documentation is:
if os.getenv('GAE_ENV', '').startswith('standard'):
# Production in the standard environment
else:
# Local execution.
Update on October 2020:
I tried using os.environ["SERVER_SOFTWARE"] and os.environ["APPENGINE_RUNTIME"] but both didn't work so I just logged all keys from the results from os.environ.
In these keys, there was GAE_RUNTIME which I used to check if I was in the local environment or cloud environment.
The exact key might change or you could add your own in app.yaml but the point is, log os.environ, perhaps by adding to a list in a test webpage, and use its results to check your environment.
I want to automatically start/stop our app engine services by running a bash script.
I know it's easy to run gcloud app versions start/stop, but I don't want to manually check the version number. I want to dynamically pass the version that is serving 100% traffic to gcloud and tell it to stop.
On the flip side, I also want to tell gcloud to start the most recently deployed version.
What's the recommended way to do this?
Thanks!
One way to do that is to use gcloud's keys and flags: projections, --format, --filters. To read more directly from the terminal use gcloud topic, for example:
gcloud topic projections
In order to see what fields/properties are available use --format=flattened, like:
gcloud app services list --format=flattened
For the sake of simplicity I will leave outside everything but gcloud.
for SERVICE in $(gcloud app services list --format='table[no-heading](id)'); do
echo "for service $SERVICE :"
RECENT=$(gcloud app versions list --format='table[no-heading](id)' --filter="service=$SERVICE" | tail -n1)
echo 'y' | gcloud app versions start $RECENT
VERSIONS=$(gcloud app versions list --format='table[no-heading](id)' --filter="service=$SERVICE AND version.servingStatus=SERVING AND NOT id=$RECENT" | tr '\n' ' ')
echo 'y' | gcloud app versions stop $VERSIONS
done
'table[no-heading](service)' outputs a table without heading, which is set in brackets, and a single column with service IDs, which is set in parentheses.
--filter="service=$SERVICE AND version.servingStatus=SERVING AND NOT id=$RECENT" will only show versions from indicated service that are serving, except the one indicated by RECENT.
Additionally, if you would want to use dates for filtering:
gcloud app versions list --format='table(id, version.servingStatus, version.createTime.date(format="%s"))' --filter="service=default" --sort-by="~version.createTime"
version.createTime.date(format="%s") is a function date converting version.createTime.date into the number of seconds since the Epoch.
%s comes from strftime(3) and returns dates in Epoch format which is easier to understand and compare.
--sort-by="~version.createTime"sorts by creation date and because of ~ in descending order.
One approach is to use the --stop-previous-version and/or --promote options when deploying with gcloud app deploy (they should be the default if I interpret the docs correctly, unless you use --no-stop-previous-version and/or --no-promote):
--promote
Promote the deployed version to receive all traffic. Overrides the
default app/promote_by_default property value for this command
invocation. Use --no-promote to disable.
--stop-previous-version
Stop the previously running version when deploying a new version that
receives all traffic. Overrides the default
app/stop_previous_version property value for this command
invocation. Use --no-stop-previous-version to disable.
But, if you're using the standard environment and dynamic scaling, you should be aware that if the previous version handles a lot of traffic there may be service degradation/interruptions during the switch (it may take a while for the GAE autoscaler to determine how many new version instances it needs to spin up to handle that traffic, see Use traffic migration or splitting when switching to a new default version. You can perform these programmatically, see Not applicable to the flex environment, which doesn't support traffic splitting.
Also potentially of interest: GAE shutdown or restart all the active instances of a service/app
You can only control at which deployed version(s) is the traffic routed to by default, you can't really stop all traffic to a deployed version, it can always be reached via targeted routing.
BTW, the gcloud app versions [start|stop] commands are only applicable to manually scaled services:
It may only be used if the scaling module for your service has been
set to manual.
Currently I'm getting access to the API though the GOOGLE_APPLICATION_CREDENTIALS environment variable. I created and downloaded a service account key as a .json file and set the environment variable to that file. Due to the requirements of the project I'm working on, I won't be able to do this. I tried using the credentials variable in the constructor for the datastore client object, but I wasn't able to get that to work. How should I be going about this?
I'm running Windows 10, but any solution should be (relatively) OS agnostic. I'm writing in python 3.6.
You can use from_service_account_json():
from google.cloud import datastore
client = datastore.Client.from_service_account_json('/path/to/credentials.json')
var = client.get(client.key('MyKind', '<key value>'))
print(var)
see: https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually
I am using the Python 2.7 runtime with NDB from the 1.6.2 SDK on Google App Engine.
I get the following error:
BadRequestError: app s~myapphr cannot access app dev~myapphr's data
Originating from this code:
device = model.Key(urlsafe=device_id).get()
I am accessing my app from dev.myapp.appspot.com that is aliased to myapphr. device_id was created on the same dev.myapphr version.
What is going on?
the dev server has a default default_partition of 'dev' and on production, HRD apps get a partition of 's'. If you create a urlsafe key on the dev server and store it as a string it will not work on a the production server with a different partition. the way to make keys portable is to save them in a ReferenceProperty with db or KeyProperty on ndb.
The prefix you see there ("s~" and "dev~") is called a partition. You can use the --default_partition flag for dev_appserver.py to change the prefix from "dev~" to "s~".
I'm trying to download data from my Google App Engine app, using the official instructions
Remote API is set up & I call:
appcfg.py download_data --application=appname --url=http://app.address.com/_ah/remote_api --filename=alldata.csv
The connection is established, I'm being asked for my e-mail & password, and then a long trace appears ending up with this:
google.appengine.api.datastore_errors.BadRequestError: app s~appname cannot access app appname's data
Any ideas?
If using the high-replication datastore, change the value passed to --application from appname to s~appname.
Found the problem - as an URL I was giving my own app's domain name instead of the 1.appname.appspot.com.
I ran into this same issue, but when attempting to use the remote API from a custom script as opposed to appcfg.py's bulk downloader.
This issue may have been addressed in more recent GAE SDK releases, but for various reasons I'm still using the older 1.7 release. I found the answer in this bug report: https://code.google.com/p/googleappengine/issues/detail?id=4374
Basically, when calling ConfigureRemoteApi, don't specify the APP_ID argument (just pass None) and specify your app ID via the host argument, e.g. 'myapp-hrd.appspot.com'. ConfigureRemoteApi will figure out your app ID correctly and won't add the 's~' that causes this problem.