I'm trying to set and use an environment variable on google app engine. My app.yaml file looks as below. However when I use os.Getenv("mytoken") I get an empty string instead the actual value I set. Is it a GAE bug?
api_version: go1
handlers:
- url: /.*
script: _go_app
env_variables:
mytoken: '88786d9b9a0359824'
The feature is now documented here. However, as I have tested just now, it doesn't work on AppEngine. It does work on local server, so don't be fooled..
Edit: It works on Google AppEngine. My previous failure is due to a mistake. Consider this:
import "os"
var consumer = OAuth1Consumer{
secret: os.Getenv("secret")
}
It does not work if you declare as global variable.
Unfortunately, the GAE Go runtime does not support environment variable setting in app.yaml -- see for example How to set GAE environment-specific environment variables? and https://groups.google.com/forum/#!msg/google-appengine-go/qzMbZapLyAU/eKOZzZO14qQJ .
The functionality is supported in PHP, per https://cloud.google.com/appengine/docs/php/config/appconfig#PHP_app_yaml_Defining_environment_variables ; Java, per https://cloud.google.com/appengine/docs/java/config/appconfig#Java_appengine_web_xml_System_properties_and_environment_variables ; and Python, per https://cloud.google.com/appengine/docs/python/config/appconfig#Python_app_yaml_Defining_environment_variables .
In the Go runtime for App Engine, however -- see https://cloud.google.com/appengine/docs/go/config/appconfig -- there is simply no equivalent functionality.
I would recommend opening a feature request at https://code.google.com/p/googleappengine/issues/list?can=2&q=language=Go&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log (I don't see any equivalent feature request already in the list of 27 open FRs, or else, of course, I would recommend just "starring" the existing FR to register your interest in it).
I can't find the docs for that field anywhere. I only see it in the python config, not the Go config. It's likely unavailable for go if you're basing this off of the python docs here
Related
Whilst developing I want to handle some things slight differently than I will when I eventually upload to the Google servers.
Is there a quick test that I can do to find out if I'm in the SDK or live?
See: https://cloud.google.com/appengine/docs/python/how-requests-are-handled#Python_The_environment
The following environment variables are part of the CGI standard, with special behavior in App Engine:
SERVER_SOFTWARE:
In the development web server, this value is "Development/X.Y" where "X.Y" is the version of the runtime.
When running on App Engine, this value is "Google App Engine/X.Y.Z".
In app.yaml, you can add IS_APP_ENGINE environment variable
env_variables:
IS_APP_ENGINE: 1
and in your Python code check if it has been set
if os.environ.get("IS_APP_ENGINE"):
print("The app is being run in App Engine")
else:
print("The app is being run locally")
Based on the same trick, I use this function in my code:
def isLocal():
return os.environ["SERVER_NAME"] in ("localhost", "www.lexample.com")
I have customized my /etc/hosts file in order to be able to access the local version by prepending a "l" to my domain name, that way it is really easy to pass from local to production.
Example:
production url is www.example.com
development url is www.lexample.com
I just check the httplib (which is a wrapper around appengine fetch)
def _is_gae():
import httplib
return 'appengine' in str(httplib.HTTP)
A more general solution
A more general solution, which does not imply to be on a Google server, detects if the code is running on your local machine.
I am using the code below regardless the hosting server:
import socket
if socket.gethostname() == "your local computer name":
DEBUG = True
ALLOWED_HOSTS = ["127.0.0.1", "localhost", ]
...
else:
DEBUG = False
ALLOWED_HOSTS = [".your_site.com",]
...
If you use macOS you could write a more generic code:
if socket.gethostname().endswith(".local"): # True in your local computer
...
Django developers must put this sample code in the file settings.py of the project.
EDIT:
According to Jeff O'Neill in macOS High Sierra socket.gethostname() returns a string ending in ".lan".
The current suggestion from Google Cloud documentation is:
if os.getenv('GAE_ENV', '').startswith('standard'):
# Production in the standard environment
else:
# Local execution.
Update on October 2020:
I tried using os.environ["SERVER_SOFTWARE"] and os.environ["APPENGINE_RUNTIME"] but both didn't work so I just logged all keys from the results from os.environ.
In these keys, there was GAE_RUNTIME which I used to check if I was in the local environment or cloud environment.
The exact key might change or you could add your own in app.yaml but the point is, log os.environ, perhaps by adding to a list in a test webpage, and use its results to check your environment.
I have taken the basic python 3 tutorial website on flask from this google cloud tutorial and I am able to set this up and the website works just fine.
In addition , I also wanted to run a python script which runs everyday to collect some data, but the cron job just doesn't work. I also added login: admin to restrict anyone to directly use that url
cron.yaml
cron:
- description: test dispatch vs target
url: /cronapp
schedule: every 5 hours
app.yaml
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
python_version: 3
handlers:
- url: /cronapp
script: cronapp.py
login: admin
I am calling this as http://myproject.appspot.com/cronapp also doesn't work and returns a 404.
what am I doing wrong ? any help is appreciated
Your app.yaml file is mixing up the standard environment Handlers element into a flexible environment configuration, so it is probably ignored. You can probably see the cron requests in the app's logs in the developer console (likely with errors, though).
You need to add a handler for /cronapp inside your app code, not in app.yaml. Not entirely sure how you do that (I'm still using only standard environment), it depends on your app and/or its framework. Take a look at the Hello World code review for a flask example.
Update:
I may not be entirely correct, I based my answer on documentation but I just noticed some inconsistencies (I sent some documentation feedback to Google for it).
The flexible environment Securing URLs for cron (which appears mostly copied from the standard environment equivalent) mentions a couple of solutions:
one indeed based on the login: admin option to handler:
You can restrict a URL by adding login: admin to the handler
configuration section in app.yaml. For more information see
Securing URLs
But handler is not mentioned in the Configuring your App with app.yaml and the Securing URLs is pointing to an inexistent tag. So I'm not sure if this is indeed working or not.
the second one is based on the X-Appengine-Cron header (same as in the standard environment):
Requests from the Cron Service will also contain a HTTP header:
X-Appengine-Cron: true
The X-Appengine-Cron header is set internally by Google App Engine.
If your request handler finds this header it can trust that the
request is a cron request. If the header is present in an external
user request to your app, it is stripped, except for requests from
logged in administrators of the application, who are allowed to set
the header for testing purposes.
But in Removed headers it is mentioned that:
In addition, some selected headers that match the following pattern
are removed from the request:
X-Appengine-*
It's unclear if this extends to X-Appengine-Cron or not. It's worth a try. This is my check in the (standard env, webapp2-based) cron handler code:
if self.request.headers.get('X-AppEngine-Cron') is None:
self.abort(403) # HTTPForbidden
I have two very simple services set up (in python) on google app engine. I have a site which has been up for a while as the defualt service of my example project, and I just deployed another service, lets call it foo. I have dns forwarding working for the default service, so I can go to example.com (which I own in this hypothetical scenario) and see my default service. I can also go to foo.example.appspot.com and see my foo service, which works at the url.
I have also registered another domain, let's call it foo.com. What I want is for foo.com to use my foo.example.appspot.com service. To make matters trickier, foo.com has a bunch of sub-domains of it's own, so I need x.foo.com to go to my foo service as well (and even x.y.foo.com would be ideal, but if that is hard I can work around that one easily enough by just substituting out another character so it would be x-y.foo.com or something along those lines).
I'm pretty new to web dev (which is honestly a generous description for this simple project), but Iv'e spent a while reading and googling and haven't found a solution to this, so any advice would be helpful. My last resort would be to un-service-ify (if you will) the two services, and just bundle it all up into one big main.py which routes differently depending on the domain, and then point both domains at the default service. The big downside to this is that I already have everything working with services (which seems like a better approach).
I expect to get very few users so scaling isn't really an issue (though it's always interesting to learn about).
EDIT:
attempted to create a dispatch.yaml file to solve this as follows:
dispatch:
- url: "foo.com"
module: foo
- url: "*.foo.com"
module: foo
but when I run appcfg.py -A <project_id_here> update_dispatch . it says
Error parsing yaml file:
Unable to assign value 'foo.com' to attribute 'url':
invalid url 'foo.com'
in "./dispatch.yaml", line 3, column 10
Yes, it should be possible. I didn't actually use 2 different top-level domains, but the development console appears to be ready to accept a second one (I can't actually check as I don't own a 2nd domain). Follow the Adding a custom domain for your application procedure.
Pay special attention to the Wildcard mappings section - since your foo.com domain already has sub-domains in use you can't use wildcards at/above those sub-domains, you'll have to specify the desired subdomains.
The procedure only maps (sub)domains to the app as a whole, not to individual services/modules. You'll also have to use a dispatch file, to route the specific subdomains to the corresponding modules/services. You can find examples here: https://stackoverflow.com/a/32103486/4495081 and https://stackoverflow.com/a/34111170/4495081 (the last one has rules to allow the module to work on the custom domain, on appspot.com and on the local development server).
I got it to work following The links in the answer. My dispatch file wound up looking like this:
dispatch:
- url: "foo.com/*"
module: foo
- url: "*.foo.com/*"
module: foo
I am trying to get a custom domain to work with Google App Engine 1.9.7 without SSL
I have done all the prerequisites;
Domain is verified with the proper TXT records.
Domain is configured in the GAE Cloud Console with the proper subdomain www.
Application is deployed the appspot.com domain and works.
But when I try to got to http://www.customdomain.com it immediately redirects to https://www.customdomain.com and I get the following error:
net::ERR_SSL_PROTOCOL_ERROR
I know that for SSL I need to set up a certificate.
I don't have any of my webapp modules configured to be secure.
I don't want SSL right now, I don't need it right now.
I found this little nugget after reading the instructions again and again:
It's okay for multiple domains and subdomains to point to the same
application. You can design your app to treat them all the same or
handle each one in a different way.
This is exactly what I want to do but I can't find any information on how to actually do this?
How do I get it to stop redirecting to the http to https?
I ran into the same problem. You say "I don't have any of my webapp modules configured to be secure." If that's the case, sorry, can't help you.
Otherwise the most likely cause for your problem would be: A "secure: always" flag for the respective handler in your app.yaml in the handlers section. Like so:
handlers:
- url: /*
secure: always
Remove the line with the "secure: always". Details in the official Google docs here (table item "secure").
How to run into this problem? I ran into it, because I copied the app.yaml from one of my other apps that didn't need to run on a custom domain, yet needed the SSL always.
For a Django/Python GAE app, by the way, the same problem is caused like this:
handlers:
- url: /.*
script: google.appengine.ext.django.main.app
secure: always
Same answer here: Remove or change the "secure" line. Python version just tested as described. Always works on the appspot.com domain, only without secure flag on a custom domain.
Just pointing out the above, as other people might run into this problem and come to this threat for help.
What I had to do was to shut down all instances and remove all versions, then do a fresh deployment from scratch, then I stopped having this problem.
To access a remote datastore locally using the original dev_appserver I would set --default_partition=s as mentioned here
In March 2013 Google made devappserver2 the default development server, and it does not support --default_partition resulting in the original, dreaded:
BadRequestError: app s~appname cannot access app dev~appname's data
It appears like the first few requests are served correctly with
os.environ["APPLICATION_ID"] == 's~appname'
Then a subsequent request results in a call to /_ah/warmup and then
os.environ["APPLICATION_ID"] == 'dev~appname'
The docs specifically mention related topics but appear geared to dev_appserver here
Warning! Do not get the App ID from the environment variable. The development server simulates the production App Engine service. One way in which it does this is to prepend a string (dev~) to the APPLICATION_ID environment variable, which is similar to the string prepended in production for applications using the High Replication Datastore. You can modify this behavior with the --default_partition flag, choosing a value of "" to match the master-slave option in production. Google recommends always getting the application ID using the get_application_id() method, and never using the APPLICATION_ID environment variable.
You can do the following dirty little trick:
from google.appengine.datastore.entity_pb import Reference
DEV = os.environ['SERVER_SOFTWARE'].startswith('Development')
def myApp(*args):
return os.environ['APPLICATION_ID'].replace("dev~", "s~")
if DEV:
Reference.app = myApp