I am using the webapp2 framework on Google App Engine, and I'm getting a basic error in one of my Request Handlers.
The app is running ok in the local instance, but causes the following traceback on the deployed version of Google App Engine:
Here's the code:
import os
from google.appengine.ext.webapp import template
import webapp2
import logging
class MainHandler(webapp2.RequestHandler):
def get(self):
logging.info('hi there 34')
template_values = {}
self.response.out.write('hello world 4')
path = os.path.join(os.path.dirname(__file__), 'index.html')
## This is the code that causes the bug ##
self.response.out.write(template.render(path, template_values))
## ## ## ##
debug = os.environ.get('SERVER_SOFTWARE', '').startswith('Dev')
app = webapp2.WSGIApplication(
[(r'/main', MainHandler)],
debug = debug)
def main():
app.run()
traceback error:
Traceback (most recent call last):
File "/base/python27_runtime/python27_dist/lib/python2.7/wsgiref/handlers.py",
line 86, in run
self.finish_response()
File "/base/python27_runtime/python27_dist/lib/python2.7/wsgiref/handlers.py",
line 127, in finish_response
self.write(data)
File "/base/python27_runtime/python27_dist/lib/python2.7/wsgiref/handlers.py",
line 202, in write
assert type(data) is StringType,"write() argument must be string"
AssertionError: write() argument must be string
What does this error mean?
I think response does not take unicode data, so you have to encode it first:
content = template.render(path, template_values)
self.response.out.write(content.encode('utf-8'))
Also I recommend Werkzeug. It works well on appengine and makes life so much easier. It helps to deal with request and response data, url routing, provides http exceptions, has great debugger for offline development and more. I think Werkzeug is a must to have for every python web dev in their toolbox.
Related
I am writing a new application using App Engine and, as the docs suggest to not use Blobstore API, I'm using the Google Cloud Storage client (GCS). All is good but I want to be able to return "signed urls" to clients so they can get the GCS resources without passing through the application. I believe that is what signet urls are for.
But how to test that? I can sucessfully test GCS calls from the client, but I have no idea how to test the client's HTTP calls using urlfetch.
Below is a full test case that illustrates my issue:
import base64
import mimetypes
import urllib
import urllib2
from datetime import datetime, timedelta
import time
from google.appengine.api import app_identity
from google.appengine.datastore import datastore_stub_util
from google.appengine.ext import testbed
from google.appengine.ext import ndb
import unittest
import cloudstorage
# IS THIS RIGHT ?
GCS_API_ACCESS_ENDPOINT = 'http://localhost:8000/_ah/gcs'
def sign_url(bucket_object, expires_after_seconds=60):
""" cloudstorage signed url to download cloudstorage object without login
Docs : https://cloud.google.com/storage/docs/access-control?hl=bg#Signed-URLs
API : https://cloud.google.com/storage/docs/reference-methods?hl=bg#getobject
"""
# source: https://github.com/voscausa/appengine-gcs-signed-url/blob/05b8a93e2777679d40af62cc5ffce933216e6a85/sign_url.py
method = 'GET'
gcs_filename = urllib.quote(bucket_object)
content_md5, content_type = None, None
# expiration : number of seconds since epoch
expiration_dt = datetime.utcnow() + timedelta(seconds=expires_after_seconds)
expiration = int(time.mktime(expiration_dt.timetuple()))
# Generate the string to sign.
signature_string = '\n'.join([
method,
content_md5 or '',
content_type or '',
str(expiration),
gcs_filename])
signature_bytes = app_identity.sign_blob(signature_string)[1]
google_access_id = app_identity.get_service_account_name()
# Set the right query parameters. we use a gae service account for the id
query_params = {'GoogleAccessId': google_access_id,
'Expires': str(expiration),
'Signature': base64.b64encode(signature_bytes)}
# Return the built URL.
result = '{endpoint}{resource}?{querystring}'.format(endpoint=GCS_API_ACCESS_ENDPOINT,
resource=gcs_filename,
querystring=urllib.urlencode(query_params))
return result
FILE_DATA = "This is file contents."
MIME = "text/plain"
class TestGCS(unittest.TestCase):
def setUp(self):
self.testbed = testbed.Testbed()
self.testbed.activate()
self.policy = datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=0)
self.testbed.init_datastore_v3_stub(consistency_policy=self.policy)
self.testbed.init_app_identity_stub()
self.testbed.init_memcache_stub()
self.testbed.init_urlfetch_stub()
self.testbed.init_blobstore_stub()
ndb.get_context().clear_cache()
def tearDown(self):
self.testbed.deactivate()
def test_gcs_works(self):
with cloudstorage.open('/mybucket/test.txt', 'w', content_type=MIME) as f:
f.write(FILE_DATA)
with cloudstorage.open('/mybucket/test.txt', 'r') as f:
data = f.read()
print(data)
self.assertEqual(data, FILE_DATA)
def test_signurl(self):
url = sign_url('/mybucket/test.txt')
# FIXME: Not yet working as we have no idea on how to access local GCS during the test.
result = urllib2.urlopen(url)
self.assertEqual(200, result.code)
self.assertEqual(FILE_DATA, result.read())
You can test GCS and service_accounts in your SDK, but you do not have a local appengine GCS service when you use a signed url.
But you can test your local app with service accounts and google cloud services.
Service accounts make it very easy to authorize appengine requests to other Google APIs and services.
To use a service account in the appengine SDK, you have to add two undocumented options when you run the development server:
--appidentity_email_address=<SERVICE_ACCOUNT_EMAIL_ADDRESS>
--appidentity_private_key_path=<PEM_KEY_PATH>
More info in this request for documentation issue
You can create or find the service account in the developers console permissions section of your appengine cloud project.
And you can create and download a p12 key for the service account.
Use OpenSSL to convert this p12 key in a RSA pem key.
I used this OpenSSL installer for Windows.
To create the pem key file in Windows use:
openssl pkcs12 -in <P12_KEY_PATH> -nocerts -nodes -passin pass:notasecret | openssl rsa -out <PEM_KEY_PATH>
Now you can use your cloud app service accounts in the development server and use app_identity to sign and authorize requests.
I started with some sample code for App Engine from Google.
My app needs to use the Directory API and the Reports API from the Google Admin SDK.
I have created a project in the API Console and turned on the Admin SDK in Services.
I added the scopes (the same ones as used in the code below) to the "Manage API client access" section of Advanced Tools in my domain's Google cpanel.
The call to the Directory API works.
After that, the call to the Reports API fails with the error message:
"HttpError: https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/admin?alt=json returned "Insufficient Permission">"
Thanks much for the assistance.
import webapp2
import os
from apiclient.discovery import build
from oauth2client.appengine import OAuth2Decorator
from oauth2client.appengine import OAuth2DecoratorFromClientSecrets
from apiclient import errors
import logging
import json
decorator = OAuth2DecoratorFromClientSecrets(
os.path.join(os.path.dirname(__file__), 'client_secrets.json'),
'https://www.googleapis.com/auth/admin.directory.user.readonly')
directoryauthdecorator = OAuth2Decorator(
client_id='123.apps.googleusercontent.com',
client_secret='456-abc',
callback_path='/oauth2callback',
scope='https://www.googleapis.com/auth/admin.directory.user.readonly '
'https://www.googleapis.com/auth/admin.reports.audit.readonly '
'https://www.googleapis.com/auth/admin.reports.usage.readonly'
)
class MainHandler(webapp2.RequestHandler):
def get(self):
self.response.write('Hello world!')
class OAuthHandler(webapp2.RequestHandler):
#directoryauthdecorator.oauth_required
def get(self):
users = []
# Get the authorized Http object created by the decorator.
auth_http = directoryauthdecorator.http()
# Get the directory service
service = build("admin", "directory_v1", http=auth_http)
result = []
page_token = None
while True:
try:
param = {}
param['domain'] = 'mydomain.com'
if page_token:
param['pageToken'] = page_token
files = service.users().list(**param).execute()
result.extend(files['users'])
page_token = files.get('nextPageToken')
if not page_token:
break
except errors.HttpError, error:
print 'An error occurred: %s' % error
break
users = []
for user in result:
logging.info(user['primaryEmail'])
users.append(user['primaryEmail'])
param = {}
param['userKey'] = 'all'
param['applicationName'] = 'admin'
service = build('admin', 'reports_v1', http=auth_http)
# this call fails with the 403 Insufficient Permissions error
results = service.activities().list(**param).execute()
logging.info(results)
app = webapp2.WSGIApplication([
('/', MainHandler),
('/users', OAuthHandler),
(directoryauthdecorator.callback_path, directoryauthdecorator.callback_handler()),
], debug=True)
I read this post and cleared the credentials from the datastore.
Hitting the /users url again I got the redirect_uri error message.
I went back to the API project, fixed the Redirect URIs, and downloaded the client_secrets.json file.
Now both calls work (one to Directory API, the other to Reports API).
I've been trying to use web.py to run a simple web app on Google App Engine but have been running into some pretty basic errors. I've searched over the site and haven't found anything to address my issue. Here is the outline of the code I'm trying to run:
import web
urls = (
"/","Index"
)
app = web.application(urls,globals())
render = web.template.render('pages/', base="layout")
class Index:
def GET(self):
#code...
if __name__ == "__main__":
app.cgirun()
and this is the app.yaml code:
application: #appname
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: home.app
- url: /static
static_dir: static
But then I get this in the log:
2013-08-22 06:11:13 Running command: "['C:\\Python27\\pythonw.exe', 'C:\\Program Files\\Google\\google_appengine\\dev_appserver.py', '--skip_sdk_update_check=yes', '--port=8080', '--admin_port=8000', 'C:\\.....\\root\\home-gae']"
INFO 2013-08-22 06:11:16,956 devappserver2.py:557] Skipping SDK update check.
WARNING 2013-08-22 06:11:16,976 api_server.py:317] Could not initialize images API; you are likely missing the Python "PIL" module.
INFO 2013-08-22 06:11:17,006 api_server.py:138] Starting API server at: http://localhost:64510
INFO 2013-08-22 06:11:17,013 dispatcher.py:164] Starting module "default" running at: http://localhost:8080
INFO 2013-08-22 06:11:17,019 admin_server.py:117] Starting admin server at: http://localhost:8000
ERROR 2013-08-22 10:11:24,303 wsgi.py:235]
Traceback (most recent call last):
File "C:\Program Files\Google\google_appengine\google\appengine\runtime\wsgi.py", line 223, in Handle
result = handler(dict(self._environ), self._StartResponse)
AttributeError: application instance has no __call__ method
INFO 2013-08-22 06:11:24,313 module.py:593] default: "GET / HTTP/1.1" 500 -
The AttributeError is confusing me because there does appear to be a call method in web/application module. Any ideas? Any thoughts would be appreciated.
I found a way to work it out.
import web
urls = (
"/.*", "hello"
)
application = web.application(urls, globals())
#app = web.application(urls, globals())
class hello:
def GET(self):
return "HelloWorld"
#app = app.gaerun()
#app.cgirun()
app = application.wsgifunc()
using "app = application.wsgifunc() ", then the code would work well.
First, there is a little problem with your app.yaml. You need to put your static handler before the catch-all handler:
handlers:
- url: /static
static_dir: static
- url: /.*
script: home.app
Otherwise, you will not be able to serve static files.
To fix your issue with the website not loading, it looks like the development server is trying to treat your CGI app as a WSGI app. Try to fit you home.py file to the official example for web.py on GAE. That is, get rid of the if __name__ == "__main__:" part and simply replace it with:
app = app.gaerun()
I am new to Google App Engine. I am able to start the appserver from the command prompt of Windows and its showing "Info : The server is running at http://localhost:8080/" but still when I enter the address on my browser its showing the Sever not found error.
There are a quite a few reasons this could be happening. If you have checked your firewall settings I'm willing to bet that there is an issue with your app.yaml file.
If your main python script is called main and is located in the root directory of your application code a working example of an app.yaml file is:
application: yourapplicationname
version: 1
runtime: python
api_version: 1
- url: .*
script: main.py
Also make sure that within your main.py file the routes are correct.
eg:
from google.appengine.ext import webapp
class Main(webapp.RequestHandler):
def get(self):
self.response.out.write('Hello World')
def main():
application = webapp.WSGIApplication([('/', Main)],
debug=True)
util.run_wsgi_app(application)
if __name__ == '__main__':
main()
I'm trying to use the App Engine datastore in my application outside of the bounds of one of the request handler pages (like main.py or the other files you can specify in app.yaml). Is this possible? When I try to run it, it says my database model does not have a method "put" associated with it, but it has no problems when I run it from a script connected to CGI and request handling (like main.py or any other .py files I declare similarly in app.yaml). Do you know what I can do to add CGI to those pages without adding request handling? Thanks!
Edit my code:
users.py file:
from google.appengine.ext import db, webapp
class User(db.Model):
email = db.EmailProperty()
password = db.StringProperty()
main.py file
from user import *
class CreateHandler(webapp.RequestHandler):
def get(self):
u = User()
u.email = "email#email.com"
u.password = "mypass"
u.put()
It gives me this error:
File "........./main.py", line 75, in get
u.put()
AttributeError: User instance has no attribute 'put'
Yes, you can access the datastore from other scripts. You don't have to add request handling to them, that can stay in your main script. Eg., you can do something like this:
app.yaml:
- url: /.*
script: main.py
main.py:
from SectionHandlers import * # This imports classes from SectionHandlers.py
application = webapp.WSGIApplication([
("/section1/.*", Section1Handler), # Map requests to handlers
("/section2/.*", Section2Handler),
], debug=True)
SectionHandlers.py:
from google.appengine.ext import db, webapp
class Section1Handler(BlogHandler):
def get(self):
# Code using 'db' here