I am trying to run a simple python 2 server code with AppEngine and Datastore. When I run dev_appserver.py app.yaml, the program immediately exits (without an error) after the following outputs:
/home/username/google-cloud-sdk/lib/third_party/google/auth/crypt/_cryptography_rsa.py:22: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
import cryptography.exceptions
INFO 2022-12-20 11:59:41,931 devappserver2.py:239] Using Cloud Datastore Emulator.
We are gradually rolling out the emulator as the default datastore implementation of dev_appserver.
If broken, you can temporarily disable it by --support_datastore_emulator=False
Read the documentation: https://cloud.google.com/appengine/docs/standard/python/tools/migrate-cloud-datastore-emulator
INFO 2022-12-20 11:59:41,936 devappserver2.py:316] Skipping SDK update check.
INFO 2022-12-20 11:59:42,332 datastore_emulator.py:156] Starting Cloud Datastore emulator at: http://localhost:22325
INFO 2022-12-20 11:59:42,981 datastore_emulator.py:162] Cloud Datastore emulator responded after 0.648865 seconds
INFO 2022-12-20 11:59:42,982 <string>:384] Starting API server at: http://localhost:38915
Ideally, it should have continued by runnning the server on port 8000. Also, it works with option --support_datastore_emulator=False.
This is the code:
import webapp2
import datetime
from google.appengine.ext import db, deferred, ndb
import uuid
from base64 import b64decode, b64encode
import logging
class Email(ndb.Model):
email = ndb.StringProperty()
class DB(webapp2.RequestHandler):
def post(self):
try:
mail = Email()
mail.email = 'Test'
mail.put()
except Exception as e:
print(e)
self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
return self.response.out.write(e)
def get(self):
try:
e1 = Email.query()
logging.critical('count is: %s' % e1.count)
e1k = e1.get(keys_only=True)
logging.critical('count 2 is: %s' % e1k.count)
e1 = e1.get()
key = unicode(e1.key.urlsafe())
logging.critical('This is a critical message: %s' % key)
logging.critical('This is a critical message: %s' % e1k)
e2 = ndb.Key(urlsafe=key).get()
self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
return self.response.out.write(str(e2.email))
except Exception as e:
print(e)
self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
return self.response.out.write(e)
app = webapp2.WSGIApplication([
('/', DB)
], debug=True)
How can I find the reason this is not working?
Edit: I figured that the dev server works and writes to a datastore even with support_datastore_emulator=False option. I am confused by this option. I also don't know where the database is stored currently.
It should be count() and not count i.e.
logging.critical('count is: %s' % e1.count())
A get returns only 1 record and so it doesn't make sense to do a count after calling a get. Besides, the count operation is a method of the query instance not the results. This means the following code is incorrect
e1k = e1.get(keys_only=True)
logging.critical('count 2 is: %s' % e1k.count)
You should replace it with
elk = e1.fetch(keys_only=True) # fetch gives an array
logging.critical('count 2 is: %s' % len(e1k))
When you first run your App, it will execute the GET part of your code and because this is the first time your App is being run, you have no record in Datastore. This means e1 = e1.get() will return None and key = unicode(e1.key.urlsafe()) will lead to an error.
You have to modify your code to first check you have a value for e1 or e2 before you attempt to use the keys.
I ran your code with dev_appserver.py and it displayed these errors for me in the logs. But I ran it with an older version of gcloud SDK (Google Cloud SDK 367.0.0). I don't know why yours exited without displaying any errors. Maybe it's due to the version...??
Separately - Don't know why you're importing db. Google moved on to ndb long ago and you don't use db in your code
The default datastore (for the older generation runtimes like Python 2) is in .config (hidden folder) > gcloud > emulators > datastore
You can also specify your own location by using the flag --datastore_path. See documentation
I'm running Rails 5.2 and the awesome Ahoy gem, with mongoid as the ORM.
In the homepage controller, I'm calling current_visit.visitor_token, which is generating the following error:
MONGODB | ds111336.mlab.com:11336 | heroku_1f375nv7.find | STARTED |
{"find"=>"ahoy_visits", "filter"=>{"visit_token"=>"4f86bf0d-f694-4a46-
b1fb-4ff48722e5c5"}, "sort"=>{"_id"=>1}, "limit"=>1,
"singleBatch"=>true}
MONGODB | ds111336.mlab.com:11336 | heroku_1f375nv7.find | SUCCEEDED |
0.048124999999999994s
Completed 500 Internal Server Error in 297ms (ActiveRecord: 202.8ms)
NoMethodError (undefined method `visitor_token' for nil:NilClass):
The ahoy_visit value stored in the cookie is b5b4fd64-ec42-4e6d-9740-e394ed4a24da, yet Ahoy seems to be searching for a different visit_token (as above).
initializers/ahoy.rb:
class Ahoy::Store < Ahoy::DatabaseStore
end
# set to true for JavaScript tracking
Ahoy.api = true
Ahoy.server_side_visits = true
If I clear the website data from my browser (or open a new session in an incognito window), the error does not occcur until I wait another day (I'm guessing until the sessions expires).
I've searched all through the code and can not figure out why Ahoy is not creating a new visit record in the DB.
Thanks for any help in advance!
With Ahoy 2.0 you visit model need to be scoped as such
class Ahoy::Visit < ApplicationRecord
# code here
end
- another walkaround is re-install Ahoy and it will place Visit and Event in the correct location.
I am trying to use HTSQL for one of my Django projects. For that I followed the procedure given HERE for furnishing HTSQL/Django requirements. Then I cloned the HTSQL repository for trying the example/demo in it from HERE. The default db used in the demo example is sqlite3.
I have tried this demo on both Django v 1.4 and Django v 1.3.1(had to make some tweaks in settings.py for Django v 1.3.1). As instructed in the HTSQL Django-gateway Blog, I wrote the following code in the django project shell:
>>> from htsql_django import produce
>>> query = "/polls_poll{question, total:=sum(polls_choice.votes)}"
>>> for row in produce(query):
>>> print "%s: %s" % (row.question, row.total)
It throws following error:
TransactionManagementError: This code isn't under transaction management
The whole error trace can be viewed at pastebin
I have also tried this on my own new project but same error.
When you use HTSQL from a Django shell, you have to open a transaction explicitly:
>>> from django.db import transaction
>>> from htsql_django import produce
>>> with transaction.commit_on_success():
... query = "/polls_poll{question, total:=sum(polls_choice.votes)}"
... for row in produce(query):
... print "%s: %s" % (row.question, row.total)
I'm sorry the documentation isn't clear about that. We may change it in a future release.
I am trying to create files in a Dropbox.com folder from a GAE application.
I have done all the steps the register a Dropbox application and installed the Python SDK from Dropbox locally on my development machine. (see dropbox.com API).
It all works perfectly when I use the cli_client.py test script in the dropbox SDK on my local machine to access dropbox - can 'put' files etc.
I now want to start working in GAE environment, so things get a bit tricky.
Some help would be useful.
For those familiar with the Dropbox API code, I had the following issues thus far:
Issue 1
The rest.py Dropbox API module uses pkg_resources to get the certs installed in site-packages of a local machine installation.
I replaced
TRUSTED_CERT_FILE = pkg_resources.resource_filename(__name__, 'trusted-certs.crt')
with
TRUSTED_CERT_FILE = file('trusted-certs.crt')
and placed the cert file in my GAE application directory. Perhaps this is not quite right; see my authentication error code below.
Issue 2
The session.py Dropbox API module uses oauth module, so I changed the include to appengine oauth.
But raised an exception that GAE's oauth does not have OAuthConsumer method used by the Dropbox session.py module. So i downloaded oauth 1.0 and added to my application an now import this instead of GAE oauth.
Issue 3
GAE ssl module does not seem to have CERT_REQUIRED property.
This is a constant, so I changed
self.cert_reqs = ssl.CERT_REQUIRED
to
self.cert_reqs = 2
This is used when calling
ssl.wrap_socket(sock, cert_reqs=self.cert_reqs, ca_certs=self.ca_certs)
Authentication Error
But I still can't connect to Dropbox:
Status: 401
Reason: Unauthorized
Body: {"error": "Authentication failed"}
Headers: [('date', 'Sun, 19 Feb 2012 15:11:12 GMT'), ('transfer-encoding', 'chunked'), ('connection', 'keep-alive'), ('content-type', 'application/json'), ('server', 'dbws')]
Here's my patched version of Dropbox Python SDK 1.4 which works well for me with Python 2.7 GAE: dropbox_python_sdk_gae_patched.7z.base64. No extra third-party libraries needed, only those provided by GAE environment.
Only file uploading (put_file) is tested. Here're setup steps:
Unpack archive to the root folder of GAE application (if main app is in the root folder). You can decode BASE64 using Base64 Encoder/Decoder: base64.exe -d dropbox_python_sdk_gae_patched.7z.base64 dropbox_python_sdk_gae_patched.7z.
Setup APP_KEY, APP_SECRET, ACCESS_TYPE, ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET. First three are configured at dropbox application creation time. Last two are obtained when granting application access to specific dropbox account, you can get them through cli_client.py (from DB Python SDK) from token_store.txt file.
Use in the code like this:
import dropbox
# ...
def DropboxUpload(path, data):
sess = dropbox.session.DropboxSession(APP_KEY, APP_SECRET, ACCESS_TYPE)
sess.set_token(ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET)
cli = dropbox.client.DropboxClient(sess)
data_file = StringIO.StringIO(data)
return cli.put_file(path, data_file)
# ...
import json
class DropboxUploadHandlerExample(webapp2.RequestHandler):
def get(self):
url = "http://www.google.com/"
result = urlfetch.fetch(url)
self.response.headers['Content-Type'] = 'application/json'
self.response.out.write(json.dumps(DropboxUpload('/fetch_result.dat', result.content)))
I successfully uploaded from Google Appengine to Dropbox with my own patched version
of the Dropbox SDK: https://github.com/cklein/dropbox-client-python
The usage of urllib2 was replaced by huTools.http: https://github.com/hudora/huTools/
This is the code that is called in a request handler:
db_client = dropbox.get_dropbox_client(consumer_key='', consumer_secret='', access_token_key='', access_token_secret='')
fileobj = StringIO.StringIO(data)
path = '/some/path/filename'
resp = db_client.put_file(path, fileobj)
fileobj.close()
As of April 2016, none of the other suggestions work. (Dropbox API version 2, Python SDK version 6.2).
If you only need a few of the SDK functions, I found it easiest to just use the HTTP API directly:
def files_upload(f, path, mode='add', autorename=False, mute=False):
args = {
'path': path,
'mode': mode,
'autorename': autorename,
'mute': mute,
}
headers = {
'Authorization': 'Bearer {}'.format(ACCESS_TOKEN),
'Dropbox-API-Arg': json.dumps(args),
'Content-Type': 'application/octet-stream',
}
request = urllib2.Request('https://content.dropboxapi.com/2/files/upload', f, headers=headers)
r = urllib2.urlopen(request)
I have patched the Dropbox Python SDK version 2.2 to work on Google App Engine. Please find the relevant code here:
https://github.com/duncanhawthorne/gae-dropbox-python
The relevant code patch (copied from github) for rest.py is here:
import io
import pkg_resources
-import socket
+#import socket
import ssl
import sys
import urllib
+import urllib2
+def mock_urlopen(method,url,body,headers,preload_content):
+ request = urllib2.Request(url, body, headers=headers)
+ r = urllib2.urlopen(request)
+ return r
+
try:
import json
except ImportError:
## -23,7 +29,10 ##
SDK_VERSION = "2.2.0"
-TRUSTED_CERT_FILE = pkg_resources.resource_filename(__name__, 'trusted-certs.crt')
+try:
+ TRUSTED_CERT_FILE = pkg_resources.resource_filename(__name__, 'trusted-certs.crt')
+except:
+ TRUSTED_CERT_FILE = file('trusted-certs.crt')
class RESTResponse(io.IOBase):
## -125,6 +134,7 ## def flush(self):
pass
def create_connection(address):
+ return
host, port = address
err = None
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
## -152,7 +162,7 ## def json_loadb(data):
class RESTClientObject(object):
- def __init__(self, max_reusable_connections=8, mock_urlopen=None):
+ def __init__(self, max_reusable_connections=8, mock_urlopen=mock_urlopen):
"""
Parameters
max_reusable_connections
## -206,7 +216,7 ## def request(self, method, url, post_params=None, body=None, headers=None, raw_re
raise ValueError("headers should not contain newlines (%s: %s)" %
(key, value))
- try:
+ if True:
# Grab a connection from the pool to make the request.
# We return it to the pool when caller close() the response
urlopen = self.mock_urlopen if self.mock_urlopen else self.pool_manager.urlopen
## -217,14 +227,14 ## def request(self, method, url, post_params=None, body=None, headers=None, raw_re
headers=headers,
preload_content=False
)
- r = RESTResponse(r) # wrap up the urllib3 response before proceeding
- except socket.error as e:
- raise RESTSocketError(url, e)
- except urllib3.exceptions.SSLError as e:
- raise RESTSocketError(url, "SSL certificate error: %s" % e)
+ #r = RESTResponse(r) # wrap up the urllib3 response before proceeding
+ #except socket.error as e:
+ # raise RESTSocketError(url, e)
+ #except urllib3.exceptions.SSLError as e:
+ # raise RESTSocketError(url, "SSL certificate error: %s" % e)
- if r.status not in (200, 206):
- raise ErrorResponse(r, r.read())
+ #if r.status not in (200, 206):
+ # raise ErrorResponse(r, r.read())
return self.process_response(r, raw_response)
## -321,10 +331,11 ## def PUT(cls, *n, **kw):
return cls.IMPL.PUT(*n, **kw)
-class RESTSocketError(socket.error):
+class RESTSocketError():
"""A light wrapper for ``socket.error`` that adds some more information."""
def __init__(self, host, e):
+ return
msg = "Error connecting to \"%s\": %s" % (host, str(e))
socket.error.__init__(self, msg)
I'm building an GAE app with web2py and am struggling with setting up a test framework.
I've looked into:
web2py_utils
nosegae
Googler Ikai Lan's suggestions
gaetestbed which was merged into GAE apis
Here's a unit test I attempted:
import unittest
import nose
from nose.tools import *
from google.appengine.ext import testbed
from google.appengine.datastore import datastore_stub_itil
class UserModelTest(unittest.TestCase):
def setUp(self):
# First, create an instance of the Testbed class.
self.testbed = testbed.Testbed()
# Then activate the testbed, which prepares the service stubs for use.
self.testbed.activate()
# Initialize the datastore stub with this policy.
self.testbed.init_datastore_v3_stub(consistency_policy=self.policy)
self.env = new_env(app='fb', controller='default')
self.db = copy_db(self.env, db_name='db', db_link='sqlite:memory')
self.user = self.db.auth_user.insert(first_name='Bob', last_name='Narley', fb_id=1, password='testtest')
self.dream = self.db.dream.insert(title='Graduate UC Santa Cruz with Computer Science degree by June 8, 2012.',
type='education', owner = self.user, progress=92.0)
self.task1 = self.db.task.insert(dream=self.dream, title='Buy batteries for calculator', solution='Go to Walmart at 12:00pm October 30, 2012 and buy Duracell AAA.',
status=1)
self.task2 = self.db.task.insert(dream=self.dream, title='Make Winston happy',
solution='Make banana milk',
status=0)
self.user.update_record(tasks=[self.task1, self.task2])
def tearDown(self):
self.testbed.deactivate()
def test_create_user(self):
assert_equal('Bob', self.user.first_name)
assert_equal(1, self.user.fb_id)
assert_equal('testtest', self.user.password)
def test_user_has_many_tasks(self):
tasks = self.db(self.db.task.id.belongs(self.user.tasks)).select()
assert_equal(2, len(tasks))
run_fb_tests.py:
from web2py_utils.test_runner import run
import sys
run(path = sys.path[0],
app = 'fb',
test_key = 'superSecret',
test_options = {'verbosity': 3, 'with-gae': True, 'without-sandbox': True},
coverage_report = 'logs/coverage_report.txt',
DO_COVER = True,
DO_NOSE = True,)
I get the following error when execute my run_fb_tests.py
ImportError: No module named google.appengine.ext
I've been banging my head for days now with many errors. This is only one of many
How do you setup a testing framework in web2py for GAE apps?
Where is your script run_fb_tests.py? If it is in the web2py folder I get a similar error, but if you place it in google_appenine, or in the containing folder of the google_appengine then I get no import errors.