I am using Go + Google AppEngine, and it seems like I have caused my datastore to become corrupted. I ran a query with Filter("ContentId !=", val), and apparently that is not a supported operation. Moreover, it corrupted my database! When I go to localhost:8000/datastore, I get a long stack trace that ends with:
File "/Users/me/go_appengine/google/appengine/api/yaml_listener.py", line 177, in _HandleEvents
raise yaml_errors.EventError(e, event_object)
EventError: could not determine a constructor for the tag '!ContentId'
in "<string>", line 15, column 11:
- name: !ContentId
^
in "<string>", line 15, column 11:
- name: !ContentId
It looks like it made a tag labeled !ContentId.
My question is, how do I reset the datastore to remove this corrupted dev_appserver.py --clear_datastore=yes myapp, but that does not fix it.
To remove the tag edit your index.yaml file in your project root and deploy it again.
!= is not in the list of supported property filters for golang.
If you'd provide your full query and some context we may come up with a good workaround.
Related
I'm having trouble deploying a flask app to the Google app engine. It works fine when I run it locally but as soon as it's deployed to the app engine it fails. With two exceptions (I'm always getting both, I'm unsure which one comes first).
Traceback (most recent call last):
File "/opt/python3.7/lib/python3.7/logging/handlers.py", line 1008, in emit
smtp = smtplib.SMTP(self.mailhost, port, timeout=self.timeout)
File "/opt/python3.7/lib/python3.7/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/opt/python3.7/lib/python3.7/smtplib.py", line 338, in connect
(code, msg) = self.getreply()
File "/opt/python3.7/lib/python3.7/smtplib.py", line 391, in getreply
+ str(e))
smtplib.SMTPServerDisconnected: Connection unexpectedly closed: timed out
and
Traceback (most recent call last):
File "/opt/python3.7/lib/python3.7/smtplib.py", line 387, in getreply
line = self.file.readline(_MAXLINE + 1)
File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
It consistently happens within the __init__ function of a form. And specifically in there when I try to append a validator to a validators list.
self.password.validators.append(InputRequired(message='Please enter password!'))
It is not specific to this one validator, the same thing happens with others too. I am at a complete loss as to why and how appending validators would interact with SMTP or sockets.
One thing I noticed is that I had an issue with the Email validator which also worked fine locally but required me to install the module 'email_validator' to work. When I did that it complained that it couldn't use append on a tuple when I tried to append a validator to - what was a list in my local app - but for some reason turned out to be a tuple when I ran the exact same code on the app engine. I removed the Email validator (and the import/requirements) entirely and now I'm at the problem I've described above. It may or may not be related to the Email validator issue.
Update: The error message does indeed seem to be related to the validators.append() call seeing as by setting up the validators at object creation and avoiding all further changes via validators.append() fixed the problem for me. This is the requirements.txt that goes with the application:
blinker>=1.4
Click>=7.0
Flask>=1.1.1
Flask-Login>=0.5.0
Flask-Mail>=0.9.1
Flask-SQLAlchemy>=2.4.1
Flask-WTF>=0.14.3
itsdangerous>=1.1.0
Jinja2>=2.11.1
MarkupSafe>=1.1.1
PyJWT>=1.7.1
PyMySQL>=0.9.3
python-dotenv>=0.11.0
SQLAlchemy>=1.3.13
Werkzeug>=1.0.0
WTForms>=2.2.1
Instead of a full minimal, reproducible example I can offer a minimized version of the form that was causing the issues when instanciated on the GAE:
class ExampleForm(FlaskForm):
name = StringField('Name')
original_name = HiddenField('Original Name')
def add_validators(self):
names = [a.name for a in Users.query.all()]
if self.original_name.data:
names.remove(self.original_name.data)
self.name.validators.append(InputRequired(message='Please enter name!'))
self.name.validators.append(NoneOf(names, message='Username already exists!'))
def __init__(self):
super().__init__(*args, **kwargs)
self.add_validators()
As I'm writing this it occurs to me that maybe I need to always initialize an empty validators list on creation to ensure new entries can be appended to it (like with a normal list) like this name = StringField('Name', validators=[]). I'd still be confused as to why it works locally, but if it fixes the problem that'd be what really counts for me.
After updating to GAE 1.8.8 and the GCS Client to r127 my code that displays images no longer works using the Development Server or Production. Code that worked previously now throughs the following error in development:
Blob with key 'encoded_gs_file:blahblahblah' does not exist
Note blahblahblah is just my shorthand for a long string.
Below is how I generate my url. This has always worked in the past.
bkey = blobstore.create_gs_key('/gs/mybucket/myfile.jpg')
url = images.get_serving_url(bkey, size=170, crop=True, secure_url=True)
Below is a stack trace from the development server:
ERROR 2013-11-21 20:35:31,767 images_stub.py:405] Blob with key 'encoded_gs_file:c2l0ZS1hdWRpdC1waW5uYWNsZWNyZS9CYXR0ZXJ5L3Bob3Rvcy8yMDEzMTEyMTAwMTEyOUBhNzllYmQwNC05YWY3LTRhMWItYTg3My1mZGIwYjI5MjNjNzguanBn' does not exist
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/images/images_stub.py", line 401, in _OpenBlob
blobstore_stub.BlobstoreServiceStub.ToDatastoreBlobKey(blob_key))
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 651, in Get
return GetAsync(keys, **kwargs).get_result()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1541, in __get_hook
entities = extra_hook(entities)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 620, in local_extra_hook
raise datastore_errors.EntityNotFoundError()
EntityNotFoundError
ApplicationError(1,)
Reading the release notes for GAE 1.8.8 I cannot see anything else I need to change in my code. Any pointers as to what I am doing wrong would be great.
When I do a filter on a ForeignKey field with __isnull=True, this exception is raised:
DatabaseError: This query is not supported by the database.
However, __isnull=False on ForeignKey works as long as there are no other inequality filters (which I would expect). And __isnull=True works for other field types.
So why does __isnull=True not work on ForeignKey? It seems that DBIndexer tries to make it work as shown here:
https://github.com/django-nonrel/django-dbindexer/blob/dbindexer-1.4/dbindexer/backends.py
But then there is an exception in djangotoolbox:
File "/Users//Documents/workspace/-gae-dev/src/django/db/models/query.py", line 107, in _result_iter
self._fill_cache()
File "/Users//Documents/workspace/-gae-dev/src/django/db/models/query.py", line 774, in _fill_cache
self._result_cache.append(self._iter.next())
File "/Users//Documents/workspace/-gae-dev/src/django/db/models/query.py", line 275, in iterator
for row in compiler.results_iter():
File "/Users//Documents/workspace/-gae-dev/src/djangotoolbox/db/basecompiler.py", line 337, in results_iter
results = self.build_query(fields).fetch(
File "/Users//Documents/workspace/-gae-dev/src/djangotoolbox/db/basecompiler.py", line 428, in build_query
self.check_query()
File "/Users//Documents/workspace/-gae-dev/src/djangotoolbox/db/basecompiler.py", line 409, in check_query
raise DatabaseError("This query is not supported by the database.")
I did come across the following commented-out test case in djangoappengine, and am wondering if is referring to the same issue?
def test_is_null(self):
self.assertEquals(FieldsWithOptionsModel.objects.filter(
floating_point__isnull=True).count(), 0)
FieldsWithOptionsModel(
integer=5.4, email='shinra.tensai#sixpaths.com',
time=datetime.datetime.now().time()).save()
self.assertEquals(FieldsWithOptionsModel.objects.filter(
floating_point__isnull=True).count(), 1)
# XXX: These filters will not work because of a Django bug.
# self.assertEquals(FieldsWithOptionsModel.objects.filter(
# foreign_key=None).count(), 1)
# (it uses left outer joins if checked against isnull)
# self.assertEquals(FieldsWithOptionsModel.objects.filter(
# foreign_key__isnull=True).count(), 1)
Alex Burgel on the NonRel project set me straight:
The NonRel/dbindexer project fixes this query (which otherwise doesn't work due to this Django bug: https://code.djangoproject.com/ticket/10790). To set up dbindexer:
of course, add it to INSTALLED_APPS
also in settings.py, set DATABASES['default]['ENGINE']= 'dbindexer'
also in settings.py, set DBINDEXER_BACKENDS to use FKNullFix. For example:
DBINDEXER_BACKENDS = (
'dbindexer.backends.BaseResolver',
'dbindexer.backends.FKNullFix',
'dbindexer.backends.InMemoryJOINResolver',
'dbindexer.backends.ConstantFieldJOINResolver',
)
I have been experimenting with Trigger.io and was successfully using it last week.
However, I have been trying new stuff out today, and have been unable to open the Trigger Toolkit properly. I'm not sure why - maybe I deleted something by accident.
The error returned was as follows:
Error in remote call to app.list: Expecting property name: line 25 column 2 (char 482) Details Close
Traceback (most recent call last):
File "/Users/josholdham/Library/Trigger Toolkit/build-tools/forge/async.py", line 96, in run
result = self._target(*self._args, **self._kwargs)
File "/Users/josholdham/Library/Trigger Toolkit/trigger/api/app.py", line 24, in list
return forge_tool.singleton.list_local_apps()
File "/Users/josholdham/Library/Trigger Toolkit/trigger/forge_tool.py", line 175, in list_local_apps
app_config = self._app_config_for_path(path)
File "/Users/josholdham/Library/Trigger Toolkit/trigger/forge_tool.py", line 147, in _app_config_for_path
app_config = json.load(app_config_file)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 278, in load
**kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
def raw_decode(self, s, idx=0):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
ValueError: Expecting property name: line 25 column 2 (char 482)
I have tried uninstalling and reinstalling the toolkit a few times, but no joy.
Any help/ advice much appreciated.
Thanks
Josh
So I figured this out, I was being a bit stupid:
I had to access hidden files on my Mac to find the Users/josholdham/Library/Trigger Toolkit folder, which I then deleted.
Launching Toolkit again then worked fine.
This is how we find hidden files on the Mac, in case it isn't clear.
http://www.mikesel.info/show-hidden-files-mac-os-x-10-7-lion/
Hope that helps some people - thanks to all previous helpers for guidance.
That looks like src/config.json is malformed in one of your apps.
I'd recommend looking at src/config.json in your apps' directories and checking the JSON syntax around line 25.
In the future, you can avoid these problems by using the App Config tab in the Toolkit to change the config, rather than editing src/config.json directly.
I'm getting this error... "ImportError: Could not find 'input_readers' on path 'map reduce'" when trying to Run my map reduce job via the http://localhost:8080/mapreduce launcher page.
It looks like my problem is similar to this post, AppEngine mapper API import error. Unfortunately, no definitive answers were given.
I've simplified it down to this tiny testmapreduce.py:
from google.appengine.ext import db
class TestEntity(db.Model):
value = db.StringProperty()
def mapperhandler(test):
print test.value
return
And my mapreduce.yaml:
mapreduce:
- name: Simplest MapReduce
mapper:
handler: testmapreduce.mapperhandler
input_reader: mapreduce.input_readers.DatastoreInputReader
params:
- name: entity_kind
default: testmapreduce.TestEntity
One possible clue is the presence of __init__.py has no effect (whether in the project root, the mapreduce directory, or both). I'm sure I'm making a beginner mistake, but over the last couple of days I have read every bit of documentation, and all the examples I can find. Thanks.
UPDATE:
I get the same error trying to invoke it via...
control.start_map(
"Give awards",
"testmapreduce.mapperhandler",
"mapreduce.input_readers.DatastoreInputReader",
{"entity_kind": "testmapreduce.TestEntity"},
shard_count=10)
UPDATE:
As requested, the stack trace -- let me know what else would be helpful...
ERROR 2011-10-16 17:09:27,216 _webapp25.py:464] Could not find 'input_readers' on path 'mapreduce'
Traceback (most recent call last):
File "/Develop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__
handler.get(*groups)
File "/Users/lc/PycharmProjects/mrtest/testmapreduce.py", line 22, in get
shard_count=10) # handle web form test interface
File "/Users/lc/PycharmProjects/mrtest/mapreduce/control.py", line 94, in start_map
transactional=transactional)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/handlers.py", line 811, in _start_map
mapper_input_reader_class = mapper_spec.input_reader_class()
File "/Users/lc/PycharmProjects/mrtest/mapreduce/model.py", line 393, in input_reader_class
return util.for_name(self.input_reader_spec)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/util.py", line 94, in for_name
module = for_name(module_name, recursive=True)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/util.py", line 102, in for_name
short_name, module_name))
ImportError: Could not find 'input_readers' on path 'mapreduce'
INFO 2011-10-16 22:09:27,253 dev_appserver.py:4247] "GET /giveawards HTTP/1.1" 500 -
This problem turned out to be that I was using the 2.7 version of the Python Interpreter in my local environment. When I switched to 2.5, it works fine.