Retrieving data from a datastore in GAE - google-app-engine

I 'm trying to retrieve data from a datastore by using this
(in example)
class Book(db.Model):
title = db.StringProperty()
author = db.StringProperty()
category = db.StringProperty()
e = Book(title ="LOTR",
author = "JRR Tolkien",
category = "Adventure")
class Result(webapp.RequestHandler):
Book = db.GqlQuery("SELECT * FROM Book")
for book in Book:
self.responce.out.write('<b>%s</b>' % book.title)
Which this solution is from a google example, but have this result
Traceback (most recent call last):
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 3858, in _HandleRequest
self._Dispatch(dispatcher, self.rfile, outfile, env_dict)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 3792, in _Dispatch
base_env_dict=env_dict)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 580, in Dispatch
base_env_dict=base_env_dict)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 2918, in Dispatch
self._module_dict)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 2822, in ExecuteCGI
reset_modules = exec_script(handler_path, cgi_path, hook)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 2697, in ExecuteOrImportScript
handler_path, cgi_path, import_hook)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 2621, in LoadTargetModule
module_code = compile(source_file.read(), cgi_path, 'exec')
File "/home/kostas89/library/library.py", line 77
for book in Book:
^
IndentationError: unexpected indent
I also tried the fetch() method without any result..
EDIT:
Yes but when i delete the indent, i have an error for the 'self'.
Traceback (most recent call last):
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 3858, in _HandleRequest
self._Dispatch(dispatcher, self.rfile, outfile, env_dict)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 3792, in _Dispatch
base_env_dict=env_dict)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 580, in Dispatch
base_env_dict=base_env_dict)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 2918, in Dispatch
self._module_dict)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 2822, in ExecuteCGI
reset_modules = exec_script(handler_path, cgi_path, hook)
File "/home/kostas89/google_appengine/google/appengine/tools/dev_appserver.py", line 2702, in ExecuteOrImportScript
exec module_code in script_module.__dict__
File "/home/kostas89/library/library.py", line 74, in <module>
class Result(webapp.RequestHandler):
File "/home/kostas89/library/library.py", line 79, in Result
self.responce.out.write('<b>%s</b>' % book.title)
NameError: name 'self' is not defined
I found that propably is because of the tabs/sapces and I try some solutions but still not working.

The properties inside your Book class should be indented. Your entity is being instanciated outside of a handler method and not being saved. Your GqlQuery is inside a handler but not inside a method, it isn't indented correctly, it re-assigns the variable you used for your model class, your for statement isn't indented correctly, and you misspelled response.
No offense, but you might want to back up and go through the App Engine Getting Started tutorial. Once you've completed those examples, make small, incremental changes and verify that the code still runs after each one.

Look at the error that's reported - it's telling you the right thing:
IndentationError: unexpected indent
Remove the indentation, to produce:
Book = db.GqlQuery("SELECT * FROM Book")
for book in Book:
self.responce.out.write('<b>%s</b>' % book.title)

You haven't put your code inside a method. It should look something like this:
class Result(webapp.RequestHandler):
def get(self):
Book = db.GqlQuery("SELECT * FROM Book")
for book in Book:
self.response.out.write('<b>%s</b>' % book.title)
Also:
Your indentation in the question doesn't seem to match what you're actually using - it's important to get this right in a language like Python, where indentation matters.
You misspelled 'response'.
This won't return any results because you haven't called .put() on the Book entity you created.
I would strongly recommend working your way through the App Engine tutorial as Drew suggests.

If want you can try reindent.py that makes sure indentation is correct or a lint tool such as pylint. I found reindent very useful and both reindent and pylint and worth trying.

I'd remove the indent:
Book = db.GqlQuery("SELECT * FROM Book")
for book in Book:
self.responce.out.write('<b>%s</b>' % book.title)
as the error suggests and see what happens. Most likely it is as is being reported:
File "/home/kostas89/library/library.py", line 77
for book in Book:
^ IndentationError: unexpected indent
indentation is significant in python code. But white space is not necessarily.

Related

django Model one-to-many save fail

I am starting developping an app that uses django model ORM. I came upon a strange behavior when creating and saving objects related by a one-to-many relation. I have made two model creation test cases of a simple one-to-many relationship, one is working and the other is failing, but I don't understand why. Here are my models :
class Document(models.Model):
pass
class Section(models.Model):
document = models.ForeignKey('Document',on_delete=models.CASCADE)
Here is the working creation test case (in manage.py shell):
>>> doc = models.Document()
>>> doc.save()
>>> section = models.Section(document=doc)
>>> section.document
<Document: Document object (5)>
>>> section.save()
>>>
And here is the test case that fails:
>>> doc = models.Document()
>>> section = models.Section(document=doc)
>>> section.document
<Document: Document object (None)>
>>> doc.save()
>>> section.document
<Document: Document object (6)>
>>> section.save()
Traceback (most recent call last):
File "<input>", line 1, in <module>
section.save()
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/models/base.py", line 741, in save
force_update=force_update, update_fields=update_fields)
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/models/base.py", line 779, in save_base
force_update, using, update_fields,
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/models/base.py", line 870, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/models/base.py", line 908, in _do_insert
using=using, raw=raw)
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1335, in execute_sql
cursor.execute(sql, params)
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 99, in execute
return super().execute(sql, params)
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 76, in execute
raise utils.IntegrityError(*tuple(e.args))
django.db.utils.IntegrityError: (1048, "Column 'document_id' cannot be null")
>>> section.document
Traceback (most recent call last):
File "<input>", line 1, in <module>
section.document
File "/home/philippe/.local/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 189, in __get__
"%s has no %s." % (self.field.model.__name__, self.field.name)
diagnosisrank.models.common.Section.document.RelatedObjectDoesNotExist: Section has no document.
>>>
The only difference with the working test case is that the related Document model is not saved before instantiating the Section model. However, after saving the Document model, we see that the related document model of the Section model is pointing to the saved Document (its id is set). But when trying to save the Section, the related id is not set and the related model is lost. Why is that? Why do the related Document model instance must be assigned to the Section only after being saved?
To clarify my problem my goal is to collect all info and instantiate all my models in a first step, then saving all to database in a section step. I can still do this in this way: say D as many S, create D and S, save D, assign D in S, save S. But I would prefer to do: create D, create S with related D, save D, save S. Why can't I?
Thanks for any help or insight!
Phil
Well it was a bug and it is fixed: https://github.com/django/django/commit/519016e5f25d7c0a040015724f9920581551cab0
However it is not in the latest stable 2.2.4 that I get out of pip3 ...

django model method fails database commit

The Configuration model below allows for uploading of a .ZIP and ties the extracted network device .TXT files within to the location where the device configurations came from.
class Configuration(models.Model):
configfile = models.FileField('Configuration File Upload', upload_to='somecompany/configs/', help_text='Select a .ZIP file which contains .TXT file configuration dumps from devices which belong to a single location.')
location_name = models.ForeignKey('Location', help_text='Associate the .ZIP file selected above to the location from which the device .TXT file configuration dumps were taken.')
I extended the default SAVE model method for the class to allow for processing of the .ZIP (code not shown for brevity). I've parsed the extracted .TXT files, collected all of the desired information into variables, and I'm trying to insert that information into my database but it's failing. Specifically, below I show an example of all of the values collected from a single one of the extracted .TXT files (modified slightly for privacy) and my attempt at DB insertion:
dbadd_ln = 'Red Rock'
dbadd_dn = 'DEVICE4'
dbadd_manu = 'cisco'
dbadd_os = 'nxos'
dbadd_dt = '-'
dbadd_prot = '-'
dbadd_cred = '-'
dbadd_ser = 'ABCD1234'
dbadd_addr = '10.10.10.10'
dbadd_model = 'N7K-C7010'
dbadd_ram = '2048256000'
dbadd_flash = '1109663744/1853116416'
dbadd_image = 'n7000-s1-dk9.5.2.9.bin'
dbadd = Device(location_name=dbadd_ln, device_name=dbadd_dn, device_type=dbadd_dt, protocol=dbadd_prot, credential=dbadd_cred, serial=dbadd_ser, address=dbadd_addr, manufacturer=dbadd_manu, model=dbadd_model, ram=dbadd_ram, flash=dbadd_flash, os=dbadd_os, image=dbadd_image)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "c:\code-projects\MYVIRTUALENV\lib\site-packages\django\db\models\base.py", line 431, in __init__
setattr(self, field.name, rel_obj)
File "c:\code-projects\MYVIRTUALENV\lib\site-packages\django\db\models\fields\related_descriptors.py", line 207, in __set__
self.field.remote_field.model._meta.object_name,
ValueError: Cannot assign "'Red Rock'": "Device.location_name" must be a "Location" instance.
'Red Rock' is a legitimate Location entry which already exists in my database...
>>> Location.objects.filter(location_name='Red Rock')
[Location: Red Rock]
... so I guess I'm unclear on what this really means:
"Device.location_name" must be a "Location" instance.
Any assistance to help resolve this issue is appreciated. Thanks in advance.
Did some more searching and found this:
Cannot assign "u''": "Company.parent" must be a "Company" instance
Now I see. When I create an instance, I get past the particular error I was describing. Now on to solve the next problem. :)

How to handle large files while reading it with python xlrd in GAE without giving DeadlineExceededError

I want to read a file having size 4 MB using python xlrd in GAE.
i am getting the file from Blobstore. Code used is given below.
book = xlrd.open_workbook(file_contents=temp_file)
sh = book.sheet_by_index(0)
for col_no in range(sh.ncols):
its gives me DeadlineExceededError.
book = xlrd.open_workbook(file_contents=file_data)
File "/base/data/home/apps/s~appid/app-version.369475363369053908/xlrd/__init__.py", line 416, in open_workbook
ragged_rows=ragged_rows,
File "/base/data/home/apps/s~appid/app-version.369475363369053908/xlrd/xlsx.py", line 756, in open_workbook_2007_xml
x12sheet.process_stream(zflo, heading)
File "/base/data/home/apps/s~appid/app-version.369475363369053908/xlrd/xlsx.py", line 520, in own_process_stream
for event, elem in ET.iterparse(stream):
DeadlineExceededError
But i am able to read files with smaller size.
Actually i need to get only first few rows(30 to 50) of the file. Is there any other method, other than adding it as a task and getting the details using channel API to get the details with out causing deadline error ?
What i can do to handle this....?
I read a file about 1000 rows excel and it works okay the library.
I leave a link that might be useful https://github.com/cjhendrix/HXLator-SpaceAppsVersion/blob/master/gae/main.py
the code I see that this crossing of columns and rows must be at lists for each row
example:
wb = xlrd.open_workbook(file_contents=inputfile.read())
sh = wb.sheet_by_index(0)
for rownum in range(sh.nrows):
val_row = sh.row_values(rownum)
#here print element of list
self.response.write(val_row[1]) #depending for number for columns
regards!!!

Cannot get records back from Google App Engine

I have harvested data, its not particularly clean data, and this has been bulk uploaded into the DataStore. However I am getting the following issue when trying to simply loop through all the records. I don't much care about validation at this point as all I want to do is perform a bulk operation but GAE appears not to let me even loop through the data records. I want to get at the bottom of this. To my knowledge all records have data in the field for the country and I could switch of validation, but can someone explain why this is happening and GAE is being sensitive. Thanks
result = Company.all()
my_count = result.count()
if result:
for r in result:
self.response.out.write("hello")
The data model has these properties:
class Company(db.Model):
companyurl = db.LinkProperty(required=True)
companyname = db.StringProperty(required=True)
companydesc = db.TextProperty(required=True)
companyaddress = db.PostalAddressProperty(required=False)
companypostcode = db.StringProperty(required=False)
companyemail = db.EmailProperty(required=True)
companycountry = db.StringProperty(required=True)
The error message is below
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__
handler.get(*groups)
File "/base/data/home/apps/XXX/1.358667163009710608/showcompanies.py", line 99, in get
for r in result:
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 2312, in next
return self.__model_class.from_entity(self.__iterator.next())
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1441, in from_entity
return cls(None, _from_entity=entity, **entity_values)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 973, in __init__
prop.__set__(self, value)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 613, in __set__
value = self.validate(value)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 2815, in validate
value = super(StringProperty, self).validate(value)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 640, in validate
raise BadValueError('Property %s is required' % self.name)
BadValueError: Property companycountry is required
If you have the bulk process you wish to run in its own script, you can construct a modified version of your Company class without validation. Since db.Model classes are just wrappers to the datastore based on the name of the class, you can have different classes in different parts of your code with different behaviors.
So you might have a model.py file with:
class Company(db.Model):
companyurl = db.LinkProperty(required=True)
# ...
companycountry = db.StringProperty(required=True)
# Normal operations go here
And, another bulk_process.py file with:
class Company(db.Model):
companyurl = db.LinkProperty()
# ...
companycountry = db.StringProperty()
result = Company.all()
my_count = result.count()
if result:
for r in result:
self.response.out.write("hello")
Because this second model class lacks the validation, it should run just fine. And, because the code is logically separated you don't have to worry about unintentional side-effects from removing the validation in the rest of your code. Just make sure that your bulk process doesn't accidentally write back data without validation (unless you're OK with this).

Migrating an App to High-Replication Datastore - now ancestor issues

Following these instructions:
http://neogregious.blogspot.com/2011/04/migrating-app-to-high-replication.html
I have managed to migrate to the high replication datastore however I am now getting the following exception:
datastore_errors.BadArgumentError('ancestor argument should match app ("%r" != "%r")' %
(ancestor.app(), app))
The model data looks something like this:
class base_business(polymodel.PolyModel):
created = db.DateTimeProperty(auto_now_add=True)
class business(base_business):
some_data = db.StringProperty()
etc..
class business_image(db.Model):
image = db.BlobProperty(default=None)
mimetype = db.StringProperty()
comment = db.StringProperty(required=False)
# the image is associated like so
image_item = business_image(parent = business_item, etc... )
image_item.put()
The new app name has not been assigned to the ancestor model data. At the moment the data is returned however the logs are being populated with this exception message.
The actual stack trace using logging.exception:
2011-11-03 16:45:40.211
======= get_business_image exception [ancestor argument should match app ("'oldappname'" != "'s~newappname'")] =======
Traceback (most recent call last):
File "/base/data/home/apps/s~newappname/3.354412961756003398/oldappname/entities/views.py", line 82, in get_business_image
business_img = business_image.gql("WHERE ANCESTOR IS :ref_business and is_primary = True", ref_business = db.Key(business_key)).get()
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/init.py", line 2049, in get
results = self.fetch(1, config=config)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/init.py", line 2102, in fetch
raw = raw_query.Get(limit, offset, config=config)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 1668, in Get
config=config, limit=limit, offset=offset, prefetch_size=limit))
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 1600, in GetBatcher
return self.GetQuery().run(_GetConnection(), query_options)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 1507, in GetQuery
order=self.GetOrder())
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 93, in positional_wrapper
return wrapped(*args, **kwds)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_query.py", line 1722, in init
ancestor=ancestor)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 93, in positional_wrapper
return wrapped(*args, **kwds)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_query.py", line 1561, in init
(ancestor.app(), app))
BadArgumentError: ancestor argument should match app ("'oldappname'" != "'s~newappname'")
Is there a way to manually set the app on the model data? Could I do something like this to resolve this?
if( ancestor.app() != app )
set_app('my_app')
put()
Before I do this or apply any other HACK is there something I should have done as part of the data migration?
This sort of error usually occurs because you're using fully qualified keys somewhere that have been stored in the datastore as strings (instead of ReferenceProperty), or outside the datastore, such as in URLs. You should be able to work around this by reconstructing any keys from external sources such that you ignore the App ID, something like this:
my_key = db.Key.from_path(*db.Key(my_key).to_path())

Resources