In Google AppEngine(GAE) environment, I can do following to convert a Protobuf bytestring back to a Datastore model:
from google.appengine.ext import db
byte_str = ....
model = db.model_from_protobuf(byte_str.decode("base64"))
Outside of GAE, I normally use the google-cloud-datastore client to access Datastore models:
from google.cloud import datastore
...
client = datastore.Client(project_id)
query = client.query(kind='Event', order=('-date',))
for result in query.fetch(limit=100):
print result
# every result is of class `google.cloud.datastore.entity.Entity`
Question: What if I'm only given the encoded byte string? How can I convert it back to a google.cloud.datastore.entity.Entity (or dict)?
Followups:
Update1:
google.cloud.proto.datastore.v1.entity_pb2.Entity.ParseFromString() is the closest I found so far. But not quite working yet..
## fetched a protobuf into `pb`
>>> pb.__class__
<class 'google.cloud.proto.datastore.v1.entity_pb2.Entity'>
>>> entity = google.cloud.datastore.helpers.entity_from_protobuf(pb)
>>> entity.__class__
<class 'google.cloud.datastore.entity.Entity'>
>>> serialized = pb.SerializeToString() ## <-- now this is the bytestring I meant.
>>> type(serialized)
<type 'str'>
>>> google.cloud.proto.datastore.v1.entity_pb2.Entity.ParseFromString(serialized)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: descriptor 'ParseFromString' requires a 'google.protobuf.pyext._message.CMessage' object but received a 'str'
>>>google.cloud.proto.datastore.v1.entity_pb2.Entity().ParseFromString(serialized)
76942 ## <--??
Update2:
Actually had to do this:
e = google.cloud.proto.datastore.v1.entity_pb2.Entity()
e.ParseFromString(byte_str)
print e
And that did it..
this worked:
e = google.cloud.proto.datastore.v1.entity_pb2.Entity()
e.ParseFromString(byte_str)
print e
Related
I'm using pydicom (installed with pip3, on python 3.7, using Idle) and I need to access pixel_array values.
I just copy-paste the example provided into the documentation and this leads to two errors:
first is about the get_testdata_files operation, which is not working because
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>>
==================== RESTART: D:\OneDrive\Desktop\test.py ====================
None
Traceback (most recent call last):
File "D:\OneDrive\Desktop\test.py", line 8, in <module>
filename = get_testdata_files("bmode.dcm")[0]
IndexError: list index out of range
I have solved this not using this operation.
second is about the pixel_array and I'm not so able to decode what is wrong, but it seems like the pixel_array is not populated. However I'm able to access other fields in the dataset and the file can be displayed (using ImageJ for example).
==================== RESTART: D:\OneDrive\Desktop\test.py ====================
None
Filename.........: bmode.dcm
Storage type.....: 1.2.840.10008.5.1.4.1.1.3.1
Patient's name...: Femoral trombenarterectomy, Case Report:
Patient id.......: Case Report 1
Modality.........: US
Study Date.......: 20110824
Image size.......: 768 x 1024, 27472108 bytes
Slice location...: (missing)
Traceback (most recent call last):
File "D:\OneDrive\Desktop\test.py", line 38, in <module>
plt.imshow(dataset.pixel_array, cmap=plt.cm.bone)
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\dataset.py", line 949, in pixel_array
self.convert_pixel_data()
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\dataset.py", line 895, in convert_pixel_data
raise last_exception
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\dataset.py", line 863, in convert_pixel_data
arr = handler.get_pixeldata(self)
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\pixel_data_handlers\pillow_handler.py", line 188, in get_pixeldata
UncompressedPixelData.extend(decompressed_image.tobytes())
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\Image.py", line 746, in tobytes
self.load()
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\ImageFile.py", line 261, in load
raise_ioerror(err_code)
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\ImageFile.py", line 58, in raise_ioerror
raise IOError(message + " when reading image file")
OSError: broken data stream when reading image file
Here is my code:
import matplotlib.pyplot as plt
import sys
import pydicom
import numpy
from pydicom.data import get_testdata_files
print(__doc__)
#filename = get_testdata_files("bmode.dcm")[0]
filename = "bmode.dcm"
dataset = pydicom.dcmread(filename)
# Normal mode:
print()
print("Filename.........:", filename)
print("Storage type.....:", dataset.SOPClassUID)
print()
pat_name = dataset.PatientName
display_name = pat_name.family_name + ", " + pat_name.given_name
print("Patient's name...:", display_name)
print("Patient id.......:", dataset.PatientID)
print("Modality.........:", dataset.Modality)
print("Study Date.......:", dataset.StudyDate)
if 'PixelData' in dataset:
rows = int(dataset.Rows)
cols = int(dataset.Columns)
print("Image size.......: {rows:d} x {cols:d}, {size:d} bytes".format(
rows=rows, cols=cols, size=len(dataset.PixelData)))
if 'PixelSpacing' in dataset:
print("Pixel spacing....:", dataset.PixelSpacing)
# use .get() if not sure the item exists, and want a default value if missing
print("Slice location...:", dataset.get('SliceLocation', "(missing)"))
# plot the image using matplotlib
plt.imshow(dataset.pixel_array, cmap=plt.cm.bone)
plt.show()
Could you help me to solve these two errors and access pixel_array values?
Don't hesitate to give me some advices /remarks/...
Thanks!
Hi Marc welcome to SO!
Your first error means that the get_testdata_files returns an empty list, so your file is not found. Have a look at the pydicom source, it shows that a search is performed in [DATA_ROOT]/test_files. Is your file located in that path?
Your second error is related to PIL and that can be quite difficult to debug and fix. First try to read the pixel_array from a dataset created from one of the supplied test files. If that works, your problem is probably that PIL cannot handle the specific encoding of your image data. You want to install and use GDCM instead of PIL to see if that solves the problem. Another user has had a similar issue as you, GDCM solved the problem. It can be a bit of a headache to get working unfortunately. Or have a look at this page, it shows some other alternatives on viewing the image data.
I'm trying to deserialize some Protobuf bytestrings, in Python, outside of GAE(AppEngine) environment. I'm wondering if I need the message schema for this use case.
Approaches I've tried so far includes google.cloud.proto.datastore.v1.entity_pb2.Entity.ParseFromString():
>>> pb = ... ## fetched a protobuf into `pb` using the Datastore Protobuf API
>>> pb.__class__
<class 'google.cloud.proto.datastore.v1.entity_pb2.Entity'>
>>> serialized = pb.SerializeToString() ## <-- now `serialized` is a bytestring.
>>> type(serialized)
<type 'str'>
>>> e = google.cloud.proto.datastore.v1.entity_pb2.Entity()
>>> e.ParseFromString(serialized) ## <-- deserialization
>>> print e
The deserialization above actually worked, so I'm assuming I don't need a to know the message schema. --Correct me if I'm wrong.
However, I tried this on a different bytestring, it didn't work. But it worked with google.appengine.ext.db.model_from_protobuf(bytestring2) in GAE environment though.
Thoughts?? Thanks
Other notes
Somewhat relevant post.
I am trying to save all exceptions thrown by a python service (running hidden on Windows 7/10) into a sqlite3 database file. In addition to other things(traceback, date, etc) I want to save the error message too.
My Problem is that there are some error messages that I cannot convert to unicode (Especially some WindowsErrors and errors with a german 'Umlaut': ä, ö, ü). As I don't know every possible error in advance I want to have a function that can handle ALL errors and convert their message to unicode.
Can someone tell me what the convertToUnicode-function has to look like?
# -*- coding: utf-8 -*-
def someErrorneousFunction():
raise RuntimeError('Not a unicode string. But with a german Umlaut: ä!')
def saveExceptionToDBThatOnlyAcceptsUnicode(msg):
"""SQL INSERT SOME STUFF... + THE UNICODE ERROR MESSAGE!"""
pass
def convertToUnicode(e):
""" What to do here ??? """
pass
try:
someErrorneousFunction()
except Exception as e:
unicode_error_msg = convertToUnicode(e)
saveExceptionToDBThatOnlyAcceptsUnicode(unicode_error_msg)
Note: I found that some Exceptions have an attribute called .msg or .str, but not all of them!
Does this method make sense at all? I know its bad practice to catch ALL exceptions with no distinction but as my software occasionally is in testmode somewhere else and I want to get the exception database per mail it seemed meaningful to me. Furthermore I distinguish the errors I know from those I don't know in advance.
I would appreciate any advise!
Thank you!
Sebastian
If, as your demo shows, the source file is known to be in UTF-8, then this should work:
# -*- coding: utf-8 -*-
import traceback
def someErrorneousFunction():
raise RuntimeError('Not a unicode string. But with a german Umlaut: ä!')
def saveExceptionToDBThatOnlyAcceptsUnicode(msg):
print type(msg)
print msg
def convertToUnicode(e):
return traceback.format_exc(e).decode('utf8')
try:
someErrorneousFunction()
except Exception as e:
unicode_error_msg = convertToUnicode(e)
saveExceptionToDBThatOnlyAcceptsUnicode(unicode_error_msg)
Output:
<type 'unicode'>
Traceback (most recent call last):
File "C:\test.py", line 15, in <module>
someErrorneousFunction()
File "C:\test.py", line 5, in someErrorneousFunction
raise RuntimeError('Not a unicode string. But with a german Umlaut: ä!')
RuntimeError: Not a unicode string. But with a german Umlaut: ä!
Use the example code from app engine will give an attribute error. The more strange thing is,
When the batch_size is 100, the first fetch will give an error while if it were set to 10, the second fetch will give the error, when the batch_size is 1, the 25th fetch will give the error. Is it due to the problem of remote API?
Python version: 2.7
App engine sdk version: 1.9.6
query = MyModel.all()
entities = query.fetch(100)
while entities:
for entity in entities:
# Do something with entity
query.with_cursor(query.cursor())
entities = query.fetch(100)
error message:
Traceback (most recent call last):
File "migrate.py", line 77, in <module>
entities = query.fetch(batch_size)
File "/home/kamel/Library/google_appengine/google/appengine/ext/db/__init__.py", line 2157, in fetch
return list(self.run(limit=limit, offset=offset, **kwargs))
File "/home/kamel/Library/google_appengine/google/appengine/ext/db/__init__.py", line 2326, in next
return self.__model_class.from_entity(self.__iterator.next())
File "/home/kamel/Library/google_appengine/google/appengine/ext/db/__init__.py", line 1435, in from_entity
entity_values = cls._load_entity_values(entity)
File "/home/kamel/Library/google_appengine/google/appengine/ext/db/__init__.py", line 1413, in _load_entity_values
value = prop.make_value_from_datastore(value)
File "/home/kamel/labola/src/model/properties.py", line 295, in make_value_from_datastore
return pickle.loads(value)
File "/usr/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj
obj = cls.__new__(cls, *args)
AttributeError: class Reference has no attribute '__new__
I encountered the same issue when trying to unpickle python3 pickles under python2. The problem was linked to new-style classes becoming default in python3. (source)
Solution for me was to replace class AClass: by class AClass(object):
I read the "what is a metaclass in Python" but am still confused over it.
I am new to python and have been thrown into upgrading it from 2.5 to 2.7.
I have the following:
class UsersDB(db.Model):
Email = db.EmailProperty(required=True,verbose_name='Email *')
Enable = db.BooleanProperty(default=True)
and
class UsersQuickAddForm(forms.ModelForm):
def is_user_exist(self, account):
users_query = UsersDB.all().filter('Email =', account).fetch(1)
if len(users_query) > 0:
return True
return False
class Meta:
model = UsersDB
exclude = ['Enable']
but when I try to execute it on the google site, I get:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 239, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 298, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 84, in LoadObject
obj = import(path[0])
File "/base/data/home/apps/s~ldsdgidev/glen27.371429613087607751/LDSGH.py", line 8, in
from core.decorators import permissionRequired
File "/base/data/home/apps/s~ldsdgidev/glen27.371429613087607751/core/decorators.py", line 7, in
from core.initialization import loginIf
File "/base/data/home/apps/s~ldsdgidev/glen27.371429613087607751/core/initialization.py", line 6, in
import photo_images
File "/base/data/home/apps/s~ldsdgidev/glen27.371429613087607751/core/photo_images.py", line 1, in
from core.db_models import ImagesDB
File "/base/data/home/apps/s~ldsdgidev/glen27.371429613087607751/core/db_models.py", line 222, in
class UsersQuickAddForm(forms.ModelForm):#only account, firstname and last name is required
File "/base/data/home/apps/s~ldsdgidev/glen27.371429613087607751/django/forms/models.py", line 205, in new
opts.exclude, opts.widgets, formfield_callback)
File "/base/data/home/apps/s~ldsdgidev/glen27.371429613087607751/django/forms/models.py", line 145, in fields_for_model
opts = model._meta
AttributeError: type object 'UsersDB' has no attribute '_meta'
and I don't understand what I need to add to the UserDB class to get rid of the error.
Any help would be great!
This isn't anything to do with Python versions, or metaclasses.
ModelForms only work with Django models. db.Model is the App Engine model class, not the Django one. You can't use a modelform with that class.
You mention django-nonrel in your question tags. That project allows you to use the Django models - subclasses of models.Model with the App Engine datastore. You probably want to do that.