I'm trying to deserialize some Protobuf bytestrings, in Python, outside of GAE(AppEngine) environment. I'm wondering if I need the message schema for this use case.
Approaches I've tried so far includes google.cloud.proto.datastore.v1.entity_pb2.Entity.ParseFromString():
>>> pb = ... ## fetched a protobuf into `pb` using the Datastore Protobuf API
>>> pb.__class__
<class 'google.cloud.proto.datastore.v1.entity_pb2.Entity'>
>>> serialized = pb.SerializeToString() ## <-- now `serialized` is a bytestring.
>>> type(serialized)
<type 'str'>
>>> e = google.cloud.proto.datastore.v1.entity_pb2.Entity()
>>> e.ParseFromString(serialized) ## <-- deserialization
>>> print e
The deserialization above actually worked, so I'm assuming I don't need a to know the message schema. --Correct me if I'm wrong.
However, I tried this on a different bytestring, it didn't work. But it worked with google.appengine.ext.db.model_from_protobuf(bytestring2) in GAE environment though.
Thoughts?? Thanks
Other notes
Somewhat relevant post.
Related
I am new to MuleSoft
when i am giving output expression like below i am getting an data sense error, can you please help
#[output application/json --- {errorType:error.errorType, error:error.description}]
Description Resource Path Location Type Scripting language error on
expression '#[output application/json --- {errorType:error.errorType,
error:error.descrip...'. Reason: Invalid input '#', expected using,
if, ???, unless or not (line 1, column 1): . validations.xml
/validations/src/main/mule Listener Message Flow Error
It is hard to know without a snippet from the configuration XML exactly what your issue is, but one problem I see is that your dataweave does not start with a
"%dw 2.0" In mule 4 there are two languages. Mule Expression language, which is default in most components (ie #[vars.name] is MEL), and dataweave 2.0 which is default in Transform Message components. You can however, as you've attempted to do, use dataweave inside of Mule Expressions. You have it mostly right, but it must start with "%dw 2.0" at the beginning. So it should look like this.
#[%dw 2.0 output application/json --- {errorType:error.errorType, error:error.description}]
However, judging by the error message, it looks like you're attempting to use MEL in a place where it is not allowed. If you provide a snippet of the configuration XML for this component, I can be more helpful.
There is no MEL in Mule 4. It is migrated to DW2.0. Please refer link https://docs.mulesoft.com/mule-runtime/4.2/migration-mel
In Google AppEngine(GAE) environment, I can do following to convert a Protobuf bytestring back to a Datastore model:
from google.appengine.ext import db
byte_str = ....
model = db.model_from_protobuf(byte_str.decode("base64"))
Outside of GAE, I normally use the google-cloud-datastore client to access Datastore models:
from google.cloud import datastore
...
client = datastore.Client(project_id)
query = client.query(kind='Event', order=('-date',))
for result in query.fetch(limit=100):
print result
# every result is of class `google.cloud.datastore.entity.Entity`
Question: What if I'm only given the encoded byte string? How can I convert it back to a google.cloud.datastore.entity.Entity (or dict)?
Followups:
Update1:
google.cloud.proto.datastore.v1.entity_pb2.Entity.ParseFromString() is the closest I found so far. But not quite working yet..
## fetched a protobuf into `pb`
>>> pb.__class__
<class 'google.cloud.proto.datastore.v1.entity_pb2.Entity'>
>>> entity = google.cloud.datastore.helpers.entity_from_protobuf(pb)
>>> entity.__class__
<class 'google.cloud.datastore.entity.Entity'>
>>> serialized = pb.SerializeToString() ## <-- now this is the bytestring I meant.
>>> type(serialized)
<type 'str'>
>>> google.cloud.proto.datastore.v1.entity_pb2.Entity.ParseFromString(serialized)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: descriptor 'ParseFromString' requires a 'google.protobuf.pyext._message.CMessage' object but received a 'str'
>>>google.cloud.proto.datastore.v1.entity_pb2.Entity().ParseFromString(serialized)
76942 ## <--??
Update2:
Actually had to do this:
e = google.cloud.proto.datastore.v1.entity_pb2.Entity()
e.ParseFromString(byte_str)
print e
And that did it..
this worked:
e = google.cloud.proto.datastore.v1.entity_pb2.Entity()
e.ParseFromString(byte_str)
print e
I am trying to save all exceptions thrown by a python service (running hidden on Windows 7/10) into a sqlite3 database file. In addition to other things(traceback, date, etc) I want to save the error message too.
My Problem is that there are some error messages that I cannot convert to unicode (Especially some WindowsErrors and errors with a german 'Umlaut': ä, ö, ü). As I don't know every possible error in advance I want to have a function that can handle ALL errors and convert their message to unicode.
Can someone tell me what the convertToUnicode-function has to look like?
# -*- coding: utf-8 -*-
def someErrorneousFunction():
raise RuntimeError('Not a unicode string. But with a german Umlaut: ä!')
def saveExceptionToDBThatOnlyAcceptsUnicode(msg):
"""SQL INSERT SOME STUFF... + THE UNICODE ERROR MESSAGE!"""
pass
def convertToUnicode(e):
""" What to do here ??? """
pass
try:
someErrorneousFunction()
except Exception as e:
unicode_error_msg = convertToUnicode(e)
saveExceptionToDBThatOnlyAcceptsUnicode(unicode_error_msg)
Note: I found that some Exceptions have an attribute called .msg or .str, but not all of them!
Does this method make sense at all? I know its bad practice to catch ALL exceptions with no distinction but as my software occasionally is in testmode somewhere else and I want to get the exception database per mail it seemed meaningful to me. Furthermore I distinguish the errors I know from those I don't know in advance.
I would appreciate any advise!
Thank you!
Sebastian
If, as your demo shows, the source file is known to be in UTF-8, then this should work:
# -*- coding: utf-8 -*-
import traceback
def someErrorneousFunction():
raise RuntimeError('Not a unicode string. But with a german Umlaut: ä!')
def saveExceptionToDBThatOnlyAcceptsUnicode(msg):
print type(msg)
print msg
def convertToUnicode(e):
return traceback.format_exc(e).decode('utf8')
try:
someErrorneousFunction()
except Exception as e:
unicode_error_msg = convertToUnicode(e)
saveExceptionToDBThatOnlyAcceptsUnicode(unicode_error_msg)
Output:
<type 'unicode'>
Traceback (most recent call last):
File "C:\test.py", line 15, in <module>
someErrorneousFunction()
File "C:\test.py", line 5, in someErrorneousFunction
raise RuntimeError('Not a unicode string. But with a german Umlaut: ä!')
RuntimeError: Not a unicode string. But with a german Umlaut: ä!
Note: I'm a beginner with these technologies.
I made a a.sch (schematron file) and b.xml which I'm validating with libxml2 C API. Everything is working good except that I don't see the values of <value-of select=''xpath here''> in my assert messages. I'm using the xmllint application which can be found in the libxml2 git repository (https://git.gnome.org/browse/libxml2). The command line and results are shown at the end of this post.
If I take the same files and I use the following python code I'm able to see the values in the assert message.
#!/usr/bin/env python
from lxml import isoschematron
from lxml import etree
def runsch(rulesFile, xmlFile):
# open files
rules = open(rulesFile, 'r') # Schematron schema
XMLhere = open (xmlFile, 'r') # XML to be checked
# Parse schema
sct_doc = etree.parse(rules)
schematron = isoschematron.Schematron(sct_doc, store_report = True)
# Parse xml
doc = etree.parse(XMLhere)
# Validate against schema
validationResult = schematron.validate(doc)
report = schematron.validation_report
# Check result
if validationResult:
print("passed")
else:
print("failed")
print(report)
reportFile = open('report.html', 'wb')
report.write(reportFile)
def main():
runsch('a.sch', 'b.xml')
if __name__ == '__main__':
main()
Here's the a.sch:
<?xml version="1.0" encoding="utf-8"?>
<schema xmlns="http://purl.oclc.org/dsdl/schematron" schemaVersion="1.0" xml:lang="en" >
<title>Different behavior of libxml2 C API vs lxml Python regarding value-of select in assert message</title>
<ns prefix="test" uri="test"/>
<pattern id="a">
<rule context="test:b">
<assert test="count(test:c) = 3">There's only <value-of select="count(test:c)"/> c element(s), it is mandatory to have 3.</assert>
</rule>
</pattern>
</schema>
Here's the b.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<?valbuddy_schematron a.sch?>
<a xmlns="test" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<b>
<c>word 1</c>
<c>word 2</c>
</b>
</a>
Here's the command line using the libxml2 C API (replace files path with your path):
xmllint --schematron ~/svn/r-oltcms/Config/a.sch
~/svn/r-oltcms/Config/b.xml
Here's a portion of the results, we can see that we don't get the value-of select in the assert message.
> /*/* line 4: There's only c element(s), it is mandatory to have 3.
> /home/oltcms/svn/r-oltcms/Config/b.xml fails to validate
Here's a portion of the result of the python script, we can see that we get the value-of select in the assert message.
> <svrl:failed-assert test="count(test:c) = 3"
> location="/*[local-name()='a' and
> namespace-uri()='test']/*[local-name()='b' and
> namespace-uri()='test']">
> <svrl:text>There's only 2 c element(s), it is mandatory to have 3.</svrl:text> </svrl:failed-assert>
So, is there a way to get these in the assert message using the libxml2 C API? Any explanation is welcome.
Thanks,
Michel
I think that libxml2 only implements an early Schematron version, probably 1.5 or 1.6 which don't support value-of in assert. The Python bindings to libxml2's Schematron validator are in lxml.etree.Schematron. But lxml also provides lxml.isoschematron.Schematron which seems to support a newer Schematron version. This validator isn't based on libxml2 directly but on XSLT transforms. So there's probably no way to get the values from a value-of in an assert using libxml2's C API or xmllint, but you could try to put value-of in a diagnostic.
I am trying to use HTSQL for one of my Django projects. For that I followed the procedure given HERE for furnishing HTSQL/Django requirements. Then I cloned the HTSQL repository for trying the example/demo in it from HERE. The default db used in the demo example is sqlite3.
I have tried this demo on both Django v 1.4 and Django v 1.3.1(had to make some tweaks in settings.py for Django v 1.3.1). As instructed in the HTSQL Django-gateway Blog, I wrote the following code in the django project shell:
>>> from htsql_django import produce
>>> query = "/polls_poll{question, total:=sum(polls_choice.votes)}"
>>> for row in produce(query):
>>> print "%s: %s" % (row.question, row.total)
It throws following error:
TransactionManagementError: This code isn't under transaction management
The whole error trace can be viewed at pastebin
I have also tried this on my own new project but same error.
When you use HTSQL from a Django shell, you have to open a transaction explicitly:
>>> from django.db import transaction
>>> from htsql_django import produce
>>> with transaction.commit_on_success():
... query = "/polls_poll{question, total:=sum(polls_choice.votes)}"
... for row in produce(query):
... print "%s: %s" % (row.question, row.total)
I'm sorry the documentation isn't clear about that. We may change it in a future release.