I was able to use the Java API to decode the binary that is read from Kafka as there is BinaryDecoder from the DecoderFactory. I am trying repeat this using the C API.
However, no luck so far. It is straightforward to set values and then get values (schema and data are bond with each other). In the case where the encoded data is put into Kafka, we need to decode the data using a provided schema. Can you anyone shed some light on how to do this using the C API? Thanks a lot!
For example, we will get the following encoded data from Kafka
" site_visit^Z111.222.333.444^#^T1446231845^#^N1209600^#^F1.0"
and we have the schema available (separated from the data above, as we do using Java):
I want to the decoded data to be:
{"segment_name": "site_visit", "ip_address": "111.222.333.444", "received_at": "T1446231845", "ttl": "1209600", "probability": "1.0"}
You can use libserdes for this purpose. There are examples (serdes-kafka-avro-client.c, kafka-serdes-avro-console-consumer.cpp) how to get such JSON result.
Related
Currently I'm using form recognizer version 2.1 preview to train a custom model. I'm able to test the model in Form Recognizer Labeling Tool and got the output. When I input the same file that I got out in labeling tool in my program I'm getting the error below.
{"status": "failed", "createdDateTime": "2020-09-25T20:03:21Z", "lastUpdatedDateTime": "2020-09-25T20:03:21Z", "analyzeResult": {"version": "2.1.0", "errors": [{"code": "2005", "message": "The file submitted couldn't be parsed. This can be due to one of the following reasons: the file format is not supported ( Supported formats include JPEG, PNG, BMP, PDF and TIFF), the file is corrupted or password protected."}]}}
The GET request code used is:
resp = requests.get(url=get_url,headers={"Ocp-Apim-Subscription-Key":FORM_RECOGNIZER_SUBSCRIPTION_KEY})
I have saved the file to server and then tried to read it from there and pass the read file to form recognizer. This worked for me. But I don't know why did it happen.
I also encountered this exact error message following this article.
The article show 4 steps:
Train Model (require SAS to blob - whole folder)
Get model result
Analyze (require SAS to a single file)
Get analyze result
Profit
I got this error on step 4.
After monkeying around, I figured that the cause is not actually in step 4 but in step 3 instead. I was providing SAS to the blob instead of SAS to the file.
After I corrected the SAS URL it works perfectly.
Here is how to get SAS to blob:
Here is how to get SAS to a file:
What file are you trying to use as input ? Form Recognizer supports PDF, Tiff and images (PNG and JPEG) as file types and inputs to the analyze API. See more details here - https://learn.microsoft.com/en-us/azure/cognitive-services/form-recognizer/quickstarts/python-labeled-data?tabs=v2-0#analyze-forms-for-key-value-pairs-and-tables
Does Flink supports Side Outputs feature in Dataset(Batch Api) ? If not, how to handle valid and invalid records when loading from file ?
You can always do something like this:
DataSet<EventOrInvalidRecord> goodAndBadTogether = input.map(new CreateObjectIfPossible())
goodAndBadTogether.filter(new KeepOnlyGood())...
goodAndBadTogether.filter(new KeepOnlyBad())...
Another reasonable option in some cases is to go ahead and use the DataStream API, even if you don't have streaming sources.
I want to send accelerometer data to IBM iot cloud using c language.
In this i am using json format to publish event in cloud.
char *data="{\"d\" : {\"x\" : 43 }}"; //is working correctly ..
I want to send instant values through this pointer in JSON Format.. Please help me how to send the instant values by using JSON format in IBMIOTfclient side
If you use the IBMIOTfclient (iotfclient.h) you can call publishEvent, like:
publishEvent(&client, "accel","json", "{\"d\" : {\"x\" : 43 }}", QOS0);
There is a sample in github.
I am trying to read XML data in Arduino from a google spreadsheet published to the web using an HTTP GET request to the following link.
https://spreadsheets.google.com/feeds/cells/SpreadsheetID/5/public/basic?&range=D10
I receive the following reply along with some headers which I can observe on the serial port.
I want to parse the data written in bold format in the above reply. The data can be a real number and can be positive & negative. Please help me to find a way to parse this data.
if you need only one-two easy(out of param) nodes, you can use my function :)
String xmlTakeParam(String inStr,String needParam)
{
if(inStr.indexOf("<"+needParam+">")>0){
int CountChar=needParam.length();
int indexStart=inStr.indexOf("<"+needParam+">");
int indexStop= inStr.indexOf("</"+needParam+">");
return inStr.substring(indexStart+CountChar+2, indexStop);
}
return "not found";
}
I tried to find a solution for a long time, leave it here for clarity.
The complex xml with repeating nodes already so will not succeed.
Have you checked out this library?
https://web.archive.org/web/20160622041818/http://interactive-matter.eu/how-to/ajson-arduino-json-library/
You're better off converting your XML to JSON and giving that a go considering the memory availability on Arduino.
Otherwise if you really want to work with XML, then there's always these resources:
https://github.com/RobTillaart/Arduino/tree/master/libraries/XMLWriter
http://john.crouchley.com/blog/archives/454
I have been struggling with some very basic understanding of how google app engine store data
I have defined a class defining a client profile as such :
class ClientProfile(ndb.Model):
nickname = ndb.StringProperty(required=True)
photo = ndb.BlobProperty()
uuid = ndb.StringProperty(required = True)
I retrieve an image data only uploading image.src via a POST using jquery.ajax(...)
The data are properly sent to Google app engine and I can assign them to a variable with
imagesrc = self.request.get('photosrcbase64')
The data content is something looking like :
"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...
So fare so good, the data is an image/png and the encoding is Base64, but should I care if it end in a Blob ?
Now if I try to put the data in the photo Blob
with for example :
clientprofile.photo = imagesrc I get a Bad Value Error, in this case :
BadValueError: Expected str, got
u'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAMgAA
I tried all kind of combinations using different solutions and get back all kind of BadValue or type errors.
My question are :
1) Why does the Blob care, if it is a binary storage, I should be able to dump in it anything without having to interpret it and/or convert it, so is a Blob really a Binary/Raw storage and why does it care about how things are stored in it ?
2) I have started having problems with this 2 years ago when still using db instead of ndb, I found a solution that I did not understand by stripping out the MIME information at the beginning of the data string, decoding the string Base64 and using db.Blob(...) to convert my string to a Blob. the problem is that db.Blob() does not seem to exist in ndb so I can't do this any more.
I am convinced that I am missing something fundamental in the way informations are exchanged between google app engine and the browser and thank you in advance for a mind clearing answer
A BlobProperty is meant to be binary data. The str type in Python is fully equivalent to a byte string since the only characters allowed are
[chr(byte_value) for byte_value in range(2**8)]
So before storing the value from self.request.get('photosrcbase64'), which is of type unicode, you'll need to cast to type str.
You can do this either by directly doing so
imagesrc = str(self.request.get('photosrcbase64'))
or first trying to decode to ascii.