Deserialize malformed Json returning empty arrays - arrays

Please, I use VB.NET Http Requests to read data from a webservice. It used to send data this way:
[
{
"id": 7532,
"nome": "LABOR INC.",
"isClient": false,
"personality": {
"id": 2,
"value": "CORPORATION"
},
"registryNumbers": [
{
"id": 9378,
"number": "20786790174"
}
],
"personality_id": 2
},
{
"id": 7537,
"nome": "JOSE SILVA",
"isClient": false,
"personality": {
"id": 1,
"value": "PERSON"
},
"gender": {
"id": 1,
"value": "MALE"
},
"cityOfBirth": {
"id": 355030,
"value": "SAO PAULO"
},
"nationality": {
"id": 85,
"value": "BRAZILIAN"
},
"registryNumbers": [
{
"id": 9383,
"number": "03217495388"
}
],
"personality_id": 1
}
]
It was ok because unused fields (as "gender" and "cityOfBirth" for corporations) were omitted. Since some days, however, it started to send back these fields as empty arrays ([]), like this:
{
"id": 7532,
"nome": "LABOR INC.",
"isClient": false,
"personality": {
"id": 2,
"value": "CORPORATION"
},
"gender": [],
"cityOfBirth": [],
"nationality": [],
"registryNumbers": [
{
"id": 9378,
"number": "20786790174"
}
],
"personality_id": 2
}
And because of that it misfit the destiny properties in deserialization class, because these are not (and can't be) enumerations/arrays but single objects.
My question: is there some deserialization extension or attribute I can add to my classes in order to deserialize those ([]) as null/Nothing? Special thanks if it comes in VB.NET, but I'm able to read and adapt C# as well.
That is, I'd like to know how I could make my code halt when an array is "forced" into a property that expects single objects, and do the proper treatment at this point.

Tried and solved the problem with a custom converter:
Public Class BogusArrayJsonConverter
Inherits JsonConverter
Public Overrides Function CanConvert(objectType As Type) As Boolean
Return GetType(MyRecord).IsAssignableFrom(objectType)
End Function
Public Overrides Function ReadJson(reader As JsonReader, objectType As Type, existingValue As Object, serializer As JsonSerializer) As Object
If reader.TokenType = JsonToken.StartArray Then
Return serializer.Deserialize(Of MyRecord())(reader).SingleOrDefault
Else
Return serializer.Deserialize(Of MyRecord)(reader)
End If
End Function
Public Overrides Sub WriteJson(writer As JsonWriter, value As Object, serializer As JsonSerializer)
Throw New NotImplementedException()
End Sub
End Class
Thanks to all who tried to help me.

Related

How to register custormize serializer for flink kafka format type

I create table as below, and the data is from kafka.
I want to deserialize the json message to Pojo object.
But the message format is not avro or simple json.
So I need to know how to register custormized serializer and use it for the 'format.type' property.
By the way, my flink version is 1.10.0.
CREATE TABLE MyUserTable(
uuid VARCHAR,
orgId VARCHAR
) with (
'connector.type' = 'kafka',
'connector.version' = '0.11',
'connector.topic' = 'topic_name',
'connector.properties.zookeeper.connect' = 'localhost:2181',
'connector.properties.bootstrap.servers' = 'localhost:9092',
'connector.properties.group.id' = 'testGroup',
'format.type' = 'cutormizeSerializer'
)
The kafka message body sample, each columnName is the key for Pojo object, and rawData is value:
{
"beforeData": [],
"byteSize": 272,
"columnNumber": 32,
"data": [{
"byteSize": 8,
"columnName": "APPLY_PERSON_ID",
"rawData": 10017,
"type": "LONG"
}, {
"byteSize": 12,
"columnName": "UPDATE_SALARY",
"rawData": "11000.000000",
"type": "DOUBLE"
}, {
"byteSize": 11,
"columnName": "UP_AMOUNT",
"rawData": "1000.000000",
"type": "DOUBLE"
}, {
"byteSize": 3,
"columnName": "CURRENCY",
"rawData": "CNY",
"type": "STRING"
}, {
"byteSize": 32,
"columnName": "EXCHANGE_RATE",
"rawData": "1.000000000000000000000000000000",
"type": "DOUBLE"
}, {
"byteSize": 11,
"columnName": "DEDUCTED_ACCOUNT",
"rawData": "1000.000000",
"type": "DOUBLE"
}, {
"byteSize": 1,
"columnName": "ENTER_AT_PROCESS",
"rawData": "Y",
"type": "STRING"
}],
"dataCount": 0,
"dataMetaData": {
"connector": "mysql",
"pos": 1000368076,
"row": 0,
"ts_ms": 1625565737000,
"snapshot": "false",
"db": "testdb",
"table": "flow_person_t"
},
"key": "APPLY_PERSON_ID",
"memorySize": 1120,
"operation": "insert",
"rowIndex": -1,
"timestamp": "1970-01-01 00:00:00"
}
The Pojo object as below:
import lombok.Data;
#Data
public class HrSalaryPersonVO {
private String uuid;
private String orgId;
private String unitId;
private String effectiveDate;
private int adjustPersonCount;
private Double adjustAmount;
private Double beforeSalaryAmount;
private Double adjustRate;
private String data0prateType;
private String status;
}
You can implement you own DeserializationFactory like shown here. The factory needs to be added to the META-INF/services/org.apache.flink.table.factories.TableFactory file to be discovered in the DDL string.
Note that this example is for Flink 1.10 as requested. Users of newer Flink versions can take a look at the full connector example that includes defining a custom format. The new stack extends from org.apache.flink.table.factories.DecodingFormatFactory/EncodingFormatFactory. And uses the META-INF/services/org.apache.flink.table.factories.Factory service file.

How to add required to sub array in a json schema?

I'm creating a json schema to define necessary data with data types. There is some data need to be set into required filed. But didn't find how to do it in its document.
For this json schema:
{
"type": "object",
"required": [
"version",
"categories"
],
"properties": {
"version": {
"type": "string",
"minLength": 1,
"maxLength": 1
},
"categories": {
"type": "array",
"items": [
{
"title": {
"type": "string",
"minLength": 1
},
"body": {
"type": "string",
"minLength": 1
}
}
]
}
}
}
json like
{
"version":"1",
"categories":[
{
"title":"First",
"body":"Good"
},
{
"title":"Second",
"body":"Bad"
}
]
}
I want to set title to be required, too. It's in a sub array. How to set it in json schema?
There are a few things wrong with your schema. I'm going to assume you're using JSON Schema draft 2019-09.
First, you want items to be an object, not an array, as you want it to apply to every item in the array.
If "items" is a schema, validation succeeds if all elements in the
array successfully validate against that schema.
If "items" is an array of schemas, validation succeeds if each
element of the instance validates against the schema at the same
position, if any.
https://datatracker.ietf.org/doc/html/draft-handrews-json-schema-02#section-9.3.1.1
Second, if the value of items should be a schema, you need to treat it like a schema in its own right.
If we take the item from your items array as a schema, it doesn't actually do anything, and you need to nest it in a properties keyword...
{
"properties": {
"title": {
"type": "string",
"minLength": 1
},
"body": {
"type": "string",
"minLength": 1
}
}
}
Finally, now your items keyword value is a schema (subschema), you can add any keywords you can normally use, such as required, the same as you have done previously.
{
"required": [
"title"
],
"properties": {
...
}
}

Apache Nifi: Parse data with UpdateRecord Processor

I'm trying to parse some data in Nifi (1.7.1) using UpdateRecord Processor.
Original data are json files, that I would like to convert to Avro, based on a schema.
The Avro conversion is ok, but in that convertion I also need to parse one array element from the json data to a different structure in Avro.
This is a sample data of the input json:
{ "geometry" : {
"coordinates" : [ [ 4.963087975800593, 45.76365595859971 ], [ 4.962874487781098, 45.76320922779652 ], [ 4.962815443439148, 45.763116079159374 ], [ 4.962744732112515, 45.763010484202866 ], [ 4.962096825239138, 45.762112721939246 ] ]} ...}
Being its schema (specified in RecordReader):
{ "type": "record",
"name": "features",
"fields": [
{
"name": "geometry",
"type": {
"type": "record",
"name": "geometry",
"fields": [
{
"name": "coordinatesJson",
"type": {
"type": "array",
"items": {
"type": "array",
"items": "double"
}
}
},
]
}
},
....
]
}
As you can see, coordinates is an array of arrays.
And I need to parse those data to Avro, based on this schema (specified in RecordWriter):
{
"name": "outputdata",
"type": "record",
"fields": [
{"name": "coordinatesAvro",
"type": {
"type": "array",
"items" : {
"type" : "record",
"name" : "coordinatesAvro",
"fields" : [ {
"name" : "X",
"type" : "double"
}, {
"name" : "Y",
"type" : "double"
} ]
}
}
},
.....
]
}
The problem here is that I'm not being able to parse from coordinatesJson to coordinatesAvro, using RecordPath functions
I tried several mappings, like:
Property: Value:
/coordinatesJson[0..-1]/X /geometry/coordinatesAvro[*][0]
/coordinatesJson[0..-1]/Y /geometry/coordinatesAvro[*][1]
It should be a pretty straighforward parsing step, but as I said, I've been going in circles to achive this for a while.
Any help would be really appreciated.
When I collide with something like that I do next:
1) Transofrm Json into Json with strcuture that I need (for example in your case: coordinatesAvro) by ExecuteScript Processor. I have used ECMAScript cause you can simple parse JSON and work with objects (transform them).
2) ConvertJsonToAvro with one common schema (coordinatesAvro in your case) for Reader and Writer.
It works very good and I have used it on BigData cases. This is one of possible resolutions for your problem.

Node.JS - How to access Values of Dictionary within an Array of a Key in a Dictionary?

I'm new in Node.JS and I'm able to parse the JSON data and do a console log to print out name and badges.
var details = JSON.parse(body);
console.log(details.name, details.badges.length);
But I don't know how I can get the data inside the arrays of the bagdes such as id, name, url.
I tried
console.log(details.badges.length.id);
But nothing shows up. How can I access that? Thank you.
{
"name": "Andrew Chalkley",
"badges": [
{
"id": 49,
"name": "Newbie",
"url": "http:\/\/teamtreehouse.com\/chalkers",
"icon_url": "https:\/\/achievement-images.teamtreehouse.com\/Generic_Newbie.png",
"earned_date": "2012-07-23T19:59:34.000Z",
"courses": [
]
},
{
"id": 26,
"name": "Introduction",
"url": "http:\/\/teamtreehouse.com\/library\/html\/introduction",
"icon_url": "https:\/\/achievement-images.teamtreehouse.com\/HTML_Basics.png",
"earned_date": "2012-07-23T21:57:24.000Z",
"courses": [
{
"title": "HTML",
"url": "http:\/\/teamtreehouse.com\/library\/html",
"badge_count": 1
},
{
"title": "Introduction",
"url": "http:\/\/teamtreehouse.com\/library\/html\/introduction",
"badge_count": 1
}
]
}
}
It is an array, so you need the index, for example: details.badges[0].id
This will return the first (index 0) element id.
.length only returns the length of the array, so it will not be useful to get the data in it.

Circular Reference - AngularJS/EF/WebAPI

I'm using AngularJS, EF and WebAPI. I have a one to many relationship between ObjectA and ObjectB.
In the UI, I want to loop through a list of ObjectA and do something like:
<table>
<tr ng-repeat="objectA in objectAs">
<td>objectA.objectB.Description</td>
<td>objectA.someValue</td>
</tr>
</table>
The problem is, if in my access layer, I do:
db.ObjectA.Include(o => o.ObjectB).ToList()
I get a nice error:
Object graph for type 'ObjectB' contains cycles and cannot be serialized if reference tracking is disabled.
Ok, no problem, I just add:
[DataContract(IsReference=true)]
to the *.tt file that generates the contracts (from EF).
WebAPI returns valid values (no error), but it looks like Angluar can't handle the "references" returned, which looks something like:
[
{
"$id": "1",
"someValue": "Pool",
"objectB": {
"$id": "2",
"Description": "Standard",
"ObjectAs": [
{
"$ref": "1"
},
{
"$id": "3",
"someValue": "Poolhouse",
"ObjectB": {
"$ref": "2"
},
},
},
{
"$ref": "3"
},
{
"$ref": "4"
},
{
"$ref": "5"
},
{
"$ref": "6"
},
{
"$ref": "7"
},
{
"$ref": "8"
},
{
"$ref": "9"
},
{
"$ref": "10"
},
{
"$ref": "11"
}
]
Now I can't really modify my DTOs to remove the DataMember attributes for certain navigation properties.
Any suggestions on best practices? Should I just return a light DTO (just object A), then have a javascript method that looks up Object B?
EDIT:
I took a look at the output from Angular. It turns out it converts the $ref props that WebAPI puts in to empty elements {}. Something like...
[
{
"$id": "1",
"someValue": "Pool",
"objectB": {
"$id": "2",
"Description": "Standard",
"ObjectAs": [
{
},
{
"$id": "3",
"someValue": "Poolhouse",
"ObjectB": {
},
},
},
{}, {},{}, {},{}, {},{}, {}
]
So from here, I see a few options:
Change code to only return a singluar object and make sep calls to get lookup fields
Set code to null out circular references:
ObjectA.objectB.ObjectAs = null;
Write my own formatter to handle the output
I would have expected this to be a common issue, but can't seem to find any posts about it. Is anyone else running into this?
I ended up add this to the WebApiConfig:
config.Formatters.JsonFormatter.SerializerSettings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore;
config.Formatters.JsonFormatter.SerializerSettings.PreserveReferencesHandling = Newtonsoft.Json.PreserveReferencesHandling.None;
That produced "better" json from WebApi (no references)

Resources