mappingMongoConverter.setMapKeyDotReplacement doesnt seem to have any effects - spring-data-mongodb

I am trying to convert a custom object that contains a map containing dotted key string value using the latest 1.7.2 spring mongodb.
Setting a dot replacement doesnt seem to do the job. Here's my code :
class FakeUser {
Map<String, String> map = new LinkedHashMap<>();
void addValue(String key, String value) {
this.map.put(key, value);
}
}
FakeUser fakeUser = new FakeUser();
fakeUser.addValue("test.dot.for.key", "test.dot.for.value");
this.mappingMongoConverter.setMapKeyDotReplacement(":");
Object convertedObject = this.mappingMongoConverter.convertToMongoType(fakeUser);
System.out.println("convertedObject: " + convertedObject.getClass() + ":" + convertedObject);
And the output:
convertedObject: class com.mongodb.BasicDBObject:{ "map" : { "test.dot.for.key" : "test.dot.for.value"}}
And i also tried:
class FakeUser {
Map<String, String> map = new LinkedHashMap<>();
void addValue(String key, String value) {
this.map.put(key, value);
}
}
FakeUser fakeUser = new FakeUser();
fakeUser.addValue("test.dot.for.key", "test.dot.for.value");
this.mappingMongoConverter.setMapKeyDotReplacement(":");
BasicDBObject dbo = new BasicDBObject();
this.mappingMongoConverter.write(fakeUser, dbo);
System.out.println("dbo: " + ":" + dbo.toMap());
And with the output of dbo: :{_class=app.security.MyClass$1FakeUser, map={ "test.dot.for.key" : "test.dot.for.value"}}
I was expecting "test.dot.for.key" to become "test:dot:for:key", so what did i do wrong ?

You need make adjustments to the converter in your SpringMongoDBConfig (which should extend AbstractMongoConfiguration) config files, rather than in the class you're trying to serialize/deserialze. If you're using annotation driven settings, you can use a custom bean to set the converter like so:
#Bean
#Override
public MappingMongoConverter mappingMongoConverter() throws Exception
{
DbRefResolver dbRefResolver = new DefaultDbRefResolver(mongoDbFactory());
MappingMongoConverter converter = new MappingMongoConverter(dbRefResolver, mongoMappingContext());
converter.setCustomConversions(customConversions());
//mongo won't accept key values with dots(.) in them, so configure it to store them as :
converter.setMapKeyDotReplacement("\\:");
return converter;
}
Once the converter is setup, it will handle both serialization and deserialization for you automagically.
If you really want to do the conversion in-line (not recommended), you can just use a string replace function to re-write the string like so:
class FakeUser
{
Map<String, String> map = new LinkedHashMap<>();
void addValue(String key, String value) {
key = key.replace(".",":");
this.map.put(key, value);
}
}
Although, if you do it inline you'll likely have issues deserializing when you get it back out of mongo.

Related

How can I add message key to KafkaSink in Apache Flink 1.14

As stated in the title I need to set a custom message key in KafkaSink. I cannot find any indication on how to achieve this in the Apache Flink 1.14 docs.
At the moment I'm correctly setting up the KafkaSink and the data payload is correctly written in the topic, but the key is null.
Any suggestions? Thanks in advance
You should implement a KafkaRecordSerializationSchema that sets the key on the ProducerRecord returned by its serialize method.
You'll create the sink more-or-less like this:
KafkaSink<UsageRecord> sink =
KafkaSink.<UsageRecord>builder()
.setBootstrapServers(brokers)
.setKafkaProducerConfig(kafkaProps)
.setRecordSerializer(new MyRecordSerializationSchema(topic))
.setDeliverGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
.setTransactionalIdPrefix("my-record-producer")
.build();
and the serializer will be something like this:
public class MyRecordSerializationSchema implements
KafkaRecordSerializationSchema<T> {
private static final long serialVersionUID = 1L;
private String topic;
private static final ObjectMapper objectMapper =
JsonMapper.builder()
.build()
.registerModule(new JavaTimeModule())
.configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false);
public MyRecordSerializationSchema() {}
public MyRecordSerializationSchema(String topic) {
this.topic = topic;
}
#Override
public ProducerRecord<byte[], byte[]> serialize(
T element, KafkaSinkContext context, Long timestamp) {
try {
return new ProducerRecord<>(
topic,
null, // choosing not to specify the partition
element.ts.toEpochMilli(),
element.getKey(),
objectMapper.writeValueAsBytes(element));
} catch (JsonProcessingException e) {
throw new IllegalArgumentException(
"Could not serialize record: " + element, e);
}
}
}
Note that this example is also setting the timestamp.
FWIW, this example comes from https://github.com/alpinegizmo/flink-mobile-data-usage/blob/main/src/main/java/com/ververica/flink/example/datausage/records/UsageRecordSerializationSchema.java.
This example is for scala programmers. Here, we are defining a key by generating UUID for each events.
import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema
import org.apache.kafka.clients.producer.ProducerRecord
import java.lang
class MyRecordSerializationSchema extends KafkaRecordSerializationSchema[String] {
override def serialize(element: String, context: KafkaRecordSerializationSchema.KafkaSinkContext, timestamp: lang.Long): ProducerRecord[Array[Byte], Array[Byte]] = {
return new ProducerRecord(
kafkaTopicName,
java.util.UUID.randomUUID.toString.getBytes,
element.getBytes
)
}
}
In the main class, will have to pass an instance of this class while defining the kafka sink like this:
val sinkKafka: KafkaSink[String] = KafkaSink.builder()
.setBootstrapServers(bootstrapServerUrl) //Bootstrap server url
.setRecordSerializer(new MyRecordSerializationSchema())
.build()

MVC Custom Validation String Array Client Side

There are examples out there of custom MVC validators that take an array parameter, but only server side - none of them show an example of implementing the client side with array parameter.
The problem is instead of outputting the array's contents in the html data- attribute, it outputs "System.String[]":
data-val-total-propertynames="System.String[]"
Here is my attribute class:
public class TotalAttribute : ValidationAttribute, IClientValidatable
{
private String[] PropertyNames { get; set; }
public TotalAttribute(String[] propertyNames)
{
PropertyNames = propertyNames;
}
protected override ValidationResult IsValid(object value, ValidationContext context)
{
float total = 0;
foreach (var propertyName in PropertyNames)
total += (float)context.ObjectInstance.GetType().GetProperty(propertyName).GetValue(context.ObjectInstance, null);
if (total != (float)value)
return new ValidationResult(FormatErrorMessage(context.DisplayName), new[] { context.MemberName });
return ValidationResult.Success;
}
public IEnumerable<ModelClientValidationRule> GetClientValidationRules(ModelMetadata metadata, ControllerContext context)
{
var rule = new ModelClientValidationRule
{
ErrorMessage = ErrorMessageString,
ValidationType = "total",
};
rule.ValidationParameters["propertynames"] = PropertyNames;
yield return rule;
}
}
Here it is implemented in the model:
[Total(new string[] { "SomeOtherField1", "SomeOtherField2" }, ErrorMessage = "'Line12Balance' must equal total of 'SomeOtherField1' and 'SomeOtherField2'")]
public decimal? Line12Balance { get; set; }
Here is the html data-val attribute output:
data-val-total-propertynames="System.String[]"
What am I doing wrong?
You get "System.String[]" string because the value of ValidationParameters["propertynames"] is written by calling ToString on it, so string[] variable returns "System.String[]" in this case. If you need to output specific value you need to format it by yourself in your validation attribute. For example, change
rule.ValidationParameters["propertynames"] = PropertyNames;
to
rule.ValidationParameters["propertynames"] = string.Join(",", PropertyNames);
and you will get
data-val-total-propertynames="SomeOtherField1,SomeOtherField2"
As Alexander mentioned in his answer, the issue you encountered is that string[] doesn't have its own ToString() implementation, and uses the base object.ToString() implementation, which just displays the type of the object rather than the contents of your array.
So you've got to somehow serialize your string array so that it can be stored in a string to then parse in your client-side JavaScript.
If you're already using JSON.Net in your solution (as many are), you can also do this by JSON serializing the array on the server side, then parsing the JSON client-side.
i.e. server-side:
rule.ValidationParameters["propertynames"] = JsonConvert.SerializeObject(PropertyNames);
Client-side: (
$.validator.addMethod('rule', function(value, element, params) {
var propertynames = JSON.parse(params.propertynames);
// do your validation
});
That prevents you from having to escape commas/delimiters in your parameters.

How to deserialize dodgy JSON (with improperly quoted strings, and missing brackets)?

I am having to parse (and ultimately reserialize) some dodgy JSON. it looks like this:
{
name: "xyz",
id: "29573f59-85fb-4d06-9905-01a3acb2cdbd",
status: "astatus",
color: colors["Open"]
},
{
name: "abc",
id: "29573f59-85fb-4d06-9905-01a3acb2cdbd",
status: "astatus",
color: colors["Open"]
}
There are a number of problems here - starting with the most severe.
color: colors["Open"]
WTF even is that? If I drop 'colors' then I can get an array of strings out but I can't tweak to work out of the box.
It is an array without square brackets. I can fix this by wrapping in them. But is there a way to support out of the box?
Properties have no quotes. Deserializing is fine for these.. but reserializing is just no dice.
Any suggestions of handling both in and out of this structure?
Answering your questions #1 - #3 in order:
Json.NET does not support reading dodgy property values in the form colors["Open"] (which, as you correctly note, violates the JSON standard).
Instead, you will need to manually fix these values, e.g. through some sort of Regex:
var regex = new Regex(#"(colors\[)(.*)(\])");
var fixedJsonString = regex.Replace(jsonString,
m => string.Format(#"""{0}{1}{2}""", m.Groups[1].Value, m.Groups[2].Value.Replace("\"", "\\\""), m.Groups[3].Value));
This changes the color property values into properly escaped JSON strings:
color: "colors[\"Open\"]"
Json.NET does, however, have the capability to write dodgy property values by calling JsonWriter.WriteRawValue() from within a custom JsonConverter.
Define the following converter:
public class RawStringConverter : JsonConverter
{
public override bool CanConvert(Type objectType)
{
return objectType == typeof(string);
}
public override bool CanRead { get { return false; } }
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
throw new NotImplementedException();
}
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
var s = (string)value;
writer.WriteRawValue(s);
}
}
Then define your RootObject as follows:
public class RootObject
{
public string name { get; set; }
public string id { get; set; }
public string status { get; set; }
[JsonConverter(typeof(RawStringConverter))]
public string color { get; set; }
}
Then, when re-serialized, you will get the original dodgy values in your JSON.
Support for deserializing comma-delimited JSON without outer brackets will be in the next release of Json.NET after 10.0.3. see Issue 1396 and Issue 1355 for details. You will need to set JsonTextReader.SupportMultipleContent = true to make it work.
In the meantime, as a workaround, you could grab ChainedTextReader and public static TextReader Extensions.Concat(this TextReader first, TextReader second) from the answer to How to string multiple TextReaders together? by Rex M and surround your JSON with brackets [ and ].
Thus you would deserialize your JSON as follows:
List<RootObject> list;
using (var reader = new StringReader("[").Concat(new StringReader(fixedJsonString)).Concat(new StringReader("]")))
using (var jsonReader = new JsonTextReader(reader))
{
list = JsonSerializer.CreateDefault().Deserialize<List<RootObject>>(jsonReader);
}
(Or you could just manually surround your JSON string with [ and ], but I prefer solutions that don't involve copying possibly large strings.)
Re-serializing a root collection without outer braces is possible if you serialize each item individually using its own JsonTextWriter with CloseOutput = false. You can also manually write a , between each serialized item to the underlying TextWriter shared by every JsonTextWriter.
Serializing JSON property names without a surrounding quote character is possible if you set JsonTextWriter.QuoteName = false.
Thus, to re-serialize your List<RootObject> without quoted property names or outer braces, do:
var sb = new StringBuilder();
bool first = true;
using (var textWriter = new StringWriter(sb))
{
foreach (var item in list)
{
if (!first)
{
textWriter.WriteLine(",");
}
first = false;
using (var jsonWriter = new JsonTextWriter(textWriter) { QuoteName = false, Formatting = Formatting.Indented, CloseOutput = false })
{
JsonSerializer.CreateDefault().Serialize(jsonWriter, item);
}
}
}
var reserializedJson = sb.ToString();
Sample .Net fiddle showing all this in action.

Unity C#: how to convert an Array of a class into JSON [duplicate]

This question already has answers here:
Serialize and Deserialize Json and Json Array in Unity
(9 answers)
Closed 5 years ago.
I need to save multiple player's data. I am doing it by making an array of PlayersInfo class and trying to convert the array into JSON. here is my code
PlayerInfo[] allPlayersArray = new PlayerInfo[1];
allPlayersArray[0] = new PlayerInfo();
allPlayersArray[0].playerName = "name 0";
string allPlayersArrayJson = JsonUtility.ToJson(allPlayersArray);
print(allPlayersArrayJson);
PlayerPrefs.SetString("allPlayersArrayJson", allPlayersArrayJson);
string newJson = PlayerPrefs.GetString("allPlayersArrayJson");
print(newJson);
PlayerInfo[] newArray = new PlayerInfo[1];
newArray = JsonUtility.FromJson<PlayerInfo[]>(newJson);
print(newArray[0].playerName);
First two print statements returns "{}" and 3rd one gives null reference error. TIA
Like I said in my comment, there is no direct support. Helper class is needed. This is only reason I am making this answer is because you are still having problems even after reading the link I provided.
Create a new script called JsonHelper. Copy and paste the code below inside it.
using UnityEngine;
using System.Collections;
using System;
public class JsonHelper
{
public static T[] FromJson<T>(string json)
{
Wrapper<T> wrapper = UnityEngine.JsonUtility.FromJson<Wrapper<T>>(json);
return wrapper.Items;
}
public static string ToJson<T>(T[] array)
{
Wrapper<T> wrapper = new Wrapper<T>();
wrapper.Items = array;
return UnityEngine.JsonUtility.ToJson(wrapper);
}
[Serializable]
private class Wrapper<T>
{
public T[] Items;
}
}
The code in your question should now work. All you have to do is to replace all JsonUtility words with JsonHelper. I did that for you below:
void Start()
{
PlayerInfo[] allPlayersArray = new PlayerInfo[1];
allPlayersArray[0] = new PlayerInfo();
allPlayersArray[0].playerName = "name 0";
string allPlayersArrayJson = JsonHelper.ToJson(allPlayersArray);
print(allPlayersArrayJson);
PlayerPrefs.SetString("allPlayersArrayJson", allPlayersArrayJson);
string newJson = PlayerPrefs.GetString("allPlayersArrayJson");
print(newJson);
PlayerInfo[] newArray = new PlayerInfo[1];
newArray = JsonHelper.FromJson<PlayerInfo>(newJson);
print(newArray[0].playerName);
}
Based on the JsonUtility documentation, naked arrays are not supported. Put the array inside a class.
Internally, this method uses the Unity serializer; therefore the
object you pass in must be supported by the serializer: it must be a
MonoBehaviour, ScriptableObject, or plain class/struct with the
Serializable attribute applied.
In general, you'll use this to serialize MonoBehaviour objects, or a custom class/struct with the Serializable attribute.
YES! It is not supported currently but there is always a work around. Use this class :
public static class JsonHelper
{
public static T[] getJsonArray<T>(string json)
{
try
{
string newJson = "{ \"array\": " + json + "}";
Wrapper<T> wrapper = JsonUtility.FromJson<Wrapper<T>> (newJson);
return wrapper.array;
}
catch (Exception ex)
{
throw new Exception(ex.Message);
}
}
[Serializable]
private class Wrapper<T>
{
public T[] array = null;
}
}
I would use Json.NET
PlayerInfo[] allPlayersArray = new PlayerInfo[] { p1, p2, p3 };
string json = JsonConvert.SerializeObject(allPlayersArray);
More On SerializeObject
(Code example not yet tested)

Camel Velocity Template - access java object properties

I have a camel route which uses Velocity template and in the body I have an object defined as following:
class MailImpl extends AbstractMail{
private BodyContext bodyContext;
public BodyContext getBodyContext() {
return bodyContext;
}
public void setBodyContext(BodyContext bodyContext) {
this.bodyContext = bodyContext;
}
private String test;
public String getTest() {
return test;
}
public void setTest(String test) {
this.test = test;
}
#Override
public String toString() {
return "MailImpl{" +
"bodyContext=" + bodyContext +
'}';
}
}
class BodyContext{
private String value;
public BodyContext(String value) {
this.value = value;
}
public BodyContext() {
}
public String getValue() {
return value;
}
public void setValue(String value) {
this.value = value;
}
#Override
public String toString() {
return "BodyContext{" +
"value='" + value + '\'' +
'}';
}
In the velocity template I would like to access the MailImpl object properties, for example I use ${body.test} and ${body.bodyContext.value} but velocity template does not transform those values (it returns as string ${body.test} and ${body.bodyContext.value}).
One solution could be creating headers for each of of the value I need to use in the template, but as my route is dynamic (I select velocity template based on header) I would like to access the body properties in the velocity context. Is this somehow possible?
You can setup a custom Velocity Context by setting the message header "CamelVelocityContext" (since Camel v2.14). From Camel's test case:
Map<String, Object> variableMap = new HashMap<String, Object>();
Map<String, Object> headersMap = new HashMap<String, Object>();
headersMap.put("name", "Willem");
variableMap.put("headers", headersMap);
variableMap.put("body", "Monday");
variableMap.put("exchange", exchange);
VelocityContext velocityContext = new VelocityContext(variableMap);
exchange.getIn().setHeader(VelocityConstants.VELOCITY_CONTEXT, velocityContext);
exchange.setProperty("item", "7");
With following template:
Dear ${headers.name}. You ordered item ${exchange.properties.item} on ${body}.
You get:
Dear Willem. You ordered item 7 on Monday.

Resources