Using Flink 1.12.
My protobuf is
syntax = "proto3";
package flink.protobuf;
message TimestampedMessage {
int64 timeMs = 1;
string message = 2;
}
and tried to use it like so
final var env = StreamExecutionEnvironment.createLocalEnvironment();
env.getConfig().registerTypeWithKryoSerializer(TimestampedMessage.class, ProtobufSerializer.class);
env.fromCollection(new EventsIter(), TimestampedMessage.class)
...
But the logs show this
flink.protobuf.Test$TimestampedMessage does not contain a setter for field timeMs_
2021-08-12 06:38:19,940 INFO org.apache.flink.api.java.typeutils.TypeExtractor Class class
flink.protobuf.Test$TimestampedMessage cannot be used as a POJO type because not all fields are
valid POJO fields, and must be processed as GenericType. Please read the Flink
documentation on "Data Types & Serialization" for details of the effect on performance.
Seems like it is using the ProtobufSerializer despite the warning.
Related
I use the new typescript supported version of reactjs
along with the redux-orm and when u try to insert a value into the store i get this issue of "Maximum call stack size exceeded" the same works with the old template
Below is the code with new typescript supported reactjs which throws the error
https://reactjs.org/docs/static-type-checking.html#typescript and the old github version https://github.com/microsoft/TypeScript-React-Starter which works.
https://drive.google.com/open?id=15eolNjeYroyubgSmbGaaKfjxbe-IZ8KK
I am unable to understand what is different that causes the error with the new version. Thanks for any help.
Properties can't be defined directly on Model classes.
The root cause lies in the create-react-app's babel preset - transpilation output introduces additional property descriptors in Model prototype chain.
These properties interfere with descriptors installed by redux-orm during schema registration, resulting in Maximum call stack size exceeded error.
This can be avoided through declaration merging, specifically by providing a class and an interface with matching names. The interface contains Model properties, the class is almost 1:1 to JS Model definition.
Example:
interface Book {
id: number
title: string
}
class Book extends Model<typeof Book> {
static modelName = 'Book' as const
static fields = {
id: attr(),
title: attr()
}
}
I've created a repo with working example: https://github.com/tomasz-zablocki/redux-orm-ex-types-react-example
Mind you, these are custom types, but it's definitely where I'd like to take the #types/redux-orm package to, likely with the 0.14 release of redux-orm.
I am posting the following object
{
skillName : "Professional Skills"
_id : {$oid: "5adf23946ab671bf6cb36aff"}
}
to the DjangoService given below:
#csrf_exempt
#api_view(['GET','POST'])
def saveSubjectView(request): #this service will add & update Subject
if request.method == 'POST':
try:
stream = StringIO(request.body)
subject = JSONParser().parse(stream)
print("The subejct is ")
pp.pprint(subject)
serializedsubject = json.loads(json_util.dumps(subject))
print("serializedsubject")
pp.pprint(serializedsubject)
The output that I am getting is
'skillType': { u'_id': { }, u'skillName': u'Professional Skills'}
The ObjectId posted from the front end (AngularJS) is not printed in the service. I know that I can fix it by removing the $oid while posting from the AngularJS application. But I would like to know why this is not happening. I have searched the documents and I couldn't get a proper reply. May be the keywords I used are wrong. Keywords used are : "JSON serialisation of ObjectId", "$oid json serialization using Django".
The complete object I am posting to the Django service is given below:
Exactly. $oid or anything prefixed with $ is an internal format and reserved, so you cannot post field names. The convention is from MongoDB Extended JSON where such prefixes are used to identify the BSON Type for proper conversion, and used as a serializable transport since these "types" are not supported in basic JSON.
So the solution is to actually use the bson.json_util to "deserialize" the JSON string right from the start:
from bson import json_util
# serializedsubject = json.loads(json_util.dumps(subject))
serializedsubject = json_util.loads(request.body) # correct usage
Or more succinctly self contained:
input = '{ "skillName" : "Professional Skills" ,"_id" : { "$oid": "5adf23946ab671bf6cb36aff"} }'
json_util.loads(input)
Returns
{u'skillName': u'Professional Skills', u'_id': ObjectId('5adf23946ab671bf6cb36aff')}
This correctly casts objects from any keys notated with the Extended JSON Syntax to their correct BSON Type, as also supported in the driver functions. And naturally the driver will then convert back to BSON when sending to MongoDB.
If for some reason your request.body contains anything other than a "string" which is valid for input to the function, then it is up to your code to convert it to that point. But there should be no need to "parse to JSON" and then "stringify" again just to input to the function.
NOTE: If you have not already done so within your JavaScript client side of the application, there is also the bson package available. This would allow where such Extended JSON is "received" from the server the translation into the BSON Types as JavaScript Objects, and of course then the serialization of such objects back into the Extended JSON Format.
This would in fact be recommended where "type" information needs to be maintained with the data transmitted and kept between client and server.
I am using the Table API on Flink 1.4.0. I have some Table objects to be convert to a DataSet of type Row. The project was built using Maven and imported on IntelliJ. I have the following code and the IDE cannot resolve the method tableenv.toDataSet() method. Please help me out. Thank you.
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
BatchTableEnvironment tableEnvironment = TableEnvironment.getTableEnvironment(env);
...
tableEnvironment.registerTableSource("table1",csvSource);
Table table1 = tableEnvironment.scan("table1");
DataSet<Row> result = tableEnvironment.toDataSet(table1, Row.class);
The last statement causes an error
"Cannot resolve toDataSet() method"
You might not import the right BatchTableEnvironment.
Please check that you import org.apache.flink.table.api.java.BatchTableEnvironment instead of org.apache.flink.table.api.BatchTableEnvironment. The former is the common base class for the Java and Scala variants.
If you want to read a DataSet from a csv file, do it like following:
DataSet<YourType> csvInput = env.readCsvFile("hdfs:///the/CSV/file") ...
More on this: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/batch/#data-sources
I'm trying to serialize/deserialize ActorRef through protobuf. According to the Akka doc, the only way to do it is to convert the ActorRef into a String, and convert it back in the remote actor system.
The doc mentions using an ExtendedActorSystem to do the deserialization (see here). However, it is unclear how to get the ExtendedActorSystem:
// Serialize
// (beneath toBinary)
val identifier: String = Serialization.serializedActorPath(theActorRef)
// Then just serialize the identifier however you like
// Deserialize
// (beneath fromBinary)
// ==== Where is this extendedSystem from? ====
val deserializedActorRef = extendedSystem.provider.resolveActorRef(identifier)
// Then just use the ActorRef
Edit
I found this question here: Akka (JVM): Serialize an actorref with protobuf within another message, which mentions casting an ActorSystem to ExtendedActorSystem. Is this the right approach? Will it always work?
dear #stackoverflower,
whenever you use ActorSystem(...) it build an instance of ActorSystemImpl.
The type-tree looks like:
ActorSystemImpl extends ExtendedActorSystem
and
ExtendedActorSystem implements ActorSystem
you can use statements like
val system: ExtendedActorSystem = ActorSystem(...).asInstanceOf[ExtendedActorSystem]
to access the correct type autocomplete. ActorSystemImpl is unfortunately scoped to [akka].
I'm absolutely loving BreezeJS and was so surprised to see that my Enum values were being displayed as their text values and not their ordinal ones! What I would love to be able to do is, on the client side, open the MetadataStore, fetch the enumeration and modify it's textual properties for display purposes.
Is this currently possible? From my research it would appear not, but I'm wondering if there is perhaps a simple workaround. Everything I've tried has involved a large number of hacks and server-side attributes, but to no avail, or the solution just seemed overly complex and not worth the benefits.
Here is what I said about how to get the enum values from raw metadata in another SO comment.
The enum values are available in the metadata generated by EF and sent to a Breeze client. I agree Breeze should pick them up automatically and put them somewhere so you don't have to make a separate request for them or extract them from the raw metadata passed to the MetadataStore.fetchMetadata success callback. It's on our backlog.
Meanwhile, you'll have to get them by hand. There is a DocCode test that shows how:
/*********************************************************
* Can can get enum types from raw EF-generated metadata
* Related to feature request #2271: Extract enum values from server metadata
*************************************************************/
asyncTest("can get enum type from raw EF-generated metadata", function() {
var serviceName = testFns.foosMetadataServiceName;
var store = new breeze.MetadataStore();
store.fetchMetadata(serviceName)
.then(metaSuccess, metaFail).fail(handleFail).fin(start);
function metaSuccess(rawMetadata) {
ok(true, "foos metadata fetched");
var enumType = rawMetadata.schema && rawMetadata.schema.enumType;
if (enumType && enumType.name ==='Color') {
var members = enumType.member;
ok(members.length,
"should have several members; members are: " + JSON.stringify(members));
} else {
ok(false, "metadata should have had one enumType, 'Color'.");
}
}
function metaFail(error) {
ok(false, "foos metadata fetch failed: " + error.message);
}
});
we're using Breeze with NHibernate and wand to do the same. Metadata generated for NHibernate are standard breeze Metadata and doesn't contain the Schema part. Any ideas how to do it?
edit: To fix our issue I added a list of all used enums in the metadata(like the structuralTypes node). Then we cache it on the client when the metadata are retrieved.
https://github.com/lnu/breeze.server.net/commit/65ad687ad13c4dd9f4a6ab6a3ed09e407e2b58ec
Thanks