Here DataStream returns keyvalue pair as a object i need key value directly not as a object becoz i need to group the values based on key.
DataStream<ObjectNode> stream = env
.addSource(new FlinkKafkaConsumer<>("test5", new JSONKeyValueDeserializationSchema (false), properties));
// stream.keyBy("record1").print();
when i give stream.keyby("record1").print();
it shows
Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: This type (GenericType<org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode>) cannot be used as key.
at org.apache.flink.api.common.operators.Keys$ExpressionKeys.<init>(Keys.java:330)
at org.apache.flink.streaming.api.datastream.DataStream.keyBy(DataStream.java:337)
at ReadFromKafka.main(ReadFromKafka.java:27)
David Anderson's response is correct, as an addition, I can add that You can simply create the KeySelector that will extract the key as String. It could look like this:
public class JsonKeySelector implements KeySelector<ObjectNode, String> {
#Override
public String getKey(ObjectNode jsonNodes) throws Exception {
return jsonNodes.get("key").asText();
}
}
This obviously assumes that the Key is supposed to be String.
There are several ways to specify the key selector in a Flink keyBy. For example, if you have a POJO of type Event with the String key in a field named "id", any of these will work:
stream.keyBy("id")
stream.keyBy(event -> event.id)
stream.keyBy(
new KeySelector<Event, String>() {
#Override
public String getKey(Event event) throws Exception {
return event.id;
}
}
)
So long as you can compute the key from the object in a deterministic way, you can make this work.
Related
I've implemented a RichFunction with following type:
RichMapFunction<GeofenceEvent, OutputRangeSet>
the class OutputRangeSet has a field of type:
com.google.common.collect.RangeSet<Instant>
When this pojo is serialized using Kryo I get null fields !
So far, I tried using a TypeInfoFactory<RangeSet>:
public class InstantRangeSetTypeInfo extends TypeInfoFactory<RangeSet<Instant>> {
#Override
public TypeInformation<RangeSet<Instant>> createTypeInfo(Type t, Map<String, TypeInformation<?>> genericParameters) {
TypeInformation<RangeSet<Instant>> info = TypeInformation.of(new TypeHint<RangeSet<Instant>>() {});
return info;
}
}
That annotate my field:
public class OutputRangeSet implements Serializable {
private String key;
#TypeInfo(InstantRangeSetTypeInfo.class)
private RangeSet<Instant> rangeSet;
}
Another solution (that doesn't work either) is registring a third party serializer:
env.getConfig().registerTypeWithKryoSerializer(RangeSet.class, ProtobufSerializer.class);
You can get the github project here:
https://github.com/elarbikonta/tonl-events
When you run the test you can see (in debug) that the rangeSet beans I get from my RichFunction has null fields, see test method com.tonl.apps.events.IsVehicleInZoneTest#operatorChronograph :
final RangeSet<Instant> rangeSet = resultList.get(0).getRangeSet(); // rangetSet.ranges = null !
Thanks for your help
I am new to Flink and doing something very similar to the below link.
Cannot see message while sinking kafka stream and cannot see print message in flink 1.2
I am also trying to add JSONDeserializationSchema() as a deserializer for my Kafka input JSON message which is without a key.
But I found JSONDeserializationSchema() is not present.
Please let me know if I am doing anything wrong.
JSONDeserializationSchema was removed in Flink 1.8, after having been deprecated earlier.
The recommended approach is to write a deserializer that implements DeserializationSchema<T>. Here's an example, which I've copied from the Flink Operations Playground:
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
/**
* A Kafka {#link DeserializationSchema} to deserialize {#link ClickEvent}s from JSON.
*
*/
public class ClickEventDeserializationSchema implements DeserializationSchema<ClickEvent> {
private static final long serialVersionUID = 1L;
private static final ObjectMapper objectMapper = new ObjectMapper();
#Override
public ClickEvent deserialize(byte[] message) throws IOException {
return objectMapper.readValue(message, ClickEvent.class);
}
#Override
public boolean isEndOfStream(ClickEvent nextElement) {
return false;
}
#Override
public TypeInformation<ClickEvent> getProducedType() {
return TypeInformation.of(ClickEvent.class);
}
}
For a Kafka producer you'll want to implement KafkaSerializationSchema<T>, and you'll find examples of that in that same project.
To solve the problem of reading non-key JSON messages from Kafka I used case class and JSON parser.
The following code makes a case class and parses the JSON field using play API.
import play.api.libs.json.JsValue
object CustomerModel {
def readElement(jsonElement: JsValue): Customer = {
val id = (jsonElement \ "id").get.toString().toInt
val name = (jsonElement \ "name").get.toString()
Customer(id,name)
}
case class Customer(id: Int, name: String)
}
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val properties = new Properties()
properties.setProperty("bootstrap.servers", "xxx.xxx.0.114:9092")
properties.setProperty("group.id", "test-grp")
val consumer = new FlinkKafkaConsumer[String]("customer", new SimpleStringSchema(), properties)
val stream1 = env.addSource(consumer).rebalance
val stream2:DataStream[Customer]= stream1.map( str =>{Try(CustomerModel.readElement(Json.parse(str))).getOrElse(Customer(0,Try(CustomerModel.readElement(Json.parse(str))).toString))
})
stream2.print("stream2")
env.execute("This is Kafka+Flink")
}
The Try method lets you overcome the exception thrown while parsing the data
and returns the exception in one of the fields (if we want) or else it can just return the case class object with any given or default fields.
The sample output of the Code is:
stream2:1> Customer(1,"Thanh")
stream2:1> Customer(5,"Huy")
stream2:3> Customer(0,Failure(com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input
at [Source: ; line: 1, column: 0]))
I am not sure if it is the best approach but it is working for me as of now.
Below is my code for entity and a function where I need to map entity TblEmployee from a key value pair.
In foreach loop I am getting values based on keys, what should be the best approach to do it?
public class TblEmployee
{
public int EmployeeId { get; set; }
public string Name { get; set; }
}
public int Create()
{
tblEmployee employee = new tblEmployee();
using (var ctx = new theparkeee_testEntities())
{
foreach (string key in HttpContext.Current.Request.Form.AllKeys)
{
string value = HttpContext.Current.Request.Form[key];
//how to map value from key value pair to entity employee.
}
}
}
You can use System.Reflection to get the Properties of an object by their name with Type.GetProperty(string name). After you got the PropertyInfo, you can use SetValue to assign a value to it.
foreach (string key in HttpContext.Current.Request.Form.AllKeys) {
// note that "value" is a reserved word, do not use it as variable name
string val = HttpContext.Current.Request.Form[key];
var propertyInfo = typeof(TblEmployee).GetProperty(key); // can maybe be moved outside of the loop
if (propertyInfo != null) {
propertyInfo.SetValue(employee, val);
}
}
This will work for string properties. If the property is of another type, you have to find the correct type (again, using reflection) and cast the string value before assigning it.
Note that this is not the correct approach to store data in MVC. You should not work with the Request.Form directly, instead your POST action should accept a ViewModel that can be mapped (e.g. using Automapper) to the DB entity. I.e. let the ASP ModelBinder do its work, instead of reinventing the wheel!
[HttpPost]
public ActionResult Submit(MyViewModel postData) {
var employee = Mapper.Map<TblEmployee>(postData);
_ctx.Employees.Add(employee);
_ctx.SaveChanges();
return new HttpStatusCodeResult((int)HttpStatusCode.OK);
}
This has baffled me for a while now and I cannot seem to get the grasp of it. I'm using Cell Value Factory to populate a simple one column table and it does not populate in the table.
It does and I click the rows that are populated but I do not see any values in them- in this case String values. [I just edited this to make it clearer]
I have a different project under which it works under the same kind of data model. What am I doing wrong?
Here's the code. The commented code at the end seems to work though. I've checked to see if the usual mistakes- creating a new column instance or a new tableview instance, are there. Nothing. Please help!
//Simple Data Model
Stock.java
public class Stock {
private SimpleStringProperty stockTicker;
public Stock(String stockTicker) {
this.stockTicker = new SimpleStringProperty(stockTicker);
}
public String getstockTicker() {
return stockTicker.get();
}
public void setstockTicker(String stockticker) {
stockTicker.set(stockticker);
}
}
//Controller class
MainGuiController.java
private ObservableList<Stock> data;
#FXML
private TableView<Stock> stockTableView;// = new TableView<>(data);
#FXML
private TableColumn<Stock, String> tickerCol;
private void setTickersToCol() {
try {
Statement stmt = conn.createStatement();//conn is defined and works
ResultSet rsltset = stmt.executeQuery("SELECT ticker FROM tickerlist order by ticker");
data = FXCollections.observableArrayList();
Stock stockInstance;
while (rsltset.next()) {
stockInstance = new Stock(rsltset.getString(1).toUpperCase());
data.add(stockInstance);
}
} catch (SQLException ex) {
Logger.getLogger(WriteToFile.class.getName()).log(Level.SEVERE, null, ex);
System.out.println("Connection Failed! Check output console");
}
tickerCol.setCellValueFactory(new PropertyValueFactory<Stock,String>("stockTicker"));
stockTableView.setItems(data);
}
/*THIS, ON THE OTHER HAND, WORKS*/
/*Callback<CellDataFeatures<Stock, String>, ObservableValue<String>> cellDataFeat =
new Callback<CellDataFeatures<Stock, String>, ObservableValue<String>>() {
#Override
public ObservableValue<String> call(CellDataFeatures<Stock, String> p) {
return new SimpleStringProperty(p.getValue().getstockTicker());
}
};*/
Suggested solution (use a Lambda, not a PropertyValueFactory)
Instead of:
aColumn.setCellValueFactory(new PropertyValueFactory<Appointment,LocalDate>("date"));
Write:
aColumn.setCellValueFactory(cellData -> cellData.getValue().dateProperty());
For more information, see this answer:
Java: setCellValuefactory; Lambda vs. PropertyValueFactory; advantages/disadvantages
Solution using PropertyValueFactory
The lambda solution outlined above is preferred, but if you wish to use PropertyValueFactory, this alternate solution provides information on that.
How to Fix It
The case of your getter and setter methods are wrong.
getstockTicker should be getStockTicker
setstockTicker should be setStockTicker
Some Background Information
Your PropertyValueFactory remains the same with:
new PropertyValueFactory<Stock,String>("stockTicker")
The naming convention will seem more obvious when you also add a property accessor to your Stock class:
public class Stock {
private SimpleStringProperty stockTicker;
public Stock(String stockTicker) {
this.stockTicker = new SimpleStringProperty(stockTicker);
}
public String getStockTicker() {
return stockTicker.get();
}
public void setStockTicker(String stockticker) {
stockTicker.set(stockticker);
}
public StringProperty stockTickerProperty() {
return stockTicker;
}
}
The PropertyValueFactory uses reflection to find the relevant accessors (these should be public). First, it will try to use the stockTickerProperty accessor and, if that is not present fall back to getters and setters. Providing a property accessor is recommended as then you will automatically enable your table to observe the property in the underlying model, dynamically updating its data as the underlying model changes.
put the Getter and Setter method in you data class for all the elements.
Hi I have some tuple Tuple2<String, Integer> that i want to convert to string an then send it to KAFKA.
Im trying to figure out a way to to iterate the tuple and create one string from it so if i have N elements in my tuple i want to create a string that contain them.
I tried flat map it but im geting new string for each element in the tuple.
SingleOutputStreamOperator<String> s = t.flatMap(new FlatMapFunction<Tuple2<String, Integer>, String>() {
#Override
public void flatMap(Tuple2<String, Integer> stringIntegerTuple2, Collector<String> collector) throws Exception {
collector.collect(stringIntegerTuple2.f0 + stringIntegerTuple2.f1);
}
});
What is the correct way to create on string out of tuple .
You can just override the .toString() method of a tuple with a custom class and use that. Like this:
import org.apache.flink.api.java.tuple.Tuple3;
public class CustomTuple3 extends Tuple3 {
#Override
public String toString(){
return "measurment color=" + this.f0.toString() + " color=" + this.f1.toString() + " color=" + this.f2.toString();
}
}
So now just use a CustomTuple3 object instead of a Tuple3 and when you populate it and call .toString() on it, it will output that formatted string.