For datasource, I listen to multiple topics, when I receive a source event, how can I know which topic this event consumed from, since topic name contains some information I need (e.g. country prefix)
I was trying to read the source code of flink KafkaConsumer but didn't find any method I can use to get topic name from an event.
If you're listening to different topics with different names and you need that information, you should probably enrich your events with the information you need with a simple map.
For instance if your topic contains events with fields like "username" and "timestamp"
{
"username": "diego",
"timestamp": 1649085282804
}
You can listen to your source and make a map right away after each source to add the information you need to each event.
val enrichedDataStream: Datastream[CountryCodeEnrichedEvent] = kafkaSourceCanada.map(event => CountryCodeEnrichedEvent(username: event.username, timestamp: event.timestamp, countryCode: "ca"))
You'll end up with something like this, for each kafka source you're listening to.
{
"username": "diego",
"timestamp": 1649085282804,
"countryCode": "ca"
}
The topic is part of the ConsumerRecord that is passed to the deserialize method of a KafkaDeserializationSchema. So if you want to access this or other Kafka metadata, you'll need to implement your own deserializer.
https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/streaming/connectors/kafka/KafkaDeserializationSchema.html
Related
What I've gathered is that new posts are published by POSTing a JSON-LD Activity Streams object of type Note to an actor's outbox.
{"#context": "https://www.w3.org/ns/activitystreams",
"type": "Note",
"to": ["https://chatty.example/ben/"],
"attributedTo": "https://social.example/alyssa/",
"content": "Say, did you finish reading that book I lent you?"}
The server will then have wrap it into an activity of type Create.
{"#context": "https://www.w3.org/ns/activitystreams",
"type": "Create",
"id": "https://social.example/alyssa/posts/a29a6843-9feb-4c74-a7f7-081b9c9201d3",
"to": ["https://chatty.example/ben/"],
"actor": "https://social.example/alyssa/",
"object": {"type": "Note",
"id": "https://social.example/alyssa/posts/49e2d03d-b53a-4c4c-a95c-94a6abf45a19",
"attributedTo": "https://social.example/alyssa/",
"to": ["https://chatty.example/ben/"],
"content": "Say, did you finish reading that book I lent you?"}}
I fail to see the usefulness of this, as the wrapping activity doesn't seem to add any useful data to the wrapped note. Worse even, it seems like it might introduce a fair bit of redundancy to the responses (in this basic example from the official page, actor and attributedTo, as well as the 2 to fields, have exactly the same purpose). Is this perhaps done just for consistency, as there are a few other other activity types that are applied to notes, and for newly created posts having just a plain object (or a collection of plain objects) as a response would not fit this way of doing things?
Also, why are other activity types (e.g., Like) able to simply reference notes by id, while Create activities enclose that data directly? Is that required or is there a specific reason for it?
{"#context": "https://www.w3.org/ns/activitystreams",
"type": "Like",
"id": "https://social.example/alyssa/posts/5312e10e-5110-42e5-a09b-934882b3ecec",
"to": ["https://chatty.example/ben/"],
"actor": "https://social.example/alyssa/",
"object": "https://chatty.example/ben/p/51086"}
ActivityStreams is a protocole to synchronise datas between different databases and softwares.
Some actions contain the detail (like for Notes) because we can create/update them and an activity stream reader must know the changes to display the good datas to its users.
Some operations like Like have a named cancel operation like Unlike. So we don't need the Create/Update container. and because they operate on existing datas, we also only need the unique ID to the concerned resource.
I am quite new to facebook chat bot.
https://developers.facebook.com/docs/messenger-platform/send-messages/buttons/
I can define button like this.
{
"type": "postback",
"title": "<BUTTON_TEXT>",
"payload": "<STRING_SENT_TO_WEBHOOK>"
}
I have different payload (E.g For course, student). How can I put ID information inside that button as additional information?
Do I have to define like course_1, course_2, student_5, student_8 in order to send type and ID?
The payload can contain a string up to 1000 characters, what you pass as payload is up to you. Looking at your example you could use something like "course_1" or a combination like "course_1, student_8".
I have a JSON which has a format as given below:
{
"header": {
"source_code": "S12345"
"user_id": "987456"
},
"body":{
"source_code": "S12345",
"wrapper_list": [
{
"item_wrapper_code": "WRAP01",
"item_amount": 10,
"item_type_amount": 2,
"creation_date": 20191115,
"worker": "W001",
"workstation_no": "1"
"item_list":[
{
"item_code": "I001"
"item_id": "",
"bar_code": "123987456"
"remark": ""
},
{
"item_code": "I002"
"item_id": "",
"bar_code": "213987456"
"remark": ""
}
]
}
]
}
}
The body of the above JSON object contains data in a parent - child relationship.
Parent: WRAP01
Child1: Item I001
Child2: Item I002
I must be able to parse this JSON object and insert these details as rows into respective tables, i.e., Parent related stuff into Parent table and Children related stuff into Child table.
I request professionals here to kindly guide me.
So, parsing Json to Entity Framwork objects is bad practise.
Firstly, create data transfer objects (DTO) that will be maps to Json.
It is like:
RequestDto, BodyDto, WrapperDto, WrapperItemDto.
For parse it you can use something like newtonsoft, see example: https://www.newtonsoft.com/json/help/html/DeserializeObject.htm
Then you should write EntityFramework objects that will be maps to Database tables.
I hope, you knows Databases theory and already reads some articles about EntityFramework and how it implements different DB relationsips. If no, so please read about it.
You needs One-To-Many relationship. The one-to-many relationship can be configured in the following ways.
By following Conventions
By using Fluent API Configurations
You can use first way. Its more easy. So you need 2 classes: WrapperEntity, WrapperItemEntity.
Then, you should write code that will convert WrapperDto to WrapperEntity and WrapperItemDto to WrapperItemEntity. And then save *Entity objects in database.
It's a best practise for implements such logic. I explain it as top level. For details you can google each question separately.
I'm trying to apply HATEOAS to the existing application and I'm having trouble with modeling a form inputs that would be driven by the API response.
The app is allowing to search & book connections between two places. First endpoint allows for searching the connections GET /connections?from={lat,lon}&to={lat,lon}&departure={dateTime} and returns following payload (response body).
[
{
"id": "aaa",
"carrier": "Fast Bus",
"price": 3.20,
"departure": "2019-04-05T12:30"
},
{
"id": "bbb",
"carrier": "Airport Bus",
"price": 4.60,
"departure": "2019-04-05T13:30"
},
{
"id": "ccc",
"carrier": "Slow bus",
"price": 1.60,
"departure": "2019-04-05T11:30"
}
]
In order to make an order for one of connections, the client needs to make a POST /orders request with one of following payloads (request body):
email required
{
"connectionId": "aaa",
"email": "passenger#example.org"
}
email & flight number required (carrier handles only aiprort connections)
{
"connectionId": "bbb",
"email": "passenger#example.org",
"flightNumber": "EA1234"
}
phone number required
{
"connectionId": "ccc",
"phoneNumber": "+44 111 222 333"
}
The payload is different, because different connections may be handled by different carriers and each of them may require some different set of information to provide. I would like to inform the API client, what fields are required when creating an order. The question I have is how do I do this with HATEOAS?
I checked different specs and this is what I could tell from reading the specs:
HAL & HAL-FORMS There are "_templates" but, there is no URI in the template itself. It’s presumed to operate on the self link, which in my case would be /connections... not /orders.
JSON-LD I couldn't find anything about forms or templates support.
JSON-API I couldn't find anything about forms or templates support.
Collection+JSON There is at most one "template" per document, therefore it's presumed that all elements of the collection have the same fields which is not the case in my app.
Siren Looks like the "actions" would fit my use case, but the project seems dead and there are no supporting libraries for many major languages.
CPHL The project seems dead, very little documentation and no libraries.
Ion There is nice support for forms, but I couldn't find any supporting libraries. Looks like it's just a spec for now.
Is such a common problem as having forms driven by the API still unsolved with spec and tooling?
In your example, it appears that Connections are resources. It's not completely clear if Orders are truly resources. I'm guessing probably yes, but to have an Order you need a Client and Connection. So, to create an Order you will need to expose a collection, likely from the Client or Connection, possibly both.
I think the disconnect is from thinking along the lines of "now that we've got a list of available connections, the client can select one and create an Order." That's perfectly valid, but it's remote procedure call (RPC) thinking, not REST. Neither is objectively better than the other, except in the context of a particular set of project requirements, and generally they shouldn't be mixed together.
With an RPC mindset, a create order method is defined (e.g. using OpenAPI) and any clients are expected to use some out-of-band information to determine the correct form required (i.e. by reading the OpenAPI spec).
With a REST/HATEOAS mindset, the correct approach would be to expose a Orders collection from Connection. Each Connection in the collection has a self link and a Orders collection (link or object, as defined by app requirements). Each item of Order has a self link, and that is where the affordances are specified. An Order is a known type (even with REST/HATEOAS the client and service have to at least agree on a shared vocabulary) that the client presumably knows how to define. That vocabulary can be defined using any mechanism that works -- json-ld, XSD, etc.
HATEOAS requires that the result contains everything the client needs to update the state. There can be no out-of-band information (other than the shared vocabulary). So, to solve your issue, you either need to expose a collection of Orders from Connection or you need to allow an Order to be created by posting to Connection. If the latter seems like a bit of a hack, it probably is.
For example, in HAL-Forms, I would do something like:
{
"connections": [{
"id": "aaa",
"carrier": "Fast Bus",
"price": 3.20,
"departure": "2019-04-05T12:30"
"_links": {
"self": { ... }, // link to this connection
"orders": {} // link to collection of orders for this connection
}
},
, ...],
"_links": {
"self": { ... } // link to the collection
},
"_templates": { ... } // post/put/patch/delete connection
}
Clients would follow the links to orders and from there would get the _templates collection that contains the instructions for managing the Order resources. The Order POST would likely require a connection identifier and client information. The HAL-Forms Spec defines a regex property that can be used to specify the type of data to supply for any particular form element. Since you have reached the order by navigating through a specific connection, you would be able to specify in your _templates for that order exactly which fields are required. e.g. /orders?connectionType=aaa would return a different set of required properties than /orders?connectionType=bbb but both use the same self link of /orders?connectionType={type} and you'd validate it on POST/PUT/PATCH.
I should note that the Spring-HATEOAS goes beyond the HAL-Forms spec and allows for multiple _links and _templates. See this GitHub issue.
It may look like HATEOAS/REST requires quite a bit more work than a simple OpenAPI/RPC API and it does. But what you are giving up in simplicity, you are gaining in flexibility and resilience, assuming well-designed clients. Which approach is correct depends on a lot of factors, most of them not technical (team skills, expected consumers, how much control you have over clients, maintenance, etc.).
We are a group of people writing a bachelor-project about storing sensor data into a noSQL-database, and we have chosen couchbase for this.
We want to store quite a few data in the same document, one document per day, per sensor, and we want to append new sensor data witch comes in every minute.
But unforunatly, we are not able to append new data into existing document without overwriting the existing data.
The structure for the documents is:
DocumentID: Sensor + date, ie: KitchenTemperature20180227
{
"topic": "Kitchen/Temp",
"type": "temperature",
"unit": "DegC"
"20180227130400": [
{
"data": "24"
}
],
..............
"20180227130500": [
{
"data": "25"
}
],
}
We are all new to couchbase and NoSql-databases, but eager to learn and understand how we the best way should implemet this.
We've tried upsert, insert and update commands, but they all overwrite the existing document or won't execute because the document already exists. As you can see, we have some top-level information, like topic, type, unit. The rest should be data coming in every minute and appended to the existing document.
Help on how to proceed would be very appriciated.
Best regards, Kenneth
In this case you can use the subdocument API. This allows you to modify portions of a document based on a "path". This image gives the idea for getting a subdocument.
You can mutate subdocuments as well. Look at the subdocument API documentation for Couchbase. There are also blog posts that go through examples in Java and Go on the Couchbase blog site.