I am reading at https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/upsert-kafka/.
It says that:
As a sink, the upsert-kafka connector can consume a changelog stream.
It will write INSERT/UPDATE_AFTER data as normal Kafka messages value,
and write DELETE data as Kafka messages with null values (indicate
tombstone for the key).
It doesn't mention that if UPDATE_BEFORE message is written to upsert kafka,then what would happen?
In the same link (https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/upsert-kafka/#full-example), the doc provides a full example:
INSERT INTO pageviews_per_region
SELECT
user_region,
COUNT(*),
COUNT(DISTINCT user_id)
FROM pageviews
GROUP BY user_region;
With the above INSERT/SELECT operation, INSERT/UPDATE_BEFORE/UPDATE_AFTER messages will be generated and will go to the upsert kafka sink, I would ask what happens when upsert kafka meets the UPDATE_BEFORE message.
From the comments on the source code
/ / partial code
// During the Upsert mode during the serialization process, if the operation is executed is Rowkind.delete or Rowkind.Update_before
// set it to NULL (corresponding to Kafka tomb news)
https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=165221669#content/view/165221669
Upsert-kafka sink doesn’t require planner to send UPDATE_BEFORE messages (planner may still send UPDATE_BEFORE messages in some cases), and will write INSERT/UPDATE_AFTER messages as normal Kafka records with key parts, and will write DELETE messages as Kafka records with null values (indicate tombstone for the key). Flink will guarantee the message ordering on the primary key by partition data on the values of the primary key columns.
Upsert-kafka source is a kind of changelog source. The primary key semantics on changelog source means the materialized changelogs (INSERT/UPDATE_BEFORE/UPDATE_AFTER/DELETE) are unique on the primary key constraints. Flink assumes all messages are in order on the primary key.
Implementation Details
Due to the upsert-kafka connector only produces upsert stream which doesn’t contain UPDATE_BEFORE messages. However, several operations require the UPDATE_BEFORE messages for correctly processing, e.g. aggregations. Therefore, we need to have a physical node to materialize the upsert stream and generate changelog stream with full change messages. In the physical operator, we will use state to know whether the key is the first time to be seen. The operator will produce INSERT rows, or additionally generate UPDATE_BEFORE rows for the previous image, or produce DELETE rows with all columns filled with values.
Related
I have to implement a use case where I have to insert data (source is a Kafka topic) into multiple postgres tables (around 10 tables) in a transactional way i.e either all table inserts happen or if there is a failure in one insert, all the inserts for that particular record are failed. The insert failed records should also be captured and written into another Kafka topic.
Based on my understanding of the JDBC sink implementation, we can only provide one prepared statement per sink. Also, according to the invoke method signature of the GenericJdbcSinkFunction class -
public void invoke(T value, SinkFunction.Context context) throws IOException
It only throws an IOException. Is it possible to catch a SQL Exception having a failed insert and then writing that record in a separate Kafka topic? If yes, what happens to the rest of the records in that batch because if one record insert fails, I believe the whole batch fails.
Is it a good idea to use Flink for such use case?
I am doing a left outer join on two tables in flink, and the code is given below, showing an exception, inner join with the same two tables worked fine and was able to convert to Datastream
Table table = customerTable.leftOuterJoin(contactTable,$("cust_custcode")
.isEqual($("contact_custcode")))
.select($("customermessage"), $("contactmessage"));
The exception is : org.apache.flink.table.api.TableException: Table sink 'anonymous_datastream_sink$3' doesn't support consuming update and delete changes which is produced by node Join(joinType=[LeftOuterJoin], where=[(f0 = f00)], select=[f0, f1, f00, f10], leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey])
When executed in streaming mode, some Flink SQL queries produce an output stream than the planner knows will only need to INSERT rows into the sink, while other queries produce an output stream that sometimes needs to UPDATE previously emitted results.
Some sinks cannot accept UPDATE streams -- including the one you are using. You'll need to either (1) change your query (e.g., by doing a windowed join), (2) use a different sink (e.g., JDBC can accept updates), or (3) write to the sink in a different format (e.g., a CDC format like debezium).
We have a scenario, where each insert happen per id_2 given id_1, for below schema, in Cassandra:
CREATE TABLE IF NOT EXISTS my_table (
id_1 UUID,
id_2 UUID,
textDetails TEXT,
PRIMARY KEY (id_1, id_2)
);
A single POST request body has the details for multiple values of id_2. This triggers multiple inserts per single POST request on single table.
Each INSERT query is performed as shown below:
insertQueryString = "INSERT INTO my_table (id_1, id_2, textDetails) " + "VALUES (?, ?, ?) IF NOT EXISTS"
cassandra.Session.Query(insertQueryString,
id1,
id2,
myTextDetails).Exec();
1
Does Cassandra ensure data consistency on multiple inserts on a single table, per POST request? Each POST request is processed on a Go-routine(thread). Subsequent GET requests should ensure retrieving consistent data(inserted through POST)
Using BATCH statements is having "Batch too large" issues in staging & production. https://github.com/RBMHTechnology/eventuate/issues/166
2
We have two data centres(for Cassandra), with 3 replica nodes per data center.
What are the consistency levels need to set for write query operation(POST request) and ready query operation(GET request), to ensure full consistency
There are multiple problems here:
Batching should be used very carefully in Cassandra - only if you're inserting data into the same partition. If you insert data into multiple partitions, then it's better to use separate queries executed in parallel (but you can collect multiple entries per partition key and batch them).
you're using IF NOT EXISTS and it's done against the same partition - as result it leads to the conflicts between multiple nodes (see documentation on lightweight transactions) plus it requires reading data from disk, so it heavily increase the load onto the nodes. But do you really need to insert data only if the row doesn't exist? What is the problem if row exists already? It's easier just to overwrite data in Cassandra when doing INSERT because it won't require reading data from the disk.
Regarding consistency level - the QUORUM (or SERIAL for LWTs) will give you the strong consistency but at expense of the increased latency (because you need to wait for answer from another DC), and lack of fault tolerance - if you lose another DC, then all your queries will fail. In most cases the LOCAL_QUORUM is enough (LOCAL_SERIAL in case of LWTs), and it will provide fault tolerance. I recommend to read this whitepaper on best practices of build fault-tolerance applications on top of Cassandra.
i have an application, where in for every request on reaching the application server, all its attributes are stored in a particular table with a unique request Id. And on completion of the processing of the request(i.e., on generation of the response but before sending the response to the client), based on the unique request id, the corresponding row is deleted from the table. Now, an application on an average can process a very large of number of requests. So this means a huge amount of insert operations and then corresponding delete operations on a single table.Will it affect the database performance and its stability ?
I have a couple of million rows in a postgresql table. I have up to 20 proceeses writing to that table (a few hundred inserts/updates per second) and I have a few processes reading from it at the same time (once a while a big select). This results in many failures (Stream Closed, Input/Ouput Error) on both sides, reading and writing.
I now think about splitting that table into multiple tables. I would split by "type" of object, which is basically a field that has only 20 possible values that are kind of equally distributed.
The question is, should I use multiple tables, multiple schemas or multiple databases to guarantee a non blocking access to the data. Or maybe I should use a completly different setup. Another database maybe? Maybe HTable?
The integrity of the data is not that important. It has to be there in the end but I do not really need an Isolation Level or Transactions. I just need a fast system that can write and read from multiple processes without performance impact and that allows to make queries based on field values.
Right now I use JDBC with Isolation Level TRANSACTION_READ_UNCOMMITTED and a connection per process.
UPDATE:
The schema looks as follows:
CREATE TABLE rev
(
id integer NOT NULL,
source text,
date timestamp with time zone,
title text,
summary text,
md5sum text,
author text,
content text,
CONSTRAINT rev_id_pk PRIMARY KEY (id),
CONSTRAINT md5sum_un UNIQUE (md5sum)
)
CREATE TABLE resp
(
id integer NOT NULL,
source text,
date timestamp with time zone,
title text,
summary text,
md5sum text,
author text,
content text,
CONSTRAINT resp_id_pk PRIMARY KEY (id),
CONSTRAINT md5sum_un UNIQUE (md5sum)
)
And I have a few indexes on some of the fields.
A sample query looks like:
SELECT * FROM rev LEFT JOIN resp ON rev.id = resp.parent_id WHERE rev.date > ? LIMIT 1000 OFFSET ?
The resp table is much smaller, but it too gets updates and is queried in the joins.
This results in many failures on both sides, reading and writing.
What kind of failures? Reading and writing on the same table should not be a problem at all in PostgreSQL, MVCC works fine.
Hard to tell you how to fix your problems without any information about the system and what the processes are doing. Could you tell us more about it? And show a database schema?
Right now I use JDBC with Isolation Level TRANSACTION_READ_UNCOMMITTED
READ UNCOMMITTED doesn't exist in PostgreSQL, it's treated like Read Committed:
In PostgreSQL, you can request any of the four standard transaction
isolation levels. But internally, there are only two distinct
isolation levels, which correspond to the levels Read Committed and
Serializable. When you select the level Read Uncommitted you really
get Read Committed, and when you select Repeatable Read you really get
Serializable, so the actual isolation level might be stricter than
what you select.
I'm not sure how much a slight delay is for getting readable data is, but you might want to look into Slony Replication. Essentially, you have a master database with a slave database. All of your inserts/writes would be put into your master database, then Slony would replicate those new entries into the slave database (this takes a little bit of time, but nothing monumental. A few minutes, perhaps.). Then you can have your applications read exclusively from the slave database.
If Slony doesn't seem right for you, you can look at some "multi-master" alternatives here. These will let you have multiple machines be writeable, and have all their contents be replicated onto the read-machine.