upsert-kafka connect in flink produces null in the final topic - apache-flink

Question

I have a sql defined this way
CREATE TABLE raw_table
(
headers VARCHAR,
id VARCHAR,
type VARCHAR,
contentJson VARCHAR
) WITH (
'connector' = 'kafka',
'topic-pattern' = 'role__.+?',
'properties.bootstrap.servers' = 'localhost:29092,localhost:39092',
'properties.group.id' = 'role_local_1',
'scan.startup.mode' = 'earliest-offset',
'format' = 'json',
'properties.allow.auto.create.topics' = 'true',
'json.timestamp-format.standard' = 'ISO-8601',
'sink.parallelism' = '3'
);
create view ROLES_NORMALIZED as
(
select
JSON_VALUE(contentJson, '$.id') as id,
rr.type as type
from raw_table rr
);
CREATE VIEW ROLES_UPSERTS_V1 AS
(
SELECT *
FROM ROLES_NORMALIZED
WHERE type in ('ROLE_CREATED', 'ROLE_UPDATED')
);
CREATE VIEW ROLES_DELETED_V1 AS
(
SELECT org,
pod,
tenantId,
id,
modified,
modified as deleted,
event_timestamp
FROM ROLES_NORMALIZED
WHERE type in ('ROLES_DELETED')
);
-------
CREATE TABLE final_topic
(
event_timestamp TIMESTAMP_LTZ,
id VARCHAR,
name VARCHAR,
deleted TIMESTAMP_LTZ,
PRIMARY KEY (pod, org, id) NOT ENFORCED
) WITH (
'connector' = 'upsert-kafka',
'topic' = 'final_topic',
'properties.bootstrap.servers' = 'localhost:29092,localhost:39092',
'properties.group.id' = 'some-group-id',
'value.format' = 'json',
'key.format' = 'json',
'properties.allow.auto.create.topics' = 'true',
'properties.replication.factor' = '3',
'value.json.timestamp-format.standard' = 'ISO-8601',
'sink.parallelism' = '3'
);
INSERT INTO final_topic
select
GREATEST(r.event_timestamp, d.event_timestamp) as event_timestamp,
r.id,
r.name,
d.deleted
from ROLES_UPSERTS_V1 r
LEFT JOIN ROLES_DELETED_V1 d
ON r.id = d.id;
The final_topic is produces the result i want to see, which is join of ROLES_UPSERTS_V1 and ROLES_DELETED_V1.
I tried this by publishing records to role__.+? topic.
What I am observing is that final-topic has null values as well. This is emitted when even the changelog kind happens to be -d -(DELETE). I understand the purpose as to why this exist. (here its saying the original message needs to be deleted and new one will follow). But I dont want such null values in my final-topic just the desired final state is this possible ?
Alternate that I am trying is to use Kafka connector. But the joins does not seems to work, as i get an error saying org.apache.flink.table.api.TableException: Table sink 'default_catalog.default_database.final_topic' doesn't support consuming update and delete changes which is produced by node Join(joinType=[LeftOuterJoin]. I get error when i use view for ROLES_UPSERTS_V1 and ROLES_DELETED_V1 . But if i have these as tables (with kafka connector) only inner join works ( left join does not work).

Answer 1

If you don't want null values, you can consider buffering records before you flush the results to the Upsert Kafka connector. See https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/table/upsert-kafka/#sink-buffer-flush-max-rows for more details
Regarding your join, as outlined in the docs For streaming queries, the grammar of regular joins is the most flexible and allow for any kind of updating (insert, update, delete) input table. Since you're running a streaming query, a future change could mean that the result of your join os an update or a delete. However, the sink that you're trying to emit to does not support this, hence the error.