while posting debezium sql-server source connector, getting following error:
{"name":"testing_sql_server_conn_2","connector":{"state":"RUNNING","worker_id":"192.168.18.137:8083"},"tasks":[{"id":0,"state":"FAILED","worker_id":"192.168.18.137:8083","trace":"org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.\n\tat io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:116)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:750)\nCaused by: java.lang.NoSuchMethodError: io.debezium.schema.SchemaChangeEvent.ofSnapshotCreate(Lio/debezium/pipeline/spi/Partition;Lio/debezium/pipeline/spi/OffsetContext;Ljava/lang/String;Lio/debezium/relational/Table;)Lio/debezium/schema/SchemaChangeEvent;\n\tat io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource.getCreateTableEvent(SqlServerSnapshotChangeEventSource.java:232)\n\tat io.debezium.relational.RelationalSnapshotChangeEventSource.lambda$createSchemaChangeEventsForTables$2(RelationalSnapshotChangeEventSource.java:277)\n\tat io.debezium.pipeline.EventDispatcher.dispatchSchemaChangeEvent(EventDispatcher.java:310)\n\tat io.debezium.relational.RelationalSnapshotChangeEventSource.createSchemaChangeEventsForTables(RelationalSnapshotChangeEventSource.java:275)\n\tat io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:118)\n\tat io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:155)\n\tat io.debezium.connector.sqlserver.SqlServerChangeEventSourceCoordinator.executeChangeEventSources(SqlServerChangeEventSourceCoordinator.java:71)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)\n\t... 5 more\n"}],"type":"source"}
connector configuration are
curl -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://192.234.15.127:8788/connectors/ -d '{
"name": "testing_sql_server_conn_2",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"database.hostname": "192.234.15.127",
"database.port": "1433",
"database.user": "SA",
"database.password": "admin#123",
"database.dbname": "MyDB",
"database.server.name": "testing",
"table.whitelist": "dbo.MyTable",
"database.history.kafka.bootstrap.servers": "192.234.15.127:9092",
"database.history.kafka.topic": "dbhistory.Posts"
}
}'
Related
I'm trying to get the aggregate agent to work with timescale db.
https://volttron.readthedocs.io/en/main/developing-volttron/developing-agents/specifications/aggregate.html
I have a file aggregation.config
{
"connection": {
"type": "postgresql",
"params": {
"dbname": "volttrondb",
"host": "127.0.0.1",
"port": 5432,
"user": "user",
"password": "password",
"timescale_dialect": true
}
},
"aggregations":[
# list of aggregation groups each with unique aggregation_period and
# list of points that needs to be collected
{
"aggregation_period": "1h",
"use_calendar_time_periods": true,
"utc_collection_start_time":"2016-03-01T01:15:01.000000",
"points": [
{
"topic_names": ["campus/building/fake/EKG_Cos", "campus/building/fake/EKG_Sin"],
"aggregation_topic_name":"campus/building/fake/avg_of_EKG_Cos_EKG_Sin",
"aggregation_type": "avg",
"min_count": 2
}
]
}
]
}
And run the following command
vctl install services/core/SQLAggregateHistorian/ --tag aggregate-historian -c config/aggregation.config --start
It starts correctly - the vctl status shows it running and there are no errors in the log.
I do not see the point campus/building/fake/avg_of_EKG_Cos_EKG_Sin in the topics table.
Any suggestions?
I'm getting error as below when creating kafka producer:
Invalid Java object for schema type STRING: class java.lang.Long for field: \"BIRTHDATE\"\n\tat.
My script as below:
{
"name": "id_srvc_producer_ODS_EB_CLIENT_202215202",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"tasks.max": "1",
"errors.tolerance": "all",
"errors.log.enable": "true",
"errors.log.include.messages": "true",
"decimal.handling.mode": "string",
"transforms": "Cast,encryptvalue,encryptkey",
"transforms.Cast.type": "com.fwd.kafka.connect.smt.Cast$Value",
"transforms.Cast.spec": "CLTDOB:string, CLTDOD:float64, USER:float64, TRDT:float64, TRTM:float64, CAPITAL:float64, SRDATE:float64, DATIME:string, AUD_TIME:string",
"transforms.encryptvalue.type": "com.xxx.kafka.connect.smt.EncryptFieldV2$Value",
"transforms.encryptvalue.encrypt.fields": "BIRTHPLACE,BIRTHDATE,IDENTITYNO,FIRSTNAME,SURNAME,EMAILID,MOBILENUMBER,PHONENUMER1,PHONENUMER2,FAXNUMBER",
"transforms.encryptvalue.encrypt.algorithm": "AES",
"transforms.encryptvalue.encrypt.key": "${securepass:/opt/secret/security.properties:aes-encryption-key.properties/aes_encryption_key}",
"transforms.encryptkey.type": "com.xxx.kafka.connect.smt.EncryptFieldVxxxkey",
"transforms.encryptkey.encrypt.fields": "pk:CLIENTCODE",
"transforms.encryptkey.encrypt.algorithm": "AES"
}
I am struggling with Xdebug + WSL2 + CakePHP 3 + VSCode. Checking the debug console it seems that I have running Xdebug correctly, but when I run a script in the browser, the Xdebug is not launched. This is the code:
Note: I forgot to mention that I am working on docker, that's why the "0.0.0.0" in the hostname parameter.
This is the xdebug.ini
zend_extension=xdebug
[xdebug]
zend_extension=xdebug
xdebug.mode=develop,debug
xdebug.client_host='host.docker.internal'
xdebug.start_with_request=yes
xdebug.client_port = 9003
xdebug.start_with_request=yes
xdebug.log=/var/log/xdebug/xdebug.log
xdebug.connect_timeout_ms=2000
launch.json
{
"version": "0.2.0",
"configurations": [
{
"name": "Listen for Xdebug",
"type": "php",
"request": "launch",
"port": 9003,
"hostname": "0.0.0.0",
"pathMappings": {
"/webroot": "${workspaceRoot}"
},
"log": true
},
{
"name": "Launch currently open script",
"type": "php",
"request": "launch",
"program": "${file}",
"cwd": "${fileDirname}",
"port": 0,
"runtimeArgs": [
"-dxdebug.start_with_request=yes"
],
"env": {
"XDEBUG_MODE": "debug,develop",
"XDEBUG_CONFIG": "client_port=${port}"
}
},
{
"name": "Launch Built-in web server",
"type": "php",
"request": "launch",
"runtimeArgs": [
"-dxdebug.mode=debug",
"-dxdebug.start_with_request=yes",
"-S",
"localhost:0"
],
"program": "",
"cwd": "${workspaceRoot}",
"port": 9003,
"serverReadyAction": {
"pattern": "Development Server \\(http://localhost:([0-9]+)\\) started",
"uriFormat": "http://localhost:%s",
"action": "openExternally"
}
}
]
}
This is the debug console:
Listening on { address: '0.0.0.0', family: 'IPv4', port: 9003 }
<- launchResponse
Response {
seq: 0,
type: 'response',
request_seq: 2,
command: 'launch',
success: true
}
<- initializedEvent
InitializedEvent { seq: 0, type: 'event', event: 'initialized' }
-> setBreakpointsRequest
{
command: 'setBreakpoints',
arguments: {
source: {
name: 'index.php',
path: '/root/server/webroot/index.php'
},
lines: [ 40 ],
breakpoints: [ { line: 40 } ],
sourceModified: false
},
type: 'request',
seq: 3
}
<- setBreakpointsResponse
Response {
seq: 0,
type: 'response',
request_seq: 3,
command: 'setBreakpoints',
success: true,
body: {
breakpoints: [
{
verified: true,
line: 40,
source: {
name: 'index.php',
path: '/root/server/webroot/index.php'
},
id: 1
}
]
}
}
The xdebug.log file
[20] Log opened at 2022-05-16 04:42:03.776649
[20] [Step Debug] INFO: Connecting to configured address/port: host.docker.internal:9003.
[20] [Step Debug] INFO: Connected to debugging client: host.docker.internal:9003 (through xdebug.client_host/xdebug.client_port). :-)
[20] [Step Debug] -> <init xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" fileuri="file:///var/www/html/webroot/info.php" language="PHP" xdebug:language_version="7.4.19" protocol_version="1.0" appid="20"><engine version="3.1.2"><![CDATA[Xdebug]]></engine><author><![CDATA[Derick Rethans]]></author><url><![CDATA[https://xdebug.org]]></url><copyright><![CDATA[Copyright (c) 2002-2021 by Derick Rethans]]></copyright></init>
[20] [Step Debug] <- feature_set -i 1 -n resolved_breakpoints -v 1
[20] [Step Debug] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="feature_set" transaction_id="1" feature="resolved_breakpoints" success="1"></response>
[20] [Step Debug] <- run -i 12
[20] [Step Debug] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="run" transaction_id="12" status="stopping" reason="ok"></response>
[20] [Step Debug] <- stop -i 13
[20] [Step Debug] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="stop" transaction_id="13" status="stopped" reason="ok"></response>
[20] Log closed at 2022-05-16 04:42:03.812679
UPDATE: Following this suggestion (I got from this link, by HolyGonzo) https://www.reddit.com/r/PHPhelp/comments/rqiw4h/need_help_troubleshooting_xdebug_configuration/ I added xdebug_break(); to my code, and then the debugger started working. It is pretty clear to understand that the issue it is in the VSCode configuration not in Xdebug.
SOLUTION:
After fight with this thing a few days, finally I found the issue:
In the launch.json in VSCode I updated this line and it works!!! (note that my path was wrong :( I had this "/var/www/webroot" instead of "/var/www/html/webroot").
"pathMappings": {
"/var/www/html/webroot": "${workspaceFolder}/webroot"
},
Update:
In order to allows Xdebug to look into the vendors folder, and the other folders outside /webroot, the code needs to be upated as follows (in my case, regarding to my server paths):
{
"name": "Listen for Xdebug",
"type": "php",
"request": "launch",
"port": 5902,
"hostname": "localhost",
"pathMappings": {
"/var/www/html": "${workspaceFolder}"
},
"log": true
}
I need to migrate a SQL Server database from on prem location to GoogleCloud
Using confluent/kafka to do it
I do have a source debezium connector
{
"name": "mssql_src",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"tasks.max": "1",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
...
...
...
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.add.fields": "op,table,source.ts_ms",
"transforms.unwrap.delete.handling.mode": "rewrite",
"transforms": "Reroute",
"transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
"transforms.Reroute.topic.regex": "source_dbname.dbo(.*)",
"transforms.Reroute.topic.replacement": "target_dbname$1"
}
}
reroute transformation does not work with conjunction of unwrap,
I still getting source_dbname.dbo.* topics instead of target_dbname.*
I need insert data into target_dbname database
jdbc sync connector has following configuration
{
"name": "mssql_trg",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics.regex": "source_dbname.dbo.*",
"table.name.format": "${topic}",
"connection.url": "jdbc:sqlserver://xxx.xxx.xxx.xxx:1433;DatabaseName=rocketlawyer3",
"connection.user": "sqlserver",
"connection.password": "sqlserver",
"dialect.name": "SqlServerDatabaseDialect",
"insert.mode": "upsert",
"auto.create": true,
"auto.evolve": true,
"pk.mode": "record_value"
}
}
Obviously it is failed because all SQL operations are referring tables as source_database_name.dbo.table_name
Here is 2 questions:
How can I change this string source_database_name.dbo.table_name just for table_name using table.name.format
or other option
How can I make both transformation (reroute and unwrap) work in source connector
You need to use chained transformations. Just combine unwrap and Rerout SMTs together:
{
"name": "mssql_src",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"tasks.max": "1",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
...
...
...
"transforms": "unwrap, Reroute",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.add.fields": "op,table,source.ts_ms",
"transforms.unwrap.delete.handling.mode": "rewrite",
"transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
"transforms.Reroute.topic.regex": "([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.Reroute.topic.replacement": "$3"
}
}
Setup: I have CDC enabled on MS SQL Server and the CDC events are fed to Kafka using debezium kafka connect(source). Also more than one table CDC events are routed to a single topic in Kafka.
Question: Since I have more than one table data in the kafka topic, I would like to have the table name and the database name in the CDC data.
I am getting the table name and database name in MySQL CDC but not in MS SQL CDC.
Below is the Debezium source connector for the SQL Server
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{
"name": "cdc-user_profile-connector",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"tasks.max": "1",
"database.hostname": "<<hostname>>",
"database.port": "<<port>>",
"database.user": "<<username>>",
"database.password": "<<password>>",
"database.server.name": "test",
"database.dbname": "testDb",
"table.whitelist": "schema01.table1,schema01.table2",
"database.history.kafka.bootstrap.servers": "broker:9092",
"database.history.kafka.topic": "digital.user_profile.schema.audit",
"database.history.store.only.monitored.tables.ddl": true,
"include.schema.changes": false,
"event.deserialization.failure.handling.mode": "fail",
"snapshot.mode": "initial_schema_only",
"snapshot.locking.mode": "none",
"transforms":"addStaticField,topicRoute",
"transforms.addStaticField.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.addStaticField.static.field":"source_system",
"transforms.addStaticField.static.value":"source_system_1",
"transforms.topicRoute.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.topicRoute.regex":"(.*)",
"transforms.topicRoute.replacement":"digital.user_profile",
"errors.tolerance": "none",
"errors.log.enable": true,
"errors.log.include.messages": true,
"errors.retry.delay.max.ms": 60000,
"errors.retry.timeout": 300000
}
}'
I am getting the below output (Demo data)
{
"before": {
"profile_id": 147,
"email_address": "test#gmail.com"
},
"after": {
"profile_id": 147,
"email_address": "test_modified#gmail.com"
},
"source": {
"version": "0.9.4.Final",
"connector": "sqlserver",
"name": "test",
"ts_ms": 1556723528917,
"change_lsn": "0007cbe5:0000b98c:0002",
"commit_lsn": "0007cbe5:0000b98c:0003",
"snapshot": false
},
"op": "u",
"ts_ms": 1556748731417,
"source_system": "source_system_1"
}
My requirement is to get as below
{
"before": {
"profile_id": 147,
"email_address": "test#gmail.com"
},
"after": {
"profile_id": 147,
"email_address": "test_modified#gmail.com"
},
"source": {
"version": "0.9.4.Final",
"connector": "sqlserver",
"name": "test",
"ts_ms": 1556723528917,
"change_lsn": "0007cbe5:0000b98c:0002",
"commit_lsn": "0007cbe5:0000b98c:0003",
"snapshot": false,
"db": "testDb",
"table": "table1/table2"
},
"op": "u",
"ts_ms": 1556748731417,
"source_system": "source_system_1"
}
This is planned as a part of https://issues.jboss.org/browse/DBZ-875 issue
Debezium Kafka-Connect generally puts data from each table in a separate topic and the topic name is of the format hostname.database.table. We generally use the topic name to distinguish between the source table & database name.
If you are putting the data from all the tables manually into one topic then you might have to add the table and database name manually as well.