Debezium SqlServer source not reading data - sql-server
I have a running Kafka Connect instance and have submitted my connector with the following configuration at the bottom of this post.
Question
The Debezium docs seem to indicate I set database.server.name=connect_test and create topics for each table I want to ingest into kafka.. So for my table, I'd create connect_test-TEST_Test_Table_Object.
I don't get any errors, but no data is ingested into Kafka. I do see some warnings about configs, but Im just trying to get a very basic test up..
Can anyone proivide any insight?
I've also pre-created the following topics:
connect-configs (1 partition)
connect-offsets (3 partitions)
connect-status (3 partitions)
schema_changes-connect_test (3 partitions)
connect_test-TEST_Test_Table_Object (3 partitions)
{
"name": "sql-server-source-connector",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"database.hostname": "redacted.public.redacted.database.windows.net",
"database.port": "3342",
"database.user": "db_user",
"database.password": "password",
"database.dbname": "TEST_DB",
"database.server.name": "connect_test",
"database.history.kafka.bootstrap.servers": "kafka-url-1:9096,kafka-url-2:9096,kafka-url-3:9096",
"database.history.kafka.topic": "schema_changes-connect_test",
"table.include.list": "TEST_Test_Table_Object",
"database.history.producer.security.protocol": "SSL",
"database.history.producer.ssl.keystore.location": "/app/.keystore.jks",
"database.history.producer.ssl.keystore.password": "password",
"database.history.producer.ssl.truststore.location": "/app/.truststore.jks",
"database.history.producer.ssl.truststore.password": "password",
"database.history.producer.ssl.key.password": "password",
"database.history.consumer.security.protocol": "SSL",
"database.history.consumer.ssl.keystore.location": "/app/.keystore.jks",
"database.history.consumer.ssl.keystore.password": "password",
"database.history.consumer.ssl.truststore.password": "/app/.truststore.jks",
"database.history.consumer.ssl.key.password": "password"
}
}
I keep seeing Failed to construct kafka producer ... caused by: Failed to load SSL keystore /app/.keystore.jks of type JKS ... failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded
I'm using Heroku Kafka and I have three certs: client_cert.pem, client_key.pem, trusted_cert.pem
I use keytool to turn my .pem's into /app/.keystore.jks and /app/.truststore.jks
Logs..
Some WARN redacted for size
2022-06-06T21:11:16.115619+00:00 app[web.3]: [2022-06-06 21:11:16,115] INFO [Consumer clientId=consumer-connect-demo-group-2, groupId=connect-demo-group] Cluster ID: some-id (org.apache.kafka.clients.Metadata)
2022-06-06T21:11:16.116945+00:00 app[web.3]: [2022-06-06 21:11:16,116] INFO [Consumer clientId=consumer-connect-demo-group-2, groupId=connect-demo-group] Subscribed to partition(s): connect-status-0, connect-status-2, connect-status-1 (org.apache.kafka.clients.consumer.KafkaConsumer)
2022-06-06T21:11:16.117022+00:00 app[web.3]: [2022-06-06 21:11:16,116] INFO [Consumer clientId=consumer-connect-demo-group-2, groupId=connect-demo-group] Seeking to EARLIEST offset of partition connect-status-0 (org.apache.kafka.clients.consumer.internals.SubscriptionState)
2022-06-06T21:11:16.117070+00:00 app[web.3]: [2022-06-06 21:11:16,117] INFO [Consumer clientId=consumer-connect-demo-group-2, groupId=connect-demo-group] Seeking to EARLIEST offset of partition connect-status-2 (org.apache.kafka.clients.consumer.internals.SubscriptionState)
2022-06-06T21:11:16.117092+00:00 app[web.3]: [2022-06-06 21:11:16,117] INFO [Consumer clientId=consumer-connect-demo-group-2, groupId=connect-demo-group] Seeking to EARLIEST offset of partition connect-status-1 (org.apache.kafka.clients.consumer.internals.SubscriptionState)
2022-06-06T21:11:15.127922+00:00 app[web.3]: [2022-06-06 21:11:15,124] INFO [Producer clientId=producer-1] Cluster ID: some-id (org.apache.kafka.clients.Metadata)
2022-06-06T21:11:16.441247+00:00 app[web.3]: [2022-06-06 21:11:16,439] INFO [Producer clientId=producer-3] Cluster ID: some-id (org.apache.kafka.clients.Metadata)
2022-06-06T21:11:16.579714+00:00 app[web.3]: [2022-06-06 21:11:16,577] WARN The configuration 'log4j.loggers' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
# WARN redacted for size limits
2022-06-06T21:11:16.580291+00:00 app[web.3]: [2022-06-06 21:11:16,580] WARN The configuration 'key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
2022-06-06T21:11:16.580291+00:00 app[web.3]: [2022-06-06 21:11:16,580] WARN The configuration 'value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
2022-06-06T21:11:16.580315+00:00 app[web.3]: [2022-06-06 21:11:16,580] WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
2022-06-06T21:11:16.580348+00:00 app[web.3]: [2022-06-06 21:11:16,580] WARN The configuration 'log4j.root.loglevel' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
2022-06-06T21:11:16.580414+00:00 app[web.3]: [2022-06-06 21:11:16,580] INFO Kafka version: 6.1.4-ccs (org.apache.kafka.common.utils.AppInfoParser)
2022-06-06T21:11:16.580457+00:00 app[web.3]: [2022-06-06 21:11:16,580] INFO Kafka commitId: c9124241a6ff43bc (org.apache.kafka.common.utils.AppInfoParser)
2022-06-06T21:11:16.580479+00:00 app[web.3]: [2022-06-06 21:11:16,580] INFO Kafka startTimeMs: 1654549876580 (org.apache.kafka.common.utils.AppInfoParser)
2022-06-06T21:11:16.607720+00:00 app[web.3]: [2022-06-06 21:11:16,607] INFO [Consumer clientId=consumer-connect-demo-group-3, groupId=connect-demo-group] Cluster ID: someId (org.apache.kafka.clients.Metadata)
2022-06-06T21:11:16.608322+00:00 app[web.3]: [2022-06-06 21:11:16,608] INFO [Consumer clientId=consumer-connect-demo-group-3, groupId=connect-demo-group] Subscribed to partition(s): connect-configs-0 (org.apache.kafka.clients.consumer.KafkaConsumer)
2022-06-06T21:11:16.608416+00:00 app[web.3]: [2022-06-06 21:11:16,608] INFO [Consumer clientId=consumer-connect-demo-group-3, groupId=connect-demo-group] Seeking to EARLIEST offset of partition connect-configs-0 (org.apache.kafka.clients.consumer.internals.SubscriptionState)
2022-06-06T21:11:16.658870+00:00 app[web.3]: [2022-06-06 21:11:16,658] INFO [Consumer clientId=consumer-connect-demo-group-3, groupId=connect-demo-group] Resetting offset for partition connect-configs-0 to position FetchPosition{offset=20, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-url-1:9096 (id: 1 rack: us-east-1a)], epoch=0}}. (org.apache.kafka.clients.consumer.internals.SubscriptionState)
2022-06-06T21:11:16.164555+00:00 app[web.3]: [2022-06-06 21:11:16,163] INFO [Consumer clientId=consumer-connect-demo-group-2, groupId=connect-demo-group] Resetting offset for partition connect-status-2 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-url-2:9096 (id: 2 rack: us-east-1b)], epoch=0}}. (org.apache.kafka.clients.consumer.internals.SubscriptionState)
2022-06-06T21:11:16.189017+00:00 app[web.3]: [2022-06-06 21:11:16,188] INFO [Consumer clientId=consumer-connect-demo-group-2, groupId=connect-demo-group] Resetting offset for partition connect-status-1 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-url-1:9096 (id: 1 rack: us-east-1a)], epoch=0}}. (org.apache.kafka.clients.consumer.internals.SubscriptionState)
2022-06-06T21:11:16.238074+00:00 app[web.3]: [2022-06-06 21:11:16,237] INFO [Consumer clientId=consumer-connect-demo-group-2, groupId=connect-demo-group] Resetting offset for partition connect-status-0 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-url-3:9096 (id: 0 rack: us-east-1c)], epoch=0}}. (org.apache.kafka.clients.consumer.internals.SubscriptionState)
2022-06-06T21:11:16.252558+00:00 app[web.3]: [2022-06-06 21:11:16,252] INFO ProducerConfig values:
2022-06-06T21:11:16.252560+00:00 app[web.3]: acks = -1
2022-06-06T21:11:16.252561+00:00 app[web.3]: batch.size = 16384
2022-06-06T21:11:16.252562+00:00 app[web.3]: bootstrap.servers = [kafka-url-2:9096, kafka-url-1:9096, kafka-url-3:9096]
2022-06-06T21:11:16.252562+00:00 app[web.3]: buffer.memory = 33554432
2022-06-06T21:11:16.252563+00:00 app[web.3]: client.dns.lookup = use_all_dns_ips
2022-06-06T21:11:16.252563+00:00 app[web.3]: client.id = producer-3
2022-06-06T21:11:16.252564+00:00 app[web.3]: compression.type = none
2022-06-06T21:11:16.252564+00:00 app[web.3]: connections.max.idle.ms = 540000
2022-06-06T21:11:16.252564+00:00 app[web.3]: delivery.timeout.ms = 2147483647
2022-06-06T21:11:16.252564+00:00 app[web.3]: enable.idempotence = false
2022-06-06T21:11:16.252565+00:00 app[web.3]: interceptor.classes = []
2022-06-06T21:11:16.252565+00:00 app[web.3]: internal.auto.downgrade.txn.commit = false
2022-06-06T21:11:16.252566+00:00 app[web.3]: key.serializer = class org.apache.kafka.common.serialization.StringSerializer
2022-06-06T21:11:16.252566+00:00 app[web.3]: linger.ms = 0
2022-06-06T21:11:16.252566+00:00 app[web.3]: max.block.ms = 60000
2022-06-06T21:11:16.252567+00:00 app[web.3]: max.in.flight.requests.per.connection = 1
2022-06-06T21:11:16.252567+00:00 app[web.3]: max.request.size = 1048576
2022-06-06T21:11:16.252567+00:00 app[web.3]: metadata.max.age.ms = 300000
2022-06-06T21:11:16.252567+00:00 app[web.3]: metadata.max.idle.ms = 300000
2022-06-06T21:11:16.252567+00:00 app[web.3]: metric.reporters = []
2022-06-06T21:11:16.252568+00:00 app[web.3]: metrics.num.samples = 2
2022-06-06T21:11:16.252568+00:00 app[web.3]: metrics.recording.level = INFO
2022-06-06T21:11:16.252568+00:00 app[web.3]: metrics.sample.window.ms = 30000
2022-06-06T21:11:16.252569+00:00 app[web.3]: partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
2022-06-06T21:11:16.252569+00:00 app[web.3]: receive.buffer.bytes = 32768
2022-06-06T21:11:16.252569+00:00 app[web.3]: reconnect.backoff.max.ms = 1000
2022-06-06T21:11:16.252570+00:00 app[web.3]: reconnect.backoff.ms = 50
2022-06-06T21:11:16.252570+00:00 app[web.3]: request.timeout.ms = 30000
2022-06-06T21:11:16.252570+00:00 app[web.3]: retries = 2147483647
2022-06-06T21:11:16.252570+00:00 app[web.3]: retry.backoff.ms = 100
2022-06-06T21:11:16.252571+00:00 app[web.3]: sasl.client.callback.handler.class = null
2022-06-06T21:11:16.252571+00:00 app[web.3]: sasl.jaas.config = null
2022-06-06T21:11:16.252571+00:00 app[web.3]: sasl.kerberos.kinit.cmd = /usr/bin/kinit
2022-06-06T21:11:16.252572+00:00 app[web.3]: sasl.kerberos.min.time.before.relogin = 60000
2022-06-06T21:11:16.252572+00:00 app[web.3]: sasl.kerberos.service.name = null
2022-06-06T21:11:16.252572+00:00 app[web.3]: sasl.kerberos.ticket.renew.jitter = 0.05
2022-06-06T21:11:16.252573+00:00 app[web.3]: sasl.kerberos.ticket.renew.window.factor = 0.8
2022-06-06T21:11:16.252573+00:00 app[web.3]: sasl.login.callback.handler.class = null
2022-06-06T21:11:16.252573+00:00 app[web.3]: sasl.login.class = null
2022-06-06T21:11:16.252574+00:00 app[web.3]: sasl.login.refresh.buffer.seconds = 300
2022-06-06T21:11:16.252574+00:00 app[web.3]: sasl.login.refresh.min.period.seconds = 60
2022-06-06T21:11:16.252574+00:00 app[web.3]: sasl.login.refresh.window.factor = 0.8
2022-06-06T21:11:16.252574+00:00 app[web.3]: sasl.login.refresh.window.jitter = 0.05
2022-06-06T21:11:16.252575+00:00 app[web.3]: sasl.mechanism = GSSAPI
2022-06-06T21:11:16.252575+00:00 app[web.3]: security.protocol = SSL
2022-06-06T21:11:16.252575+00:00 app[web.3]: security.providers = null
2022-06-06T21:11:16.252575+00:00 app[web.3]: send.buffer.bytes = 131072
2022-06-06T21:11:16.252576+00:00 app[web.3]: socket.connection.setup.timeout.max.ms = 127000
2022-06-06T21:11:16.252576+00:00 app[web.3]: socket.connection.setup.timeout.ms = 10000
2022-06-06T21:11:16.252576+00:00 app[web.3]: ssl.cipher.suites = null
2022-06-06T21:11:16.252577+00:00 app[web.3]: ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
2022-06-06T21:11:16.252577+00:00 app[web.3]: ssl.endpoint.identification.algorithm =
2022-06-06T21:11:16.252577+00:00 app[web.3]: ssl.engine.factory.class = null
2022-06-06T21:11:16.252578+00:00 app[web.3]: ssl.key.password = [hidden]
2022-06-06T21:11:16.252578+00:00 app[web.3]: ssl.keymanager.algorithm = SunX509
2022-06-06T21:11:16.252578+00:00 app[web.3]: ssl.keystore.certificate.chain = null
2022-06-06T21:11:16.252578+00:00 app[web.3]: ssl.keystore.key = null
2022-06-06T21:11:16.252579+00:00 app[web.3]: ssl.keystore.location = /app/.keystore.jks
2022-06-06T21:11:16.252579+00:00 app[web.3]: ssl.keystore.password = [hidden]
2022-06-06T21:11:16.252579+00:00 app[web.3]: ssl.keystore.type = JKS
2022-06-06T21:11:16.252580+00:00 app[web.3]: ssl.protocol = SSL
2022-06-06T21:11:16.252580+00:00 app[web.3]: ssl.provider = null
2022-06-06T21:11:16.252580+00:00 app[web.3]: ssl.secure.random.implementation = null
2022-06-06T21:11:16.252580+00:00 app[web.3]: ssl.trustmanager.algorithm = PKIX
2022-06-06T21:11:16.252581+00:00 app[web.3]: ssl.truststore.certificates = null
2022-06-06T21:11:16.252581+00:00 app[web.3]: ssl.truststore.location = /app/.truststore.jks
2022-06-06T21:11:16.252581+00:00 app[web.3]: ssl.truststore.password = [hidden]
2022-06-06T21:11:16.252582+00:00 app[web.3]: ssl.truststore.type = JKS
2022-06-06T21:11:16.252582+00:00 app[web.3]: transaction.timeout.ms = 60000
2022-06-06T21:11:16.252582+00:00 app[web.3]: transactional.id = null
2022-06-06T21:11:16.252582+00:00 app[web.3]: value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
2022-06-06T21:11:16.252583+00:00 app[web.3]: (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391195+00:00 app[web.3]: [2022-06-06 21:11:16,390] WARN The configuration 'log4j.loggers' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391271+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391272+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'rest.advertised.port' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391272+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391273+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'status.storage.partitions' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391273+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'metrics.context.connect.kafka.cluster.id' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391304+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'offset.storage.partitions' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391812+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'topic.creation.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391813+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391855+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'config.storage.partitions' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391894+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'config.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391894+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391918+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391963+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.391988+00:00 app[web.3]: [2022-06-06 21:11:16,391] WARN The configuration 'log4j.root.loglevel' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
2022-06-06T21:11:16.392047+00:00 app[web.3]: [2022-06-06 21:11:16,392] INFO Kafka version: 6.1.4-ccs (org.apache.kafka.common.utils.AppInfoParser)
2022-06-06T21:11:16.392077+00:00 app[web.3]: [2022-06-06 21:11:16,392] INFO Kafka commitId: c9124241a6ff43bc (org.apache.kafka.common.utils.AppInfoParser)
2022-06-06T21:11:16.392131+00:00 app[web.3]: [2022-06-06 21:11:16,392] INFO Kafka startTimeMs: 1654549876391 (org.apache.kafka.common.utils.AppInfoParser)
2022-06-06T21:11:16.401532+00:00 app[web.3]: [2022-06-06 21:11:16,401] INFO ConsumerConfig values:
2022-06-06T21:11:16.401533+00:00 app[web.3]: allow.auto.create.topics = true
2022-06-06T21:11:16.401534+00:00 app[web.3]: auto.commit.interval.ms = 5000
2022-06-06T21:11:16.401534+00:00 app[web.3]: auto.offset.reset = earliest
2022-06-06T21:11:16.401535+00:00 app[web.3]: bootstrap.servers = [kafka-url-2:9096, kafka-url-1:9096, kafka-url-3:9096]
2022-06-06T21:11:16.401536+00:00 app[web.3]: check.crcs = true
2022-06-06T21:11:16.401536+00:00 app[web.3]: client.dns.lookup = use_all_dns_ips
2022-06-06T21:11:16.401536+00:00 app[web.3]: client.id = consumer-connect-demo-group-3
2022-06-06T21:11:16.401537+00:00 app[web.3]: client.rack =
2022-06-06T21:11:16.401537+00:00 app[web.3]: connections.max.idle.ms = 540000
2022-06-06T21:11:16.401537+00:00 app[web.3]: default.api.timeout.ms = 60000
2022-06-06T21:11:16.401538+00:00 app[web.3]: enable.auto.commit = false
2022-06-06T21:11:16.401538+00:00 app[web.3]: exclude.internal.topics = true
2022-06-06T21:11:16.401538+00:00 app[web.3]: fetch.max.bytes = 52428800
2022-06-06T21:11:16.401538+00:00 app[web.3]: fetch.max.wait.ms = 500
2022-06-06T21:11:16.401539+00:00 app[web.3]: fetch.min.bytes = 1
2022-06-06T21:11:16.401539+00:00 app[web.3]: group.id = connect-demo-group
2022-06-06T21:11:16.401539+00:00 app[web.3]: group.instance.id = null
2022-06-06T21:11:16.401540+00:00 app[web.3]: heartbeat.interval.ms = 3000
2022-06-06T21:11:16.401540+00:00 app[web.3]: interceptor.classes = []
2022-06-06T21:11:16.401540+00:00 app[web.3]: internal.leave.group.on.close = true
2022-06-06T21:11:16.401541+00:00 app[web.3]: internal.throw.on.fetch.stable.offset.unsupported = false
2022-06-06T21:11:16.401541+00:00 app[web.3]: isolation.level = read_uncommitted
2022-06-06T21:11:16.401541+00:00 app[web.3]: key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
2022-06-06T21:11:16.401542+00:00 app[web.3]: max.partition.fetch.bytes = 1048576
2022-06-06T21:11:16.401542+00:00 app[web.3]: max.poll.interval.ms = 300000
2022-06-06T21:11:16.401542+00:00 app[web.3]: max.poll.records = 500
2022-06-06T21:11:16.401542+00:00 app[web.3]: metadata.max.age.ms = 300000
2022-06-06T21:11:16.401543+00:00 app[web.3]: metric.reporters = []
2022-06-06T21:11:16.401544+00:00 app[web.3]: metrics.num.samples = 2
2022-06-06T21:11:16.401544+00:00 app[web.3]: metrics.recording.level = INFO
2022-06-06T21:11:16.401544+00:00 app[web.3]: metrics.sample.window.ms = 30000
2022-06-06T21:11:16.401544+00:00 app[web.3]: partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
2022-06-06T21:11:16.401545+00:00 app[web.3]: receive.buffer.bytes = 65536
2022-06-06T21:11:16.401545+00:00 app[web.3]: reconnect.backoff.max.ms = 1000
2022-06-06T21:11:16.401545+00:00 app[web.3]: reconnect.backoff.ms = 50
2022-06-06T21:11:16.401546+00:00 app[web.3]: request.timeout.ms = 30000
2022-06-06T21:11:16.401546+00:00 app[web.3]: retry.backoff.ms = 100
2022-06-06T21:11:16.401546+00:00 app[web.3]: sasl.client.callback.handler.class = null
2022-06-06T21:11:16.401546+00:00 app[web.3]: sasl.jaas.config = null
2022-06-06T21:11:16.401547+00:00 app[web.3]: sasl.kerberos.kinit.cmd = /usr/bin/kinit
2022-06-06T21:11:16.401547+00:00 app[web.3]: sasl.kerberos.min.time.before.relogin = 60000
2022-06-06T21:11:16.401547+00:00 app[web.3]: sasl.kerberos.service.name = null
2022-06-06T21:11:16.401548+00:00 app[web.3]: sasl.kerberos.ticket.renew.jitter = 0.05
2022-06-06T21:11:16.401548+00:00 app[web.3]: sasl.kerberos.ticket.renew.window.factor = 0.8
2022-06-06T21:11:16.401548+00:00 app[web.3]: sasl.login.callback.handler.class = null
2022-06-06T21:11:16.401548+00:00 app[web.3]: sasl.login.class = null
2022-06-06T21:11:16.401549+00:00 app[web.3]: sasl.login.refresh.buffer.seconds = 300
2022-06-06T21:11:16.401549+00:00 app[web.3]: sasl.login.refresh.min.period.seconds = 60
2022-06-06T21:11:16.401549+00:00 app[web.3]: sasl.login.refresh.window.factor = 0.8
2022-06-06T21:11:16.401550+00:00 app[web.3]: sasl.login.refresh.window.jitter = 0.05
2022-06-06T21:11:16.401550+00:00 app[web.3]: sasl.mechanism = GSSAPI
2022-06-06T21:11:16.401550+00:00 app[web.3]: security.protocol = SSL
2022-06-06T21:11:16.401551+00:00 app[web.3]: security.providers = null
2022-06-06T21:11:16.401551+00:00 app[web.3]: send.buffer.bytes = 131072
2022-06-06T21:11:16.401551+00:00 app[web.3]: session.timeout.ms = 10000
2022-06-06T21:11:16.401551+00:00 app[web.3]: socket.connection.setup.timeout.max.ms = 127000
2022-06-06T21:11:16.401552+00:00 app[web.3]: socket.connection.setup.timeout.ms = 10000
2022-06-06T21:11:16.401558+00:00 app[web.3]: ssl.cipher.suites = null
2022-06-06T21:11:16.401558+00:00 app[web.3]: ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
2022-06-06T21:11:16.401559+00:00 app[web.3]: ssl.endpoint.identification.algorithm =
2022-06-06T21:11:16.401559+00:00 app[web.3]: ssl.engine.factory.class = null
2022-06-06T21:11:16.401559+00:00 app[web.3]: ssl.key.password = [hidden]
2022-06-06T21:11:16.401560+00:00 app[web.3]: ssl.keymanager.algorithm = SunX509
2022-06-06T21:11:16.401560+00:00 app[web.3]: ssl.keystore.certificate.chain = null
2022-06-06T21:11:16.401560+00:00 app[web.3]: ssl.keystore.key = null
2022-06-06T21:11:16.401561+00:00 app[web.3]: ssl.keystore.location = /app/.keystore.jks
2022-06-06T21:11:16.401561+00:00 app[web.3]: ssl.keystore.password = [hidden]
2022-06-06T21:11:16.401561+00:00 app[web.3]: ssl.keystore.type = JKS
2022-06-06T21:11:16.401561+00:00 app[web.3]: ssl.protocol = SSL
2022-06-06T21:11:16.401561+00:00 app[web.3]: ssl.provider = null
2022-06-06T21:11:16.401562+00:00 app[web.3]: ssl.secure.random.implementation = null
2022-06-06T21:11:16.401562+00:00 app[web.3]: ssl.trustmanager.algorithm = PKIX
2022-06-06T21:11:16.401562+00:00 app[web.3]: ssl.truststore.certificates = null
2022-06-06T21:11:16.401563+00:00 app[web.3]: ssl.truststore.location = /app/.truststore.jks
2022-06-06T21:11:16.401563+00:00 app[web.3]: ssl.truststore.password = [hidden]
2022-06-06T21:11:16.401563+00:00 app[web.3]: ssl.truststore.type = JKS
2022-06-06T21:11:16.401563+00:00 app[web.3]: value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
2022-06-06T21:11:16.401564+00:00 app[web.3]: (org.apache.kafka.clients.consumer.ConsumerConfig)
2022-06-06T21:11:16.730633+00:00 app[web.3]: [2022-06-06 21:11:16,730] INFO [Worker clientId=connect-1, groupId=connect-demo-group] Cluster ID: someid (org.apache.kafka.clients.Metadata)
2022-06-06T21:11:16.732682+00:00 app[web.3]: [2022-06-06 21:11:16,732] INFO [Worker clientId=connect-1, groupId=connect-demo-group] Discovered group coordinator kafka-url-1:9096 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:16.736604+00:00 app[web.3]: [2022-06-06 21:11:16,736] INFO [Worker clientId=connect-1, groupId=connect-demo-group] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:16.765579+00:00 app[web.3]: [2022-06-06 21:11:16,765] INFO [Worker clientId=connect-1, groupId=connect-demo-group] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:19.337307+00:00 app[web.2]: [2022-06-06 21:11:19,337] INFO [Worker clientId=connect-1, groupId=connect-demo-group] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:19.337364+00:00 app[web.2]: [2022-06-06 21:11:19,337] INFO [Worker clientId=connect-1, groupId=connect-demo-group] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:19.342703+00:00 app[web.2]: [2022-06-06 21:11:19,342] INFO [Worker clientId=connect-1, groupId=connect-demo-group] Successfully joined group with generation Generation{generationId=39, memberId='connect-1-id-1', protocol='sessioned'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:19.347949+00:00 app[web.2]: [2022-06-06 21:11:19,347] INFO [Worker clientId=connect-1, groupId=connect-demo-group] Successfully synced group in generation Generation{generationId=39, memberId='connect-1-id-1', protocol='sessioned'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:19.342608+00:00 app[web.3]: [2022-06-06 21:11:19,342] INFO [Worker clientId=connect-1, groupId=connect-demo-group] Successfully joined group with generation Generation{generationId=39, memberId='connect-1-id-2', protocol='sessioned'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:19.347995+00:00 app[web.3]: [2022-06-06 21:11:19,347] INFO [Worker clientId=connect-1, groupId=connect-demo-group] Successfully synced group in generation Generation{generationId=39, memberId='connect-1-id-2', protocol='sessioned'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:19.339182+00:00 app[web.1]: [2022-06-06 21:11:19,339] INFO [Worker clientId=connect-1, groupId=connect-demo-group] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:19.339236+00:00 app[web.1]: [2022-06-06 21:11:19,339] INFO [Worker clientId=connect-1, groupId=connect-demo-group] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:19.341477+00:00 app[web.1]: [2022-06-06 21:11:19,341] INFO [Worker clientId=connect-1, groupId=connect-demo-group] Successfully joined group with generation Generation{generationId=39, memberId='connect-1-id-3', protocol='sessioned'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
2022-06-06T21:11:19.346767+00:00 app[web.1]: [2022-06-06 21:11:19,346] INFO [Worker clientId=connect-1, groupId=connect-demo-group] Successfully synced group in generation Generation{generationId=39, memberId='connect-1-id-3', protocol='sessioned'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
Related
Flink KafkaSink connector with exactly once semantics too many logs
Configuring a KafkaSink from new Kafka connector API (since version 1.15) with DeliveryGuarantee.EXACTLY_ONCE and transactionalId prefix produce an excessive amount of logs each time a new checkpoint is triggered. Logs are these 2022-11-02 10:04:10,124 INFO org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer [] - Flushing new partitions 2022-11-02 10:04:10,125 INFO org.apache.kafka.clients.producer.ProducerConfig [] - ProducerConfig values: acks = -1 batch.size = 16384 bootstrap.servers = [localhost:9092] buffer.memory = 33554432 client.dns.lookup = use_all_dns_ips client.id = producer-flink-1-24 compression.type = none connections.max.idle.ms = 540000 delivery.timeout.ms = 120000 enable.idempotence = true interceptor.classes = [] internal.auto.downgrade.txn.commit = false key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer linger.ms = 0 max.block.ms = 60000 max.in.flight.requests.per.connection = 5 max.request.size = 1048576 metadata.max.age.ms = 300000 metadata.max.idle.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner receive.buffer.bytes = 32768 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retries = 2147483647 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT security.providers = null send.buffer.bytes = 131072 socket.connection.setup.timeout.max.ms = 30000 socket.connection.setup.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.3] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLSv1.3 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS transaction.timeout.ms = 60000 transactional.id = flink-1-24 value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer 2022-11-02 10:04:10,131 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-flink-1-24, transactionalId=flink-1-24] Overriding the default enable.idempotence to true since transactional.id is specified. 2022-11-02 10:04:10,161 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-flink-0-24, transactionalId=flink-0-24] Overriding the default enable.idempotence to true since transactional.id is specified. 2022-11-02 10:04:10,161 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-flink-0-24, transactionalId=flink-0-24] Instantiated a transactional producer. 2022-11-02 10:04:10,162 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-flink-0-24, transactionalId=flink-0-24] Overriding the default acks to all since idempotence is enabled. 2022-11-02 10:04:10,159 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-flink-1-24, transactionalId=flink-1-24] Instantiated a transactional producer. 2022-11-02 10:04:10,170 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-flink-1-24, transactionalId=flink-1-24] Overriding the default acks to all since idempotence is enabled. 2022-11-02 10:04:10,181 INFO org.apache.kafka.common.utils.AppInfoParser [] - Kafka version: 2.8.1 2022-11-02 10:04:10,184 INFO org.apache.kafka.common.utils.AppInfoParser [] - Kafka commitId: 839b886f9b732b15 2022-11-02 10:04:10,184 INFO org.apache.kafka.common.utils.AppInfoParser [] - Kafka startTimeMs: 1667379850181 2022-11-02 10:04:10,185 INFO org.apache.kafka.clients.producer.internals.TransactionManager [] - [Producer clientId=producer-flink-0-24, transactionalId=flink-0-24] Invoking InitProducerId for the first time in order to acquire a producer ID 2022-11-02 10:04:10,192 INFO org.apache.kafka.common.utils.AppInfoParser [] - Kafka version: 2.8.1 2022-11-02 10:04:10,192 INFO org.apache.kafka.common.utils.AppInfoParser [] - Kafka commitId: 839b886f9b732b15 2022-11-02 10:04:10,192 INFO org.apache.kafka.common.utils.AppInfoParser [] - Kafka startTimeMs: 1667379850192 2022-11-02 10:04:10,209 INFO org.apache.kafka.clients.producer.internals.TransactionManager [] - [Producer clientId=producer-flink-1-24, transactionalId=flink-1-24] Invoking InitProducerId for the first time in order to acquire a producer ID 2022-11-02 10:04:10,211 INFO org.apache.kafka.clients.Metadata [] - [Producer clientId=producer-flink-0-24, transactionalId=flink-0-24] Cluster ID: MCY5mzM1QWyc1YCvsO8jag 2022-11-02 10:04:10,216 INFO org.apache.kafka.clients.producer.internals.TransactionManager [] - [Producer clientId=producer-flink-0-24, transactionalId=flink-0-24] Discovered transaction coordinator ubuntu:9092 (id: 0 rack: null) 2022-11-02 10:04:10,233 INFO org.apache.kafka.clients.Metadata [] - [Producer clientId=producer-flink-1-24, transactionalId=flink-1-24] Cluster ID: MCY5mzM1QWyc1YCvsO8jag 2022-11-02 10:04:10,241 INFO org.apache.kafka.clients.producer.internals.TransactionManager [] - [Producer clientId=producer-flink-1-24, transactionalId=flink-1-24] Discovered transaction coordinator ubuntu:9092 (id: 0 rack: null) 2022-11-02 10:04:10,345 INFO org.apache.kafka.clients.producer.internals.TransactionManager [] - [Producer clientId=producer-flink-0-24, transactionalId=flink-0-24] ProducerId set to 51 with epoch 0 2022-11-02 10:04:10,346 INFO org.apache.flink.connector.kafka.sink.KafkaWriter [] - Created new transactional producer flink-0-24 2022-11-02 10:04:10,353 INFO org.apache.kafka.clients.producer.internals.TransactionManager [] - [Producer clientId=producer-flink-1-24, transactionalId=flink-1-24] ProducerId set to 52 with epoch 0 2022-11-02 10:04:10,354 INFO org.apache.flink.connector.kafka.sink.KafkaWriter [] - Created new transactional producer flink-1-24 ProducerConfig values log is repeated for each new producer created (based on sink parallelism level). Configuring checkpoint interval to 10 or 15 seconds, I lose valuable job logs. There is a way to disable these logs without setting WARN level?
InfluxDB not starting: 8086 bind address already in use
I have an InfluxDB Version 1.8.9, but I can't start it. In this example I'm logged in as a root. netstat -lptn gives me a range of services, none of them seem to listen to 8086. (there are other services running like grafana or MySQL, which seem to work fine) To further confirm nothing is on 8086,I listened to that related Issue run: open server: open service: listen tcp :8086: bind: address already in use on starting influxdb and run netstat -a | grep 8086 which results in no results. My config file on /etc/influxdb/influxdb.conf looks like this: reporting-disabled = false bind-address = "127.0.0.1:8086" [meta] #dir = "/root/.influxdb/meta" dir = "/var/lib/influxdb/meta" retention-autocreate = true logging-enabled = true [data] dir = "/var/lib/influxdb/data" index-version = "inmem" wal-dir = "/var/lib/influxdb/wal" wal-fsync-delay = "0s" validate-keys = false strict-error-handling = false query-log-enabled = true cache-max-memory-size = 1073741824 cache-snapshot-memory-size = 26214400 cache-snapshot-write-cold-duration = "10m0s" compact-full-write-cold-duration = "4h0m0s" compact-throughput = 50331648 compact-throughput-burst = 50331648 max-series-per-database = 1000000 max-values-per-tag = 100000 max-concurrent-compactions = 0 max-index-log-file-size = 1048576 series-id-set-cache-size = 100 series-file-max-concurrent-snapshot-compactions = 0 trace-logging-enabled = false tsm-use-madv-willneed = false ... [http] enabled = true bind-address = ":8086" auth-enabled = false log-enabled = true suppress-write-log = false write-tracing = false flux-enabled = false flux-log-enabled = false pprof-enabled = true pprof-auth-enabled = false debug-pprof-enabled = false ping-auth-enabled = false prom-read-auth-enabled = false https-enabled = false https-certificate = "/etc/ssl/influxdb.pem" https-private-key = "" max-row-limit = 0 max-connection-limit = 0 shared-secret = "" realm = "InfluxDB" unix-socket-enabled = false unix-socket-permissions = "0777" bind-socket = "/var/run/influxdb.sock" max-body-size = 25000000 access-log-path = "" max-concurrent-write-limit = 0 max-enqueued-write-limit = 0 enqueued-write-timeout = 30000000000 ... So i tried to start my database: service influxdb start Which gives me ob for influxdb.service failed because a timeout was exceeded. See "systemctl status influxdb.service" and "journalctl -xe" for details. result of systemctl status influxdb.service ● influxdb.service - InfluxDB is an open-source, distributed, time series database Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled) Active: activating (start) since Tue 2021-09-21 18:37:12 CEST; 1min 7s ago Docs: https://docs.influxdata.com/influxdb/ Main PID: 32016 (code=exited, status=1/FAILURE); Control PID: 5874 (influxd-systemd) Tasks: 2 (limit: 4915) CGroup: /system.slice/influxdb.service ├─5874 /bin/bash -e /usr/lib/influxdb/scripts/influxd-systemd-start.sh └─5965 sleep 10 Sep 21 18:37:23 s22227708 influxd-systemd-start.sh[5874]: ts=2021-09-21T16:37:23.515897Z lvl=info msg="Registered diagnostics client" log_id=0WjJLI7l000 service=monitor name=runtime Sep 21 18:37:23 s22227708 influxd-systemd-start.sh[5874]: ts=2021-09-21T16:37:23.515907Z lvl=info msg="Registered diagnostics client" log_id=0WjJLI7l000 service=monitor name=network Sep 21 18:37:23 s22227708 influxd-systemd-start.sh[5874]: ts=2021-09-21T16:37:23.515923Z lvl=info msg="Registered diagnostics client" log_id=0WjJLI7l000 service=monitor name=system Sep 21 18:37:23 s22227708 influxd-systemd-start.sh[5874]: ts=2021-09-21T16:37:23.515977Z lvl=info msg="Starting precreation service" log_id=0WjJLI7l000 service=shard-precreation check_interval=10m advanc Sep 21 18:37:23 s22227708 influxd-systemd-start.sh[5874]: ts=2021-09-21T16:37:23.515995Z lvl=info msg="Starting snapshot service" log_id=0WjJLI7l000 service=snapshot Sep 21 18:37:23 s22227708 influxd-systemd-start.sh[5874]: ts=2021-09-21T16:37:23.516015Z lvl=info msg="Starting continuous query service" log_id=0WjJLI7l000 service=continuous_querier Sep 21 18:37:23 s22227708 influxd-systemd-start.sh[5874]: ts=2021-09-21T16:37:23.516011Z lvl=info msg="Storing statistics" log_id=0WjJLI7l000 service=monitor db_instance=_internal db_rp=monitor interval= Sep 21 18:37:23 s22227708 influxd-systemd-start.sh[5874]: ts=2021-09-21T16:37:23.516037Z lvl=info msg="Starting HTTP service" log_id=0WjJLI7l000 service=httpd authentication=false Sep 21 18:37:23 s22227708 influxd-systemd-start.sh[5874]: ts=2021-09-21T16:37:23.516052Z lvl=info msg="opened HTTP access log" log_id=0WjJLI7l000 service=httpd path=stderr Sep 21 18:37:23 s22227708 influxd-systemd-start.sh[5874]: run: open server: open service: listen tcp :8086: bind: address already in use I can't really understand where I did something wrong, since I configured :8086 in the config file. Can you help me?
It appears to be a typo in the configuration file. As stated in the documentation, the configuration file should hold http-bind-address instead of bind-address. As well as a locked port by the first configuration. The first few lines of the file /etc/influxdb/influxdb.conf should look like so: reporting-disabled = false http-bind-address = "127.0.0.1:8086" A suggested approach would be to: bind-address to http-bind-address Changing the port from default 8086 to a known free port (Optional) Back to the default port.
From your config: reporting-disabled = false bind-address = "127.0.0.1:8086" ... [http] enabled = true bind-address = ":8086" Both your 'native' service and the 'http' service are configured to use the same port 8086. This cannot work and you probably want to change the 'native' port back to its default of 8088.
flink sql read hive table throw java.lang.ArrayIndexOutOfBoundsException: 1024
when i use pyflink hive sql read data insert into es ,throw the follow exeception : the environment ： flink 1.11.2 flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar hive 3.1.2 2020-12-17 21:10:24,398 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: HiveTableSource(driver_id, driver_base_lc_p1, driver_90d_lc_p1, driver_30d_lc_p1, driver_14d_lc_p1, driver_180d_lc_p1, vehicle_base_lc_p1, driver_active_zone, is_incremental, dt) TablePath: algorithm.jiawei_oas_driver_features_for_incremental_hive2kafka, PartitionPruned: false, PartitionNums: null, ProjectedFields: [0, 8, 9] -> Calc(select=[driver_id, is_incremental, dt, () AS bdi_feature_create_time]) -> Sink: Sink(table=[default_catalog.default_database.0_demo4_903157246_tmp], fields=[driver_id, is_incremental, dt, bdi_feature_create_time]) (1/1) (98f4259c3d00fac9fc3482a4cdc8df3c) switched from RUNNING to FAILED. at org.apache.orc.impl.ConvertTreeReaderFactory$AnyIntegerTreeReader.nextVector(ConvertTreeReaderFactory.java:445) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1300) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.flink.connectors.hive.read.HiveVectorizedOrcSplitReader.reachedEnd(HiveVectorizedOrcSplitReader.java:99) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:90) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) ~[flink-dist_2.11-1.11.2.jar:1.11.2] 2020-12-17 21:10:24,402 INFO org.apache.flink.runtime.taskmanager.Task [] - Freeing task resources for Source: HiveTableSource(driver_id, driver_base_lc_p1, driver_90d_lc_p1, driver_30d_lc_p1, driver_14d_lc_p1, driver_180d_lc_p1, vehicle_base_lc_p1, driver_active_zone, is_incremental, dt) TablePath: algorithm.jiawei_oas_driver_features_for_incremental_hive2kafka, PartitionPruned: false, PartitionNums: null, ProjectedFields: [0, 8, 9] -> Calc(select=[driver_id, is_incremental, dt, () AS bdi_feature_create_time]) -> Sink: Sink(table=[default_catalog.default_database.0_demo4_903157246_tmp], fields=[driver_id, is_incremental, dt, bdi_feature_create_time]) (1/1) (98f4259c3d00fac9fc3482a4cdc8df3c). java.lang.ArrayIndexOutOfBoundsException: 1024 at org.apache.flink.orc.shim.OrcShimV210.nextBatch(OrcShimV210.java:35) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.flink.orc.shim.OrcShimV210.nextBatch(OrcShimV210.java:29) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.flink.orc.OrcSplitReader.ensureBatch(OrcSplitReader.java:134) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.orc.impl.TreeReaderFactory$LongTreeReader.nextVector(TreeReaderFactory.java:612) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.orc.impl.TreeReaderFactory$TreeReader.nextVector(TreeReaderFactory.java:269) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.orc.impl.ConvertTreeReaderFactory$StringGroupFromAnyIntegerTreeReader.nextVector(ConvertTreeReaderFactory.java:1477) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextBatch(TreeReaderFactory.java:2012) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.flink.orc.OrcSplitReader.reachedEnd(OrcSplitReader.java:101) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.flink.connectors.hive.read.HiveTableInputFormat.reachedEnd(HiveTableInputFormat.java:261) ~[flink-sql-connector-hive-3.1.2_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) ~[flink-dist_2.11-1.11.2.jar:1.11.2] 2020-12-17 21:10:24,406 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor how to solve this question?
The reason is that ORC format exists bug when array batch size >= 1024, but Flink use default size is 2048 which will meet the ORC format bug. https://issues.apache.org/jira/browse/ORC-598 https://issues.apache.org/jira/browse/ORC-672 We've created an issue to workaround the orc format bug in Flink. https://issues.apache.org/jira/browse/FLINK-20667
Trouble starting dgraph with TLS
I'm trying to start a dgraph server with TLS enabled, my server config file is defined as following:
# Folder in which to store exports.
export: export
# Fraction of dirty posting lists to commit every few seconds.
gentlecommit: 0.33
# RAFT ID that this server will use to join RAFT groups.
idx: 1
# Port to run server on. (default 8080)
port: 8080
# GRPC port to run server on. (default 9080)
grpc_port: 9080
# Port used by worker for internal communication.
workerport: 12345
# Estimated memory the process can take. Actual usage would be slightly more
memory_mb: 4096
# The ratio of queries to trace.
trace: 0.33
# Directory to store posting lists.
p: p
# Directory to store raft write-ahead logs.
w: w
# Debug mode for testing.
debugmode: true
# Address of dgraphzero
peer: localhost:8888
# Use TLS connections with clients.
tls.on: true
# CA Certs file path.
#tls.ca_certs: /Users/pauloferreira/Workspace/RagnarTech/Node/base_backend_njs/certificates/development/development-server-root-CA.pem
# Include System CA into CA Certs.
tls.use_system_ca: true
# Certificate file path.
tls.cert: /Users/pauloferreira/Workspace/RagnarTech/Node/base_backend_njs/certificates/development/development-server-root-CA.pem
# Certificate key file path.
tls.cert_key: /Users/pauloferreira/Workspace/RagnarTech/Node/base_backend_njs/certificates/development/development-server-root-CA.key
# Certificate key passphrase.
#tls.cert_key_passphrase string
# Enable TLS client authentication
#tls.client_auth string
# TLS max version. (default "TLS12")
#tls.max_version string
# TLS min version. (default "TLS11")
#tls.min_version string
As soon as I start dgraphzero and dgraph, if the configuration tls.on is equal true, this output is shown:
Setting up listener at: localhost:8888
Setting up listener at: localhost:8889
2017/10/19 16:09:36 main.go:163: Loading configuration from file: development.conf
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["export" = export]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["grpc_port" = 9080]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["workerport" = 12345]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["p" = p]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["tls.ca_certs" = /Users/pauloferreira/Workspace/RagnarTech/Node/base_backend_njs/certificates/development/development-server-root-CA.pem]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["memory_mb" = 4096]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["peer" = localhost:8888]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["gentlecommit" = 0.33]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["idx" = 1]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["port" = 8080]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["trace" = 0.33]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["tls.on" = true]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["tls.cert" = /Users/pauloferreira/Workspace/RagnarTech/Node/base_backend_njs/certificates/development/development-server-root-CA.pem]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["w" = w]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["debugmode" = true]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["tls.use_system_ca" = true]
2017/10/19 16:09:36 init.go:74: Picked flag from config: ["tls.cert_key" = /Users/pauloferreira/Workspace/RagnarTech/Node/base_backend_njs/certificates/development/development-server-root-CA.key]
Dgraph version : v0.8.3
Commit SHA-1 : 40175d0
Commit timestamp : 2017-10-18 15:55:02 +1100
Branch : HEAD
2017/10/19 16:09:36 node.go:234: Found hardstate: {Term:2 Vote:1 Commit:4 XXX_unrecognized:[]}
2017/10/19 16:09:36 node.go:246: Group 0 found 4 entries
2017/10/19 16:09:36 raft.go:292: Restarting node for dgraphzero
2017/10/19 16:09:36 raft.go:567: INFO: 1 became follower at term 2
2017/10/19 16:09:36 raft.go:315: INFO: newRaft 1 [peers: [], term: 2, commit: 4, applied: 0, lastindex: 4, lastterm: 2]
Running Dgraph zero...
2017/10/19 16:09:36 open : no such file or directory
I can't find what is causing the error open : no such file or directory, anyone experienced this? I'm using MacOS 10.12.3 (16D32) and installed dgraph version v0.8.3 using the command curl https://get.dgraph.io -sSf | bash
Thanks in advance.
I think this is a bug (update: it actually was confirmed as a bug and was fixed).
I've tried running it on Ubuntu and I've got the same error with tls.on.
Next I found the semi-manual test suite for tls here.
Running it confirmed the error, tests required small adjustments (add --memory_mb 2048), but after that the same failure was reproduced.
To confirm it I also downloaded dgraph sources and checked what's going on under delve debugger:
1) The config file is parsed and parameters are saved into global vars
2) TLS-related parameters are used to create the tlsCfg
- Here we already can see the problem: not all the parameters are passed, for example, tlsKey and tlsKeyPath are missing
3) If we look deeper, into tls_helper.go, where the TLS actually configured, we can find that parameters from the config are
passed into the parseCertificate method
4) Here we use config.Key and config.KeyPassphrase, but they are empty
182: func GenerateTLSConfig(config TLSHelperConfig) (tlsCfg *tls.Config, reloadConfig func(), err error) {
183: wrapper := new(wrapperTLSConfig)
184: tlsCfg = new(tls.Config)
185: wrapper.config = tlsCfg
186:
=> 187: cert, err := parseCertificate(config.CertRequired, config.Cert, config.Key, config.KeyPassphrase)
188: if err != nil {
189: return nil, nil, err
190: }
191:
192: if cert != nil {
(dlv) p config.CertRequired
true
(dlv) p config.Cert
"/home/seb/web/dgraph-test/test2.crt"
(dlv) p config.Key
""
(dlv) p config.KeyPassphrase
Then it fails inside the parseCertificate when it tries to read the file with certificate key.
I posted the issue on github.
How to set RocksDBStateBackend parameter in Flink?
I use the follow code to set RocksDBStateBackend and it option, it can run correctly locally, but can't be submitted to cluster.
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
RocksDBStateBackend rocksDBBackEnd = new RocksDBStateBackend("file:///Users/zsh/tmp/rocksdb");
rocksDBBackEnd.setOptions(new OptionsFactory() {
#Override
public DBOptions createDBOptions(DBOptions currentOptions) {
return currentOptions;
}
#Override
public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions) {
final long blockCacheSize = 8 * 1024 * 1024;
final long blockSize = 4 * 1024;
final long targetFileSize = 2 * 1024 * 1024;
final long writeBufferSize = 64 * 1024 * 1024;
final int writeBufferNum = 1; //default 2
final int minBufferToMerge = 1; //default 2
return currentOptions
.setCompactionStyle(CompactionStyle.LEVEL)
.setTargetFileSizeBase(targetFileSize)
.setWriteBufferSize(writeBufferSize)
.setMaxWriteBufferNumber(writeBufferNum)
.setMinWriteBufferNumberToMerge(minBufferToMerge)
.setTableFormatConfig(
new BlockBasedTableConfig()
.setBlockCacheSize(blockCacheSize)
.setBlockSize(blockSize)
);
}
});
env.setStateBackend(rocksDBBackEnd);
....
env.execute();
When i submit my job this way:
flink run -d -c gerryzhou.metricTest target/gerryzhou.flink-1.0-SNAPSHOT.jar
it throw below exception:
org.apache.flink.client.program.ProgramInvocationException: The program execution failed: JobManager did not respond within 60000 milliseconds
at org.apache.flink.client.program.ClusterClient.runDetached(ClusterClient.java:505)
at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:103)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442)
at org.apache.flink.client.program.DetachedEnvironment.finalizeExecute(DetachedEnvironment.java:76)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:387)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1129)
And the jobmanager.log look like this
2017-06-29 15:37:16,651 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#10.242.98.255:51891] has failed, address is now gated for [5000] ms. Reason: [gerryzhou.metricTest$1]
2017-06-29 15:37:16,651 ERROR Remoting - gerryzhou.metricTest$1
java.lang.ClassNotFoundException: gerryzhou.metricTest$1
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:677)
I have changed my code and implemented the OptionsFactory with a single class file MRocksDBFactory, use it like this rocksDBBackEnd.setOptions(new MRocksDBFactory());. The error info in jobManager.log become this:
2017-06-29 16:29:27,162 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#10.242.98.255:52638] has failed, address is now gated for [5000] ms. Reason: [gerryzhou.MRocksDBFactory]
2017-06-29 16:29:27,163 ERROR Remoting - gerryzhou.MRocksDBFactory
java.lang.ClassNotFoundException: gerryzhou.MRocksDBFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:677)
at akka.util.ClassLoaderObjectInputStream.resolveClass(ClassLoaderObjectInputStream.scala:19)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1819)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1986)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:192)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:967)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:437)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Can anybody help me ?