Issue in writing data from Apache Kafka to text file - file

Question

My development environment setup: Windows 10 Enterprise Edition, 16GB RAM, 2.81GHz 64Bit OS. I installed Virtual Box and imported Ubuntu image in it. With in Ubuntu I installed Confluent CLI https://github.com/confluentinc/confluent-cli to run Kafka, zookeeper and other services.
Scenario: I want to write data from Apache Kafka topic to a text file. I am using Sink connector and following the below link to accomplish this task. Also, I didn't write any code to accomplish the same.
Using this Link to accomplish my task
http://bigdatums.net/2017/06/22/writing-data-from-apache-kafka-to-text-file/
Steps accomplished successfully so far:
Able to run Ubuntu image with in Virtual Box.
Able to run Confluent CLI.
Able to bring up Confluent Kafka, ZooKeeper and other services using bin/confluent start command.
Able to create Topics within Confluent CLI
Try to run the following message to read the text message from Kafka Topic
osboxes#osboxes:~/ganesh/confluent-5.1.0$ bin/connect-standalone /home/osboxes/ganesh/confluent-5.1.0/etc/kafka/csx-connect-standalone.properties /home/osboxes/ganesh/confluent-5.1.0/etc/kafka/csx-connect-file-sink.properties
Property configuration details below
connect-file-sink.properties (details)
name=local-file-sink
connector.class=FileStreamSink
tasks.max=1
file=/home/osboxes/ganesh/ptc/messages/output/trainstartevent/MBCDTSKB02.json
topics=TrainStartEvent
connect-file-source.properties (details)
name=local-file-source
connector.class=FileStreamSource
tasks.max=1
file=/home/osboxes/ganesh/ptc/messages/input/trainstartevent/MBCDTSKB02.json
topic=TrainStartEvent
connect-standalone.properties (details)
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.flush.interval.ms=10000
plugin.path=share/java
Actual Error Message
[2019-01-20 21:14:17,413] INFO Started
o.e.j.s.ServletContextHandler#546394ed{/,null,AVAILABLE}
(org.eclipse.jetty.server.handler.ContextHandler:850)
[2019-01-20 21:14:17,428] ERROR Stopping after connector error
(org.apache.kafka.connect.cli.ConnectStandalone:113)
org.apache.kafka.connect.errors.ConnectException: Unable to start REST
server
[2019-01-20 21:13:19,927] INFO Kafka Connect standalone worker
initializing ... (org.apache.kafka.connect.cli.ConnectStandalone:67)
[2019-01-20 21:13:20,021] INFO WorkerInfo values:
jvm.args = -Xms256M, -Xmx2G, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20,
-XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=bin/../logs, -Dlog4j.configuration=file:bin/../etc/kafka/connect-log4j.properties
jvm.spec = Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM,
1.8.0_201, 25.201-b09
[2019-01-20 21:14:13,427] WARN The configuration 'plugin.path' was
supplied but isn't a known config.
(org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-01-20 21:14:13,431] WARN The configuration 'value.converter' was
supplied but isn't a known config.
(org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-01-20 21:14:13,431] WARN The configuration
'internal.key.converter.schemas.enable' was supplied but isn't a known
config. (org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-01-20 21:14:13,432] WARN The configuration 'key.converter' was
supplied but isn't a known config.
(org.apache.kafka.clients.admin.AdminClientConfig:287)
[2019-01-20 21:14:13,433] INFO Kafka version : 2.1.0-cp1
(org.apache.kafka.common.utils.AppInfoParser:109)
[2019-01-20 21:14:13,433] INFO Kafka commitId : 3bce825d5f759863
(org.apache.kafka.common.utils.AppInfoParser:110)
[2019-01-20 21:14:14,047] INFO Kafka cluster ID:
jPHHwv39Riyn1krFQyhYkA (org.apache.kafka.connect.util.ConnectUtils:59)
[2019-01-20 21:14:14,139] INFO Logging initialized #55198ms to
org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log:193)
[2019-01-20 21:14:14,565] INFO Added connector for http://:8083
(org.apache.kafka.connect.runtime.rest.RestServer:119)
[2019-01-20 21:14:14,681] INFO Advertised URI: http://127.0.1.1:8083/
(org.apache.kafka.connect.runtime.rest.RestServer:267)
[2019-01-20 21:14:14,705] INFO Kafka version : 2.1.0-cp1
(org.apache.kafka.common.utils.AppInfoParser:109)
[2019-01-20 21:14:14,705] INFO Kafka commitId : 3bce825d5f759863
(org.apache.kafka.common.utils.AppInfoParser:110)
[2019-01-20 21:14:15,228] INFO JsonConverterConfig values:
converter.type = key
schemas.cache.size = 1000
schemas.enable = false
(org.apache.kafka.connect.json.JsonConverterConfig:279)
[2019-01-20 21:14:15,238] INFO JsonConverterConfig values:
converter.type = value
schemas.cache.size = 1000
schemas.enable = false
(org.apache.kafka.connect.json.JsonConverterConfig:279)
[2019-01-20 21:14:15,251] INFO Kafka Connect standalone worker
initialization took 55315ms
(org.apache.kafka.connect.cli.ConnectStandalone:92)
[2019-01-20 21:14:15,251] INFO Kafka Connect starting
(org.apache.kafka.connect.runtime.Connect:49)
[2019-01-20 21:14:15,256] INFO Herder starting
(org.apache.kafka.connect.runtime.standalone.StandaloneHerder:88)
[2019-01-20 21:14:15,256] INFO Worker starting
(org.apache.kafka.connect.runtime.Worker:172)
[2019-01-20 21:14:15,256] INFO Starting FileOffsetBackingStore with
file /tmp/connect.offsets
(org.apache.kafka.connect.storage.FileOffsetBackingStore:58)
[2019-01-20 21:14:15,258] INFO Worker started
(org.apache.kafka.connect.runtime.Worker:177)
[2019-01-20 21:14:15,259] INFO Herder started
(org.apache.kafka.connect.runtime.standalone.StandaloneHerder:90)
[2019-01-20 21:14:15,259] INFO Starting REST server
(org.apache.kafka.connect.runtime.rest.RestServer:163)
[2019-01-20 21:14:15,565] INFO jetty-9.4.12.v20180830; built:
2018-08-30T13:59:14.071Z; git:
27208684755d94a92186989f695db2d7b21ebc51; jvm 1.8.0_201-b09
(org.eclipse.jetty.server.Server:371)
[2019-01-20 21:14:15,733] INFO DefaultSessionIdManager
workerName=node0 (org.eclipse.jetty.server.session:365)
[2019-01-20 21:14:15,746] INFO No SessionScavenger set, using defaults
(org.eclipse.jetty.server.session:370)
[2019-01-20 21:14:15,748] INFO node0 Scavenging every 600000ms
(org.eclipse.jetty.server.session:149)
Jan 20, 2019 9:14:16 PM org.glassfish.jersey.internal.inject.Providers
checkProviderRuntime
WARNING: A provider
org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
registered in SERVER runtime does not implement any provider
interfaces applicable in the SERVER runtime. Due to constraint
configuration problems the provider
org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
will be ignored.
Jan 20, 2019 9:14:16 PM org.glassfish.jersey.internal.inject.Providers
checkProviderRuntime
WARNING: A provider
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource
registered in SERVER runtime does not implement any provider
interfaces applicable in the SERVER runtime. Due to constraint
configuration problems the provider
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource
will be ignored.
Jan 20, 2019 9:14:16 PM org.glassfish.jersey.internal.inject.Providers
checkProviderRuntime
WARNING: A provider
org.apache.kafka.connect.runtime.rest.resources.RootResource
registered in SERVER runtime does not implement any provider
interfaces applicable in the SERVER runtime. Due to constraint
configuration problems the provider
org.apache.kafka.connect.runtime.rest.resources.RootResource will be
ignored.
Jan 20, 2019 9:14:17 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The
(sub)resource method listConnectors in
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource
contains empty path annotation.
WARNING: The (sub)resource method createConnector in
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource
contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in
org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
contains empty path annotation.
WARNING: The (sub)resource method serverInfo in
org.apache.kafka.connect.runtime.rest.resources.RootResource contains
empty path annotation.
[2019-01-20 21:14:17,413] INFO Started
o.e.j.s.ServletContextHandler#546394ed{/,null,AVAILABLE}
(org.eclipse.jetty.server.handler.ContextHandler:850)
[2019-01-20 21:14:17,428] ERROR Stopping after connector error
(org.apache.kafka.connect.cli.ConnectStandalone:113)
org.apache.kafka.connect.errors.ConnectException: Unable to start REST
server
at
org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:214)
at org.apache.kafka.connect.runtime.Connect.start(Connect.java:53)
at
org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:95)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at
org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:339)
at
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:307)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:395)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:212)
... 2 more
[2019-01-20 21:14:17,437] INFO Kafka Connect stopping
(org.apache.kafka.connect.runtime.Connect:65)
[2019-01-20 21:14:17,437] INFO Stopping REST server
(org.apache.kafka.connect.runtime.rest.RestServer:223)
[2019-01-20 21:14:17,442] INFO Stopped
http_8083#1b90fee4{HTTP/1.1,[http/1.1]}{0.0.0.0:8083}
(org.eclipse.jetty.server.AbstractConnector:341)
[2019-01-20 21:14:17,460] INFO node0 Stopped scavenging
(org.eclipse.jetty.server.session:167)
[2019-01-20 21:14:17,493] INFO Stopped
o.e.j.s.ServletContextHandler#546394ed{/,null,UNAVAILABLE}
(org.eclipse.jetty.server.handler.ContextHandler:1040)
[2019-01-20 21:14:17,507] INFO REST server stopped
(org.apache.kafka.connect.runtime.rest.RestServer:241)
[2019-01-20 21:14:17,508] INFO Herder stopping
(org.apache.kafka.connect.runtime.standalone.StandaloneHerder:95)
[2019-01-20 21:14:17,509] INFO Worker stopping
(org.apache.kafka.connect.runtime.Worker:184)
[2019-01-20 21:14:17,510] INFO Stopped FileOffsetBackingStore
(org.apache.kafka.connect.storage.FileOffsetBackingStore:66)
[2019-01-20 21:14:17,522] INFO Worker stopped
(org.apache.kafka.connect.runtime.Worker:205)
[2019-01-20 21:14:17,523] INFO Herder stopped
(org.apache.kafka.connect.runtime.standalone.StandaloneHerder:112)
[2019-01-20 21:14:17,529] INFO Kafka Connect stopped
(org.apache.kafka.connect.runtime.Connect:70)

Answer 1

Caused by: java.net.BindException: Address already in use
Sound like you ran confluent start and a Kafka Connect server is therefore already running at port 8083.
You therefore need to use confluent load /home/osboxes/ganesh/confluent-5.1.0/etc/kafka/csx-connect-file-sink.properties, or convert the properties file into JSON, which you can then do curl -XPOST -d#csx-connect-file-sink.json http://localhost:8083
See Kafka Connect REST API
Note that to write to a file, you could also do it all from the console consumer
kafka-console-consumer --from-beginning --property print.key=true --topic x --bootstrap-server localhost:9092 --group to-file >> /tmp/file.txt