NoSuchMethodError in apache camel java - apache-camel

I ran the following code:
package com.dinesh.example4;
import javax.jms.ConnectionFactory;
//import org.apache.camel.support.HeaderFilterStrategyComponent;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.camel.CamelContext;
import org.apache.camel.Component;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.jms.JmsComponent;
import org.apache.camel.impl.DefaultCamelContext;
//import org.apache.camel.impl.*;
public class FileToActiveMQ {
public static void main(String[] args) throws Exception {
CamelContext context = new DefaultCamelContext();
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory();
context.addComponent("jms",JmsComponent.jmsComponentAutoAcknowledge(connectionFactory));
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from("file:input_box?noop=true")
.to("activemq:queue:my_queue");
}
});
while(true)
context.start();
}
}
for transforming data from input_box folder to activemq.
I am getting the following error:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.camel.impl.HeaderFilterStrategyComponent.<init>(Ljava/lang/Class;)V
at org.apache.camel.component.jms.JmsComponent.<init>(JmsComponent.java:71)
at org.apache.camel.component.jms.JmsComponent.<init>(JmsComponent.java:87)
at org.apache.camel.component.jms.JmsComponent.jmsComponent(JmsComponent.java:102)
at org.apache.camel.component.jms.JmsComponent.jmsComponentAutoAcknowledge(JmsComponent.java:127)
at com.dinesh.example4.FileToActiveMQ.main(FileToActiveMQ.java:18)
Here 18 th line in above code:
context.addRoutes(new RouteBuilder() {
Pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dinesh</groupId>
<artifactId>camel-application1</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.camel/camel-core -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>2.14.4</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jms</artifactId>
<version>2.24.0</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-activemq</artifactId>
<version>3.0.0</version>
</dependency>
</dependencies>
</project>
Please help.

According to your POM you mix 3 different Camel versions: 2.14.4, 2.24.0 and 3.0.0.
You have to use the same version for all Camel components, as #claus-ibsen already commented.
Do it for example like the example below (with properties for the framework versions and use it in all its dependencies).
However, as Sneharghya already answered, Camel 2.x has no camel-activemq but instead can use the dependency activemq-camel of ActiveMQ.
Therefore the POM should look like this. But I think the Camel version can vary.
<properties>
<amq.version>5.15.4</amq.version>
<camel.version>2.19.5</camel.version>
</properties>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>${camel.version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jms</artifactId>
<version>${camel.version}</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-camel</artifactId>
<version>${amq.version}</version>
</dependency>
You can also use Maven dependencyManagement.

camel-activemq is not available for versions older than 3.0.
If you want to keep using camel 2.24.3, then remove the camel-activemq dependency from your pom file and add
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-camel</artifactId>
<version>5.15.13</version>
</dependency>

1. Update the class
There are two issues.
One you were registering your component in one name jms and send the message to different component activemq. They should be same.
You were doing a while loop and instead the while loop starting the context many times.
public class FileToActiveMQ {
public static void main(String[] args) throws Exception {
CamelContext context = new DefaultCamelContext();
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory();
context.addComponent("activemq",
JmsComponent.jmsComponentAutoAcknowledge(connectionFactory));
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from("file:input_box?noop=true")
.to("activemq:queue:my_queue");
}
});
context.start();
while(true) {
Thread.sleep(10000);
}
}
}
2. Replace the dependencies in pom as follows:
<dependencies>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>2.25.1</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jms</artifactId>
<version>2.25.1</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-client</artifactId>
<version>5.15.7</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-camel</artifactId>
<version>5.15.7</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-pool</artifactId>
<version>5.15.7</version>
</dependency>
</dependencies>

Related

Apache Flink: java.lang.NoClassDefFoundError for FlinkKafkaConsumer

I am trying to test simple flink kafka example.
mvn package works fine and then I ran ../../flink-1.14.3/bin/flink run -c com.comapny.flinktest.App target/flinktest-1.0-SNAPSHOT.jar and got the following error:
java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer
at com.optiver.flinktest.App.main(App.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
My pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.company.flinktest</groupId>
<artifactId>flinktest</artifactId>
<version>1.0-SNAPSHOT</version>
<name>flinktest</name>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-core</artifactId>
<version>1.14.3</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.14.3</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.12</artifactId>
<version>1.14.3</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.12</artifactId>
<version>1.14.3</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.12</artifactId>
<version>1.14.3</version>
<scope>Compile</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<archive>
<manifest>
<mainClass>com.company.flinktest.App</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
</project>
Folder structure: src/main/java/com/company/flinktest/App.java
Editor: Vscode
App.java Code:
package com.company.flinktest;
import java.util.Properties;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import java.util.InvalidPropertiesFormatException;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.apache.flink.streaming.util.serialization.SimpleStringSchema;
public class App{
public static void main(String[] args) throws Exception{
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "kafkaserver:9092");
properties.setProperty("group.id", "testgroup");
FlinkKafkaConsumer<String> myConsumer = new FlinkKafkaConsumer<String>(
"mytopic",
new SimpleStringSchema(),
properties);
DataStream<String> stream = env.addSource(myConsumer);
stream.print();
env.execute("flink from kafka");
}
}
It looks like your post is mostly code; please add some more details.
It looks like your post is mostly code; please add some more details.
It looks like your post is mostly code; please add some more details.
It looks like your post is mostly code; please add some more details.
It looks like your post is mostly code; please add some more details.
It looks like your post is mostly code; please add some more details.
You can refer to my answer in one of the posts. Check the Scala language version though.

Apache Camel 3.x + Spring Boot + JDBC DataSource: java.lang.IllegalStateException: No supported DataSource type found

This is no doubt a configuration/dependency issue.
failing code snippet
#Bean
#ConfigurationProperties(prefix="rmw.datasource")
public DataSource getDataSource() {
return DataSourceBuilder.create().build();
}
application.properties
rmw.datasource.url=jdbc:sqlserver://123.123.123.123
rmw.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
rmw.datasource.username=xxxx
rmw.datasource.password=zzzz
rmw.datasource.platform=sqlserver
I have tried with and without rmw.datasource.platform=sqlserver - it makes no difference.
pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>biz.suitej.poc</groupId>
<artifactId>http-producer</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<java.version>1.11</java.version>
<camel.version>3.1.0</camel.version>
<maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-spring-boot-dependencies</artifactId>
<version>${camel.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-core-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-http-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-log-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-jackson-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-jdbc-starter</artifactId>
</dependency>
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>8.2.2.jre11</version>
</dependency>
</dependencies>
<build>
<finalName>${project.artifactId}</finalName>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler-plugin.version}</version>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
I have camel-jdbc-starter and mssql-jdbc JDBC driver.
I have tried camel-sql-starter and get the same error.
console output
2020-06-17 13:15:45.978 WARN 128833 --- [ main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getDataSource' defined in class path resource [c10y/sql/query/GetRemodelWorksRouteBuilder.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method 'getDataSource' threw exception; nested exception is java.lang.IllegalStateException: No supported DataSource type found
2020-06-17 13:15:45.998 INFO 128833 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-06-17 13:15:46.009 ERROR 128833 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getDataSource' defined in class path resource [c10y/sql/query/GetRemodelWorksRouteBuilder.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method 'getDataSource' threw exception; nested exception is java.lang.IllegalStateException: No supported DataSource type found
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:484) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
...
The solution listed elsewhere to add spring-boot-starter-data-jpa didn't work for me (it just returned a different error):
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
<version>1.5.6.RELEASE</version>
</dependency>
I'm baffled why this just doesn't work 'straight out of the box'.
Thanks in advance for your help.
Just after posting this I found the solution that worked for me:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
<version>1.5.6.RELEASE</version>
</dependency>

kafka flink connection error shows NoSuchMethodError

new error appeared when i change from flinkkafkaconsumer09 to flinkkafkaconsumer
Flink code:
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.apache.flink.streaming.util.serialization.SimpleStringSchema;
import java.util.Properties;
#SuppressWarnings("deprecation")
public class ReadFromKafka {
public static void main(String[] args) throws Exception {
// create execution environment
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "localhost:9092");
properties.setProperty("group.id", "test-consumer-group");
DataStream<String> stream = env
.addSource(new FlinkKafkaConsumer<String>("test4", new SimpleStringSchema(), properties));
stream.map(new MapFunction<String, String>() {
private static final long serialVersionUID = -6867736771747690202L;
#Override
public String map(String value) throws Exception {
return "Stream Value: " + value;
}
}).print();
env.execute();
}
}
ERROR:
log4j:WARN No appenders could be found for logger (org.apache.flink.api.java.ClosureCleaner).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:626)
at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:117)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1507)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1489)
at ReadFromKafka.main(ReadFromKafka.java:33)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dataartisans</groupId>
<artifactId>kafka-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>kafkaex</name>
<description>this is flink kafka example</description>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.9.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.12</artifactId>
<version>1.9.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.12</artifactId>
<version>1.9.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.12</artifactId>
<version>1.9.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-core</artifactId>
<version>1.9.1</version>
</dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1</version>
</dependency>
</dependencies>
</project>
flink-connector-kafka_2.12 isn't compatible with FlinkKafkaConsumer09.
flink-connector-kafka_2.12 is a "universal" kafka connector, compiled for use with Scala 2.12. This universal connector can be used with any version of Kafka from 0.11.0 onward.
FlinkKafkaConsumer09 is for use with Kafka 0.9.x. If your Kafka broker is running Kafka 0.9.x, then you will need flink-connector-kafka-0.9_2.11 or flink-connector-kafka-0.9_2.12, depending on which version of Scala you want.
On the other hand, if your Kafka broker is running a recent version of Kafka (0.11.0 or newer), then stick with flink-connector-kafka_2.12 and use FlinkKafkaConsumer instead of FlinkKafkaConsumer09.
See the documentation for more info.

Problems with hystrix-dashboard turbine using FileBasedInstanceDiscovery

I am trying to set up a hystrix-dashboard with turbine. I am not using Eureka and want to use the FileBasedInstanceDiscovery. I am trying to configure this but following the online documentation doesn't seem to work. It always tries to use the Eureka discovery client. I tried excluding the eureka from my pom but then it falls back to another discovery client CommonsInstanceDiscovery
Here is my application.properties:
turbine.aggregator.clusterConf=mycluster
turbine.instanceUrlSuffix.mycluster=8080/hystrix.stream
turbine.FileBasedInstanceDiscovery.filePath=turbine.hostnames.txt
turbine.InstanceMonitor.eventStream.skipLineLogic.enabled=false
InstanceDiscovery.impl=com.netflix.turbine.discovery.FileBasedInstanceDiscovery
Is there another way to do this? Also, for the filePath for my turbine hostnames, where does it start looking? Can I have the file under my resources directory of my jar?
here is my pom file dependencies:
<properties>
<java.version>1.8</java.version>
<spring-cloud.version>Greenwich.SR1</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web-services</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-turbine</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
Some of the documentation is unclear and conflicting for turbine, I think this is because there is the spring-cloud-netflix project and a standalone turbine project. At any rate, if you are building a spring-boot app then this property is not useful:
InstanceDiscovery.impl=com.netflix.turbine.discovery.FileBasedInstanceDiscovery
If you want to change the implementation of InstanceDiscovery it is pretty simple, just create a bean that implements InstanceDiscovery like this:
#Bean
public InstanceDiscovery instanceDiscovery() {
//choose either one of the provided implementations from spring or
//create your own
return new ConfigPropertyBasedDiscovery();
//return new FileBasedInstanceDiscovery();
}
Make sure you put this in a #Configuration class. I originally had mine just in my SpringBootApplication class but this doesn't allow you to override the default implementation.

How to write directly druid from Flink?

I want to write the data in flink to druid.
I know the tranquility which enables above scenario however I cannot find the working project written in Java.
If someone achieve the above scenario, I really appreciate it if you could guide me how to resolve my issue.
Thanks.
Assuming that Apache Flink and Druid are running in local.
All you need to do is enable Tranquility server for Druid as defined in the link https://druid.apache.org/docs/latest/tutorials/tutorial-tranquility.html
If your tranquility server configurations for Druid are set correctly. The following Spring Boot Java Code with Flink should allow you to push data into Druid using Http Post method.
FlinkDruidApp.java
#SpringBootApplication
public class FlinkDruidApp {
private static String url = "http://localhost:8200/v1/post/wikipedia";
private static RestTemplate template;
private static HttpHeaders headers;
FlinkDruidApp() {
template = new RestTemplate();
headers = new HttpHeaders();
headers.setAccept(Arrays.asList(MediaType.APPLICATION_JSON));
headers.setContentType(MediaType.APPLICATION_JSON);
}
public static void main(String[] args) throws Exception {
SpringApplication.run(FlinkDruidApp.class, args);
// Creating Flink Execution Environment
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
//Define data source
DataSet<String> data = env.readTextFile("/wikiticker-2015-09-12-sampled.json");
// Trasformation on the data
data.map(x -> {
return httpsPost(x).toString();
}).print();
}
// http post method to post data in Druid
private static ResponseEntity<String> httpsPost(String json) {
HttpEntity<String> requestEntity =
new HttpEntity<>(json, headers);
ResponseEntity<String> response =
template.exchange("http://localhost:8200/v1/post/wikipedia", HttpMethod.POST, requestEntity,
String.class);
return response;
}
#Bean
public RestTemplate restTemplate() {
return new RestTemplate();
}
}
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.8.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.flinkdruid</groupId>
<artifactId>FlinkDruid</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>FlinkDruid</name>
<description>Flink Druid Connection</description>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-core</artifactId>
<version>1.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.12</artifactId>
<version>1.9.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
Output on console
<200,{"result":{"received":1,"sent":1}},[Date:"Mon, 23 Sep 2019 13:29:39 GMT", Content-Type:"application/json; charset=UTF-8", Content-Length:"34", Server:"Jetty(9.2.5.v20141112)"]>

Resources