I have a below example flink application which I am trying to run on a cluster.
public class ClusterConnect {
public static void main(String[] args) throws Exception {
ExecutionEnvironment env = ExecutionEnvironment
.createRemoteEnvironment("X.X.X.X", 6123, "");
// get input data
DataSet<String> text = env.fromElements("To be, or not to be,--that is the question:--",
"Whether 'tis nobler in the mind to suffer", "The slings and arrows of outrageous fortune",
"Or to take arms against a sea of troubles,");
DataSet<Tuple2<String, Integer>> counts = text
.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
#Override
public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
for (String word : s.split(" ")) {
collector.collect(new Tuple2<String, Integer>(word, 1));
}
}
})
.groupBy(0)
.sum(1);
// execute and print result
counts.print();
env.execute();
}
Cluster is setup with one jobmanager(free aws instance) and two tasks managers(free aws instances). While trying to run the above flink application on different AWS(which could reach the jobmanager, taskmanagers) hitting the following error.
Error from application:
WARN [akka.remote.ReliableDeliverySupervisor] Association with remote system [akka.tcp://flink#172.31.29.190:6123] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
Logs from cluster job manager:
016-11-30 22:00:42,796 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#172.31.6.190:33619] has failed, address is now gated for [5000] ms. Reason is: [scala.Option; local class incompatible: stream classdesc serialVersionUID = -2062608324514658839, local class serialVersionUID = -114498752079829388]
Related
I am trying to access adls gen2 file from apache flink using flink-azure-fs-hadoop.jar with the following code snippet and getting "java.lang.IllegalArgumentException : invalid authority" for the adls file , the same service principal that have been used has the access to the container, but still showing the error.
public class FlinkAvroADLSgen2 {
public static void main(String[] args) throws Exception{
final StreamExecutionEnvironment streamexecenv = StreamExecutionEnvironment.getExecutionEnvironment();
Configuration conf = new Configuration();
conf.setString("fs.azure.account.auth.type", "OAuth");
conf.setString("fs.azure.account.oauth.provider.type", "");
conf.setString("fs.azure.account.oauth2.client.id", "");
conf.setString("fs.azure.account.oauth2.client.secret", "");
conf.setString("fs.azure.account.oauth2.client.endpoint", "https://login.microsoftonline.com//oauth2/token");
String mountUrl = "abfss:///container#storageaccount.dfs.core.windows.net/sample.txt";
URI uri = URI.create(mountUrl);
AbstractAzureFSFactory azureFSFactory = new SecureAzureDataLakeStoreGen2FSFactory();
azureFSFactory.configure(conf);
azureFSFactory.create(uri);
}
}
I have following code in my flink job;
#Override
public void open(Configuration config) {
this.counter = getRuntimeContext()
.getMetricGroup()
.counter("myCounter");
}
#Override
public Tuple2<String, String> map(String s) throws Exception {
this.counter.inc();
Thread.sleep(5000);
return new Tuple2<String, String>(s, s.toUpperCase());
}
In prometheus.yml inside prometheus distribution, I have following:
- job_name: 'flink-prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9999']
metrics_path: /
And in flink-conf.yaml inside flink distribution:
metrics.reporters: prom
metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
metrics.reporter.prom.host: 127.0.0.1
metrics.reporter.prom.port: 9999
On prometheus board, I can see localhost:9999 as target, and also various metric logs. But there is no log for the counter I have added in the code. I searched for string "myCounter" as well as "flink-prometheus", but zero results.
What else I need to do for my metrics to show up?
The main difference I see between the example in https://github.com/mbode/flink-prometheus-example and your own config is that the example is scraping the job manager as well as the task manager(s):
scrape_configs:
- job_name: 'flink'
static_configs:
- targets: ['job-cluster:9249', 'taskmanager1:9249', 'taskmanager2:9249']
In my own example -- see Flink Timing Explorer -- I found it necessary to do this as well. Here's what worked for me:
flink-conf.yaml
metrics.reporters: prom
metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
metrics.reporter.prom.port: 9250-9260
prometheus.yaml
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: 'flink'
static_configs:
- targets: ['host.docker.internal:9250', 'host.docker.internal:9251']
I am creating an application using Apache Camel to transfer messages from AMQP to Kafka. Code can also be seen here - https://github.com/prashantbhardwaj/qpid-to-kafka-using-camel
I thought of creating it as standalone SpringBoot app using spring, amqp and kafka starters. Created a route like
#Component
public class QpidToKafkaRoute extends RouteBuilder {
public void configure() throws Exception {
from("amqp:queue:destinationName")
.to("kafka:topic");
}
}
And SpringBoot application configuration is
#SpringBootApplication
public class CamelSpringJmsKafkaApplication {
public static void main(String[] args) {
SpringApplication.run(CamelSpringJmsKafkaApplication.class, args);
}
#Bean
public JmsConnectionFactory jmsConnectionFactory(#Value("${qpidUser}") String qpidUser, #Value("${qpidPassword}") String qpidPassword, #Value("${qpidBrokerUrl}") String qpidBrokerUrl) {
JmsConnectionFactory jmsConnectionFactory = new JmsConnectionFactory(qpidPassword, qpidPassword, qpidBrokerUrl);
return jmsConnectionFactory;
}
#Bean
#Primary
public CachingConnectionFactory jmsCachingConnectionFactory(JmsConnectionFactory jmsConnectionFactory) {
CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory(jmsConnectionFactory);
return cachingConnectionFactory;
}
jmsConnectionFactory bean which is created using Spring Bean annotation should be picked by amqp starter and should be injected into the route. But it is not happening. When I started this application, I got following exception -
org.apache.camel.FailedToStartRouteException: Failed to start route route1 because of Route(route1)[From[amqp:queue:destinationName] -> [To[kafka:.
Caused by: java.lang.IllegalArgumentException: connectionFactory must be specified
If I am not wrong connectionFactory should be created automatically if I pass right properties in application.properties file.
My application.properties file looks like :
camel.springboot.main-run-controller = true
camel.component.amqp.enabled = true
camel.component.amqp.connection-factory = jmsCachingConnectionFactory
camel.component.amqp.async-consumer = true
camel.component.amqp.concurrent-consumers = 1
camel.component.amqp.map-jms-message = true
camel.component.amqp.test-connection-on-startup = true
camel.component.kafka.brokers = localhost:9092
qpidBrokerUrl = amqp://localhost:5672?jms.username=guest&jms.password=guest&jms.clientID=clientid2&amqp.vhost=default
qpidUser = guest
qpidPassword = guest
Could you please help suggest why during autoconfiguring connectionFactory object is not being used? When I debug this code, I can clearly see that connectionFactory bean is getting created.
I can even see one more log line -
CamelContext has only been running for less than a second. If you intend to run Camel for a longer time then you can set the property camel.springboot.main-run-controller=true in application.properties or add spring-boot-starter-web JAR to the classpath.
however if you see my application.properties file, required property is present at the very first line.
One more log line, I can see at the beginning of application startup -
[main] trationDelegate$BeanPostProcessorChecker : Bean 'org.apache.camel.spring.boot.CamelAutoConfiguration' of type [org.apache.camel.spring.boot.CamelAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
Is this log line suggesting anything?
Note - One interesting fact that exactly same code was running fine last night, just restarted my desktop and there is not even a single word changed and now it is throwing exception.
This just refers to an interface
camel.component.amqp.connection-factory = javax.jms.ConnectionFactory
Instead it should refer to an existing factory instance, such as
camel.component.amqp.connection-factory = #myFactory
Which you can setup via spring boot #Bean annotation style.
I was trying integrate Apache camel with Kafka and wrote a sample program to read a file and write to Kafka Topic. But I am getting below error while doing so. I could be able to do it the reverse way read from Kafka topic and write to a file.
Stacktrace
org.apache.kafka.common.errors.SerializationException: Can't convert value of class org.apache.camel.component.file.GenericFile to class org.apache.kafka.common.serialization.StringSerializer specified in value.serializer
[#0 - file://C:%5Cshare%5Cinput] KafkaProducer WARN No message key or partition key set
[#0 - file://C:%5Cshare%5Cinput] GenericFileOnCompletion WARN Rollback file strategy: org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy#7127845b for file: GenericFile[C:\share\input\file.txt]
[#0 - file://C:%5Cshare%5Cinput] DefaultErrorHandler ERROR Failed delivery for (MessageId: ID-L8-CWBL462-49953-1480494317350-0-21 on ExchangeId: ID-L8-CWBL462-49953-1480494317350-0-22). Exhausted after delivery attempt: 1 caught: org.apache.kafka.common.errors.SerializationException: Can't convert value of class org.apache.camel.component.file.GenericFile to class org.apache.kafka.common.serialization.StringSerializer specified in value.serializer
Code
#ContextName("myCdiCamelContext")
public class MyRoutes extends RouteBuilder {
#Inject
#Uri("file:C:\\share\\input?fileName=file.txt&noop=true")
private Endpoint inputEndpoint;
#Inject
#Uri("kafka:localhost:9092?topic=test&groupId=testing&autoOffsetReset=earliest&consumersCount=1")
private Endpoint resultEndpoint;
#Override
public void configure() throws Exception {
from(inputEndpoint)
.to(resultEndpoint);
}
}
After adding a new processor it worked for me
public void configure() throws Exception {
from(inputEndpoint).process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
exchange.getIn().setBody(exchange.getIn().getBody(),String.class);
exchange.getIn().setHeader(KafkaConstants.PARTITION_KEY, 0);
exchange.getIn().setHeader(KafkaConstants.KEY, "1");
}
})
.to(resultEndpoint);
}
My log lines throw nullpointer exception when I run unit tests. I get no errors when I run it on local server or upload to appengine. Have I forgotten to include a test library somewhere?
java.lang.NullPointerException
at javax.servlet.GenericServlet.getServletContext(GenericServlet.java:160)
at javax.servlet.GenericServlet.log(GenericServlet.java:254)
at se.stuff.servlet.MyServlet.doGet(MyServlet.java:14)
at se.stuff.MyServletTest.test(MyServletTest.java:14)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
My servlet:
public class MyServlet extends HttpServlet {
#Override
public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
// Do stuff...
log("log stuff");
}
}
My test:
public class MyServletTest {
#Test
public void test() throws IOException {
MyServlet s = new MyServlet();
s.doGet(null, null);
}
}
The nulls in the test call s.doGet(null, null) cause the NullPointerException. The servlet is probably fine, but the test does not make sense. I suggest scrapping that test and first adding some content into doGet, then use QUnit to test your servlet from the outside.