Trying to run Flink(v 1.7.0) job on Amazon EMR Flink(5.21.0).
I get the exception
java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
The exception looks like SCALA version issue. I found flink libraries there is SCALA 2.11 and my job build using 2.12. Flink 1.7.0 supports SCALA 2.12.
The question is, how do I get Amazon EMR supporting SCALA 2.12 instead of 2.11? or Is there any better solution other my code going back 2.11.
EMR is using Flink 1.6.1, I believe. So that's the version you'd want to build your Flink workflow against.
There's a description here in the docs of how to manually install whatever version of Flink you want to use with EMR.
Related
I met a problem when I use NebulaGraph Database like below：
I want to write offline spark data into NebulaGraph Database by spark-nebula-connector.
But I encountered two problems:
First, the NebulaGraph Database version I use only support spark v2.4 and scala v2.11. For this one, I solve it by downgrading the spark and scala version.
Second, spark connector writes data via client, but clients has strong dependence on guava-14：nebula-java/pom.xml at v3.3.0 · vesoft-inc/nebula-java · GitHub
And my spark also has strong dependence on guava，guava-27.0-jre
If I use guava-27.0, it will give java.lang.NoSuchMethodError (com.google.common.net.HostAndPort.getHostText()
If I use guava-14.0, EROOR will be give when the spark reads hive, like Exception in thread “main” java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument
How should I solve this?
Maybe you can refer this solution. Guava-14.0 and Guava-27.0 of the HostAndPort have different methods to acqure hosts. You can change the Guava version in connector or exchange, modify the HostAndPort.getHostTex, and then package it locally.
I am looking into the issue FLINK-28060. And I think that it is resolved and it will be included as the resolved one in Apache Flink 1.15.1. When I look at the release note of the Apache Flink at release note, the issue still remains unresolved for 1.15.1. But the jira-issue is closed and the resolution level is fixed.
At the version 1.16 the issue will be included but I am wondering that why this issue is not included in the 1.15.1 or 1.15.2 but in the 1.16 ? Does this issue have special reason to be updated later?
No, that issue is not resolved in Flink 1.15.1. The necessary fix to Kafka wasn't made until well after 1.15.1 was released.
FLINK-28060 was caused a bug that was introduced into Kafka in the version of Kafka bundled with Flink 1.15.0. The necessary fix to the Kafka client was recently made available by the Kafka community, and the Flink community decided there was enough time to thoroughly test this new Kafka release before 1.16.0 will be completed, so it will be included there.
If you are using Flink 1.15.x and are troubled by this bug, you can work around it by excluding from your JAR the buggy version of the kafka client bundled with Flink, and build against the newer version instead. (This should be possible unless you are running a pure SQL job, rather than submitting a JAR file.)
I am looking for integrating Apache Atlas with Apache Flink to capture Job lineage. I found some references around it from Cloudera (CDP) and they are using Atlas-flink hook , But not able to find any documentation or implementation without CDP and Atlas-Flink connection JAR as well.
Let me know if anyone has done integration between them?
Is there a place where we can quickly refer and see version compatibility between Apache beam and the runtime environments (Flink, Spark etc). I am learning it the hard way :(. Not sure how many of my issues are contributed by version mismacth.
Currently I am using Beam 2.4.0 and looking at he flink runner pom (https://search.maven.org/#artifactdetails%7Corg.apache.beam%7Cbeam-runners-flink_2.11%7C2.4.0%7Cjar) , it uses flink 1.4.0 jars. So I am assuming Apache beam 2.4.0 works with Apache Flink 1.4.0. Is this correct interpretation?
Unfortunately there is no official docker image for Flink 1.4.0 (https://hub.docker.com/_/flink/)
Flink 1.4.0 is compatible with Beam 2.4.0.
One easy way to find out compatibility is to check the parent pom.xml for Beam release versions and see which Flink version it was tested with. Note that this doesn't mean that it is incompatible with other Flink verions. It may or may not be, you need to find out or ask someone in Beam community.
Hi I would like to know which camel version is used in Fuse esb (jboss-fuse-6.0.0.redhat-024),
according to the migration guide it's 2.10.0 .
how can I update it to use 2.10.5 ?
JBoss fuse is using customer version of Camel which is based on the Camel 2.10.x branch.
As there are lots other projects which may relates to this Camel version, I don't suggest you to change the version yourself.