I am working on Kafka Flink integration actually I am done with that integration , I have written a simple word count program in Java using Flink API, when I ran it by java -jar myjarname it worked fine but when I tried to ran it with ./bin/flink run myjarname command it was giving me following error,
NoSuchMethodError:org.apache.flink.streaming.api.operators.isCheckpointingEnabled
The respected jar is there but still it is giving me above error.
Related
I want to run my first flink code, so I created a virtual environement and I run it with: python tab.py
I find :
What's wrong with my Pyflink setup that Python UDFs throw py4j exceptions?
but it doesn't work.
I have an Apache Beam Pipeline that I am trying to deploy on a Flink Docker Cluster deployed locally.
The pipeline fails with
The RemoteEnvironment cannot be instantiated when running in a pre-defined context (such as Command Line Client, Scala Shell, or TestEnvironment)
org.apache.flink.api.java.RemoteEnvironmentConfigUtils.validate(RemoteEnvironmentConfigUtils.java:52)
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.validateAndGetEffectiveConfiguration(RemoteStreamEnvironment.java:178)
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.<init>(RemoteStreamEnvironment.java:158)
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.<init>(RemoteStreamEnvironment.java:144)
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.<init>(RemoteStreamEnvironment.java:113)
org.apache.beam.runners.flink.FlinkExecutionEnvironments$BeamFlinkRemoteStreamEnvironment.<init>(FlinkExecutionEnvironments.java:319)
org.apache.beam.runners.flink.FlinkExecutionEnvironments.createStreamExecutionEnvironment(FlinkExecutionEnvironments.java:177)
org.apache.beam.runners.flink.FlinkExecutionEnvironments.createStreamExecutionEnvironment(FlinkExecutionEnvironments.java:139)
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:98)
org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:108)
ApacheBeamPocJava.main(ApacheBeamPocJava.java:262)
This is how I am setting up the pipeline
Options options = PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class); options.setRunner(FlinkRunner.class);
options.setFlinkMaster(“localhost:6123”);
options.setFilesToStage(Arrays.asList("path to the beam jar"));
FlinkRunner flinkRunner = FlinkRunner.fromOptions(options);
Pipeline p= Pipeline.create(options);
And after defining the steps of the pipeline. I run it like this
flinkRunner.run(p);
This is how I submit the job
flink run -c ClassName PATH_TO_JAR
Can someone advise what is going wrong here?
Also if someone has a Beam <-> Flink examples handy for Java. I would definitely appreciate that too.
It seems that you have defined the running environment inside the pipeline itself. Have you tried launching your pipeline like described in the Flink runner documentation? (Remove the parts of your code where you are defining a runner, or configuring it.)
As Beam is a framework that decouples your code from the runner that is executing it, it's not necessary to have the Flink runner configuration in your pipeline code itself. If you can execute your pipeline locally with the direct runner, it will also work on the Flink runner (or any other one that is supported) when being compiled with the right profile.
bin/flink run -c org.apache.beam.examples.WordCount /path/to/your.jar –runner=FlinkRunner –other-parameters-for-your-pipeline-or-the-runner
Please be aware that there is currently a bug in Beam 2.25.0 for the Flink runner, so try it with version 2.24.0, or a later version when it's released.
I was trying to execute the apache-beam word count having Kafka as input and output. But on submitting the jar to the flink cluster, this error came -
The RemoteEnvironment cannot be used when submitting a program through a client, or running in a TestEnvironment context.
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.<init>(RemoteStreamEnvironment.java:174)
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.<init>(RemoteStreamEnvironment.java:142)
org.apache.beam.runners.flink.FlinkExecutionEnvironments$BeamFlinkRemoteStreamEnvironment.<init>(FlinkExecutionEnvironments.java:331)
org.apache.beam.runners.flink.FlinkExecutionEnvironments.createStreamExecutionEnvironment(FlinkExecutionEnvironments.java:180)
org.apache.beam.runners.flink.FlinkExecutionEnvironments.createStreamExecutionEnvironment(FlinkExecutionEnvironments.java:141)
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:98)
org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:110)
org.apache.beam.sdk.Pipeline.run(Pipeline.java:315)
org.apache.beam.sdk.Pipeline.run(Pipeline.java:301)
org.apache.beam.examples.WordCount.runWordCount(WordCount.java:295)
org.apache.beam.examples.WordCount.main(WordCount.java:406)
the command I used to submit jar -
./flink run -m localhost:8081 --class org.apache.beam.examples.WordCount /users/word-count-beam/target/word-count-beam-bundled-0.1.jar --runner=FlinkRunner --flinkMaster=localhost --parallelism=2 --checkpointingInterval=10000 --checkpointTimeoutMillis=5000 --minPauseBetweenCheckpoints=500
I guess you use StreamExecutionEnvironment.createRemoteEnvironment, thats way you cannot submit your jar with 'flink run', you have to run it as regular java jar (java -jar ...).
If you want submit it in your cluster you should use StreamExecutionEnvironment.getExecutionEnvironment, it will return the executionEnvironment of the cluster been submitted.
flink run /home/admin/Documents/flink_cep/Flink-master/dist/Kinesis.jar
When I am trying to run Jar file in command line, getting error but my code is running fine in Netbeans IDE:
A NoSuchMethodError indicates a version conflict.
You should verify that you compiled your Flink job with the same Flink version as your cluster is running.
Recently I installed Zeppelin from git using mvn clean package -Pspark-1.5 -Dspark.version=1.5.1 -Phadoop-2.4 -Pyarn -Ppyspark -DskipTests and I can't run the tutorial because of this error:
java.net.ConnectException
Any idea why this is happening? I haven't modified any of the conf files because I am interested in running it using the embedded Spark binaries.
I already check most of the threads here and none of them has worked.
Thanks
EDIT: I am using a Mac
Apache Zeppelin uses multi-process architecture, where ZeppelinServer process communicates with InterpreterGroup process though Apache Thrift API
This error usually indicates that ZeppelinServer process can not reach Interpreter process, running on same machine due to abnormal executor termination of the latter.
More details can be found in Interpreter process logs ./logs/zeppelin-interpreter-<interpreter name>-<username>-<hostname>.log and ZeppelinServer process logs under ./logs/zeppelin-<username>-<hostname>.log