I want to run my first flink code, so I created a virtual environement and I run it with: python tab.py
I find :
What's wrong with my Pyflink setup that Python UDFs throw py4j exceptions?
but it doesn't work.
Related
I have an Apache Beam Pipeline that I am trying to deploy on a Flink Docker Cluster deployed locally.
The pipeline fails with
The RemoteEnvironment cannot be instantiated when running in a pre-defined context (such as Command Line Client, Scala Shell, or TestEnvironment)
org.apache.flink.api.java.RemoteEnvironmentConfigUtils.validate(RemoteEnvironmentConfigUtils.java:52)
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.validateAndGetEffectiveConfiguration(RemoteStreamEnvironment.java:178)
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.<init>(RemoteStreamEnvironment.java:158)
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.<init>(RemoteStreamEnvironment.java:144)
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.<init>(RemoteStreamEnvironment.java:113)
org.apache.beam.runners.flink.FlinkExecutionEnvironments$BeamFlinkRemoteStreamEnvironment.<init>(FlinkExecutionEnvironments.java:319)
org.apache.beam.runners.flink.FlinkExecutionEnvironments.createStreamExecutionEnvironment(FlinkExecutionEnvironments.java:177)
org.apache.beam.runners.flink.FlinkExecutionEnvironments.createStreamExecutionEnvironment(FlinkExecutionEnvironments.java:139)
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:98)
org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:108)
ApacheBeamPocJava.main(ApacheBeamPocJava.java:262)
This is how I am setting up the pipeline
Options options = PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class); options.setRunner(FlinkRunner.class);
options.setFlinkMaster(“localhost:6123”);
options.setFilesToStage(Arrays.asList("path to the beam jar"));
FlinkRunner flinkRunner = FlinkRunner.fromOptions(options);
Pipeline p= Pipeline.create(options);
And after defining the steps of the pipeline. I run it like this
flinkRunner.run(p);
This is how I submit the job
flink run -c ClassName PATH_TO_JAR
Can someone advise what is going wrong here?
Also if someone has a Beam <-> Flink examples handy for Java. I would definitely appreciate that too.
It seems that you have defined the running environment inside the pipeline itself. Have you tried launching your pipeline like described in the Flink runner documentation? (Remove the parts of your code where you are defining a runner, or configuring it.)
As Beam is a framework that decouples your code from the runner that is executing it, it's not necessary to have the Flink runner configuration in your pipeline code itself. If you can execute your pipeline locally with the direct runner, it will also work on the Flink runner (or any other one that is supported) when being compiled with the right profile.
bin/flink run -c org.apache.beam.examples.WordCount /path/to/your.jar –runner=FlinkRunner –other-parameters-for-your-pipeline-or-the-runner
Please be aware that there is currently a bug in Beam 2.25.0 for the Flink runner, so try it with version 2.24.0, or a later version when it's released.
I have two hosts, one with zeppelin and another where I want to run jdbc interpreter, problem is that in 0.7 I could run it like this:
<br> /opt/zeppelin/bin/interpreter.sh -d /opt/zeppelin/interpreter/jdbc -p xxxx<br>
but from 0.8 they add new parameters CALLBACK_HOST and INTP_PORT and I script throws errors:
Exception in thread "main"
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:266).
Please advise how to start this interpreter with zeppelin 0.8.0 and what exactly this parameters mean because there is almost nothing on github.
Although you can find callback_host in the configuration such functionality is not available now.
The JDBC interpreter could run only on the same host as Zeppelin server.
flink run /home/admin/Documents/flink_cep/Flink-master/dist/Kinesis.jar
When I am trying to run Jar file in command line, getting error but my code is running fine in Netbeans IDE:
A NoSuchMethodError indicates a version conflict.
You should verify that you compiled your Flink job with the same Flink version as your cluster is running.
I am working on Kafka Flink integration actually I am done with that integration , I have written a simple word count program in Java using Flink API, when I ran it by java -jar myjarname it worked fine but when I tried to ran it with ./bin/flink run myjarname command it was giving me following error,
NoSuchMethodError:org.apache.flink.streaming.api.operators.isCheckpointingEnabled
The respected jar is there but still it is giving me above error.
Greetings,
I am trying to start a scala/liftweb project for deployment on Google App Engine. To do this, i need to package it up as a .war using maven.
However, whenever I run the 'mvn' command, I am met with:
Error opening zip file or JAR manifest missing : /Applications/JRebel/jrebel.jar
Error occurred during initialization of VM
agent library failed to init: instrument
Is there something wrong with my maven or do I need Jrebel? I see jrebel is not free which is why I am so surprised.
thanks!
No, JRebel is definitely not required to run Maven.
As Matt mentioned, JRebel is not required to run Maven. However, ZeroTurnaround does offer a free version that works with Scala. You can get it here:
http://sales.zeroturnaround.com/
As for your error - it indicates you are trying to start the JVM as though you are using JRebel. What is the full Maven command you are running? What is in your MAVEN_OPTS environment variable? If either of them contain something like -noverify -javaagent:/Applications/JRebel/jrebel.jar, then that's your problem.
One of the reason of the problem is a blank in the path of jrebel.jar
Make sure that there is no blank in the path like in "Program Files"