I am trying to run a flink steaming program that uses kafka connector(latest universal connector).
The jobs runs without any problem on IntelliJ but when I am submitting the code build into jar using sbt package is giving me below error.
java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase
I also used the jar built using traditional IntellIJ option but still i get the above error.
Most probably the issue is the fact that You are not including the dependencies in Your JAR file. Connector dependencies are not included in the Flink binary.
Generally, the preferred way of tackling this issue is to use the proper plugin for Your build tool like shade-plugin for Maven or assembly for sbt to create so-called fat-jar i.e. the JAR with the dependencies included.
Related
Hello Flink Community,
following the documentation to troubleshoot unloading of dynamically loaded classes in Flink I added the database driver library to the opt/flink/lib folder on both the Flink JobManager Container and TaskManager Containers running on K8s (Flink Session Cluster, version: 1.11).
I marked the library as provided in my build.sbt file.
The rest of the user code is p[art of the fat jar build by sbt assembly.
Now when I submit a job to the flink cluster using the Flink API (upload and run endpoints) it won't accept the job due to the following error:
java.lang.ClassNotFoundException: com.vertica.jdbc.Driver
Why is the jar not picked up by the Flink classloader?
I even added the class pattern to the config option without any difference:
classloader.parent-first-patterns-additional: com.vertica.jdbc.;
Link: https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/debugging/debugging_classloading.html#unloading-of-dynamically-loaded-classes-in-user-code
Any recommendation would be highly appreciated.
Cheers
Please confirm your jdbc maven dependency is not provided.
when the library is provided, the library is active when compile and test.
I would like to read and write some data with Apache Flink 1.11.2 from S3. The documentation recommends to use the presto plugin for checkpoints and the hadoop plugin for pipeline data.
After reading this section you have to copy the plugins from /opt to /plugin. I can find the flink-s3-fs-presto-1.11.2.jar under /opt but there is no flink-s3-fs-hadoop-1.11.2.jar. Where can i find the s3-hadoop plugin for setting up my production environment?
And how can i use these plugins in the IDE? Simply adding these to pom.xml als provided dependencies? And then how can i pass the crentials in IDE?
That is weird I can see that they are both present in the official binaries in opt in 1.11.1. However if You can't find them, You can simply try to get the jars from Maven here and copy them to the required place. Another thing that may work is adding the dependency into the project with compile scope.
Running the job locally is described here. There are various ways of configuring the credentials when running the job in IDE, one might be adding core-site.xml to resources folder with proper configruation.
EDIT:
As for the local execution it was explained here a little bit.
I'm beginner in Apache Flink and can't find any information about extending Flink core. I want start with simple: to build in one test-class. should I save that class as jar-file somewhere in flink?
I recently started developing small programs with Apache Flink, so I had the same problem. I advise you to use maven because it will allow to link all the dependencies you need and easily build your own classes.
You need to first install maven and then you can create a Flink Maven project with the following command:
mvn archetype:generate -DarchetypeGroupId=org.apache.flink -DarchetypeArtifactId=flink-quickstart-java -DarchetypeVersion=1.0.2
after that can add your classes to the src/main/java/[GroupId]../ and compile them with:
mvn package
Then, to run you newly created program you can execute the following command:
mvn exec:java -Dexec.mainClass="Package.YourClass" -Dexec.args="YOUR PROGRAM ARGUMENTS"
There are also tutorials all over the internet of how to run maven with an IDE if you want to.
I am relatively new to Selenium WebDriver and a self learner. I have created a webDriver project which consists of different packages and have made use of Page Factory concepts extensively. I use TestNg framework to run the suite to generate my report. So my test cases are also following the TestNG framework concepts.
Now my team wants to use my script to be run at every build to test the sanity. The build team just wants to run my whole script by just a click. It can be shell script command or .exe or jar. Build team uses linux m/c and they dont have Ecplise , TestNg installed in their machine. Their intention is whenever build is given they want to run my script by just a click or a command in Command Prompt.( it has to be that simple for them) and a report should be generated in some location in hdfs
My script runs on FF version 32 and Selenium webDriver 2.44.0
Would really appreciate if someone could give me a solution that actually works for my requirement.
I found a similar query from someone but i am not sure if the answer still suits.
How to make Java executable Jar file of WebDriver project
Could somebody please give me a solution. Or the solution mentioned in the above link stil is the best?
Regards
There are few ways to do it:
Use CI tool (Jenkins): You can set up your project in Jenkins and it will allow you to run the project in single click. Jenkins also provides you an option to run the tests periodically so you can configure it in such a way that it will always get executed after certain time period.Jenkins is also capable to trigger the execution if there is any changes in source code of tests. I'd suggest to use this tool. https://jenkins-ci.org/
Use .bat or .sh file: I am not sure if you are using any build tool like ANT or Maven, if you are using any one of them then you can write a .bat or .sh file to run the tests. ANT
If you are not using any build tool then start using, it will help to run the tests in simplest way.For me to run entire suite, I just type below command in terminal from root directory of project
ant run -Dsuite=all
If you are more concerned about system configuration for e.g. the system which will be used by build team does not have a specific version of browser or specific required library. Then I'd suggest to keep all essentials like browser installation files, libraries in your project directory. This is not a good practice but yes it will help others to run the tests smoothly.
I would suggest just exporting a jar file for your project. All the libraries will be packaged together in the jar files(including TestNG) and you can simply double click on the jar file to start your tests.
External resources (if any), will have to be available for the jar file though. So, you can provide the jar file and the external resources together. The external resources might include your test data (if any) or Portable Firefox(if you are using the portable version).
Steps:
Right click on Project -> Export -> Runnable jar file
Give a name and file path for the jar file
Select option - Extract required libraries into generated JAR
And Finish
Troubleshooting:
Check the java version for the machines that you will be running your jar file on. Programs compiled with java 7 will mostly not run if the machine has java 6. Either compile with java 6 or update the jre on the target machines.
If the jar file does not launch, try using Jarfix.
I'm using appengine-sdk 1.9.3.
In devserver, works in Eclipse and Ant normally.
When I deploy (update) to appengine (production environment), I get this error:
event.getResults(): [<pre>Error for /p7/formPanelServlet
java.lang.NoClassDefFoundError: Could not initialize class
com.google.appengine.tools.cloudstorage.oauth.OauthRawGcsServiceFactory
at com.google.appengine.tools.cloudstorage.GcsServiceFactory.createRawGcsService(GcsServiceFactory.java:42)
at com.google.appengine.tools.cloudstorage.GcsServiceFactory.createGcsService(GcsServiceFactory.java:34)
at com.bitvisio.p7.server.FormPanelServlet.<init>(FormPanelServlet.java:27)
At FormPanelServlet.java:27, the code is:
private final GcsService gcsService = GcsServiceFactory
.createGcsService(new RetryParams.Builder()
.initialRetryDelayMillis(10)
.retryMaxAttempts(10)
.totalRetryPeriodMillis(15000)
.build());
I put the lib appengine-gcs-client-0.3.9.jar in war/WEB-INF/lib. I think there is a problem with this lib.
Thanks for help.
Always use tools like Maven or Ivy to resolve dependencies for you. Copying JARs to war/WEB-INF/lib/ directory and editing .classpath file manually will be painful and may not help you always. If you use Eclipse & Google App Engine plugin, use Add Google APIs... as mentioned here - Google Plugin for Eclipse. In my case, adding Cloud Storage API via Google Plugin for Eclipse helped resolve this NoClassDefFoundError.
I had the same problem. I use Ivy to resolve dependencies and always get the latest.integration (with Maven use RELEASE) for revisions.
However I usually ignore transitive libraries. It looks like Google is expanding the API family - at least splitting out discrete functionality.
There are now quite a few transitive dependencies and it seems they released a new version of the GCS client around the same time as 1.9.3.
Getting all dependencies and packaging them in my deployment fixed my issue. I did not have the issue in development which made it more confusing.
You are facing this issue because you are not adding the some of the jar like
google-api-services-storage-v1-rev78-1.22.0.jar download link
joda-time-2.94.jar download link
guava-19.0.jar link to download
you can use the different version of jar according to your appengine-gcs-client jar file.
Note : Add all these jar and build path with the project and problem will get solve.