While creating a new session using apache Livy 0.7.0 I am getting below error.
I am also using zeppelin notebook(livy interpreter) to create the session.
Using Scala version 2.12.10, Java HotSpot(TM) 64-Bit Server VM, 11.0.11
Spark 3.0.2
zeppelin 0.9.0
Any idea why I am getting the error?
I have already checked that we have livy-repl_2.11-0.7.1-incubating.jar in the classpath and the JAR already have the class it is not able to find.
$ jar tf livy-repl_2.11-0.7.1-incubating.jar | grep "ReplDriver"
org/apache/livy/repl/ReplDriver$$anonfun$addFile$1.class
org/apache/livy/repl/ReplDriver$$anonfun$1.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$1.class
org/apache/livy/repl/ReplDriver$$anonfun$addJarOrPyFile$1.class
org/apache/livy/repl/ReplDriver.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$2.class
org/apache/livy/repl/ReplDriver$$anonfun$2.class
Error Log
21/06/17 13:13:28 INFO LineBufferedStream: Exception in thread "main" java.lang.ClassNotFoundException: org.apache.livy.repl.ReplDriver
21/06/17 13:13:28 INFO LineBufferedStream: at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
21/06/17 13:13:28 INFO LineBufferedStream: at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
21/06/17 13:13:28 INFO LineBufferedStream: at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
21/06/17 13:13:28 WARN ContextLauncher: Child process exited with code 1.
21/06/17 13:13:28 ERROR RSCClient: Failed to connect to context.
java.io.IOException: Child process exited with code 1.
at org.apache.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:397)
at org.apache.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:448)
at java.base/java.lang.Thread.run(Thread.java:834)
21/06/17 13:13:28 ERROR SparkProcApp: spark-submit exited with code 1
21/06/17 13:13:28 INFO RSCClient: Failing pending job f05c2bac-6e0a-4732-a419-0bde4dcfd06b due to shutdown.
21/06/17 13:13:28 INFO InteractiveSession: Stopping InteractiveSession 0...
21/06/17 13:13:28 INFO InteractiveSession: Failed to ping RSC driver for session 0. Killing application.
21/06/17 13:13:28 INFO InteractiveSession: Stopped InteractiveSession 0.
21/06/17 13:13:28 WARN InteractiveSession: Fail to get rsc uri
java.util.concurrent.ExecutionException: java.io.IOException: Child process exited with code 1.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
at org.apache.livy.server.interactive.InteractiveSession$$anonfun$18.apply(InteractiveSession.scala:411)
Spark 3.0.x came with version of scala 2.12. You will need to be build with livy with Spark 3.0.x using scal 2.12 to solve this issue. Then you need to adjust your livy.conf Here is the article on how to rebuild your livy using maven (How to rebuild apache Livy with scala 2.12)
Good luck. I ran into the same issue and was able to solve with above steps.
Related
I've just upgraded my flink from version 1.9.1 to 1.11.2 (using docker)
I have already many flink jobs running in version 1.9.1
When I try to upgrade to 1.11.1 and re run my job, it shows error.
2020-11-12 06:49:17,731 WARN org.apache.zookeeper.ClientCnxn []
- SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-1135609831848314731.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2020-11-12 06:49:17,739 INFO org.apache.zookeeper.ClientCnxn [] - Opening socket connection to server xxxxxx:2181
2020-11-12 06:49:17,741 ERROR org.apache.curator.ConnectionState [] - Authentication failed
And this is the error after deploying my flink job:
Caused by: java.lang.RuntimeException: API paths not defined
and also:
java.lang.NoSuchMethodError: org.apache.flink.api.common.state.OperatorStateStore.getSerializableListState(Ljava/lang/String;)Lorg/apache/flink/api/common/state/ListState;
Do I need to change every pom for my flink jobs?
Is there any work around without changing my source code?
Thanks
Yes, you do have to rebuild your Flink jobs whenever you update the Flink version being used to run them. The libraries you use should be from the same exact version used by the Job Manager and Task Managers.
If you are trying to automate deployments for a CI/CD pipeline, you could inject the version number into the pom.xml using an environment variable -- but doing things like that can make it hard to debug when things go wrong.
Why I am seeing the error:
Async exception during distributed update: java.io.IOException: Broken pipe when trying to post a document to solr
Solr version: 8.4.1
Zookeeper: 3.4.14
OpenJDK 11;
2 solr node + 1 zookeeper (hosted zookeeper in one of the solr node)
Using basic authentication on solr and also with TLS1.2.
I am not seeing any error when trying to post to solr node which is a leader and it's working consistently; it is the issue only when I post to the non-leader node which is throwing the below error.
java.io.IOException: java.io.IOException: Broken pipe
at org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:172)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:103)
at org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
at org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:339)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:236)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/0000000000000000.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Suppressed: java.io.IOException: java.io.IOException: Broken pipe
at org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
at org.apache.solr.common.util.JavaBinCodec.close(JavaBinCodec.java:1269)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:104)
... 10 more
Caused by: java.io.IOException: Broken pipe
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:927)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:393)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380)
at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:247)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
at org.eclipse.jetty.http2.HTTP2Session.frame(HTTP2Session.java:755)
at org.eclipse.jetty.http2.HTTP2Session.frames(HTTP2Session.java:734)
at org.eclipse.jetty.http2.client.HTTP2ClientConnectionFactory$HTTP2ClientConnection.onOpen(HTTP2ClientConnectionFactory.java:130)
at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:441)
at org.eclipse.jetty.io.NegotiatingClientConnection.replaceConnection(NegotiatingClientConnection.java:115)
at org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:85)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
When we had this issue it was because we were running the wrong JDK. We had installed OpenJDK 16 and that was still being used by Solr and Zookeeper. There was some incompatibility that was causing the broken pipe error when the leader was communicating with followers.
To fix this we had to install Open JDK Java 1.8 and force Solr and Zookeeper to use that.
Client disconnected the connection because of too much waiting time
on the server. Higher timeouts may cause performance issues.
Another reason might be the OS cache. Check the disk space and try to
increase the disk space.
UPDATE: I've tried starting TomCat from commandline. During the startup messages I get this:
15-Mar-2019 09:05:08.603 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [C:\Program Files\ASF\Tomcat9\webapps\thredds.war]
15-Mar-2019 09:05:15.900 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
15-Mar-2019 09:05:18.286 INFO [main] org.hibernate.validator.internal.util.Version.<clinit> HV000001: Hibernate Validator 4.3.2.Final
15-Mar-2019 09:05:19.382 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
15-Mar-2019 09:05:19.383 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/thredds] startup failed due to previous errors
15-Mar-2019 09:05:19.460 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [thredds] appears to have started a thread named [Log4j2-TF-12-Scheduled-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
15-Mar-2019 09:05:19.469 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [C:\Program Files\ASF\Tomcat9\webapps\thredds.war] has finished in [10,866] ms
I'm starting to believe that Gerhard is right that it is not a batch issue?
I'm trying to deploy the THREDDS server (version 4.6.13) for Windows using Tomcat 9. However, when I try to start the server I get:
FAIL - Application at context path [/thredds] could not be started
I've tried creating setenv.bat from a working setenv.sh on a Linux-machine, but my batch-scripting is more than rusty, but I certainly believe it's down to wrong syntax.
My script file is as follows:
#ECHO OFF
set "CATALINA_HOME=%ProgramFiles%/ASF/Tomcat9"
set "CATALINA_BASE=%ProgramFiles%/ASF/Tomcat9"
set "JAVA_HOME=%ProgramFiles%/AdoptOpenJDK/jdk8u202-b08-jre"
:: TDS specific ENVARS
::
:: Define where the TDS content directory will live
::
set "CONTENT_ROOT=-Dtds.content.root.path=%ProgramFiles%/ASF/Tomcat9/content"
:: set java prefs related variables (used by the wms service, for example)
set "JAVA_PREFS_ROOTS=-Djava.util.prefs.systemRoot=%CATALINA_HOME%/content/thredds/javaUtilPrefs -Djava.util.prefs.userRoot=%CATALINA_HOME%/content/thredds/javaUtilPrefs"
:: Some commonl used JAVA_OPTS settings:
set NORMAL="-d64 -Xmx4096m -Xms512m -server -ea"
set HEAP_DUMP="-XX:+HeapDumpOnOutOfMemoryError"
set HEADLESS="-Djava.awt.headless=true"
Where did I mess up?
I'm trying to setup a cluster (Apache Flink 1.6.1) with Kubernetes and get following error when I run a job on it:
2018-10-09 14:29:43.212 [main] INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-10-09 14:29:43.214 [main] INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.flink.runtime.entrypoint.ClusterConfiguration.<init>(Ljava/lang/String;Ljava/util/Properties;[Ljava/lang/String;)V
at org.apache.flink.runtime.entrypoint.EntrypointClusterConfiguration.<init>(EntrypointClusterConfiguration.java:37)
at org.apache.flink.container.entrypoint.StandaloneJobClusterConfiguration.<init>(StandaloneJobClusterConfiguration.java:41)
at org.apache.flink.container.entrypoint.StandaloneJobClusterConfigurationParserFactory.createResult(StandaloneJobClusterConfigurationParserFactory.java:78)
at org.apache.flink.container.entrypoint.StandaloneJobClusterConfigurationParserFactory.createResult(StandaloneJobClusterConfigurationParserFactory.java:42)
at org.apache.flink.runtime.entrypoint.parser.CommandLineParser.parse(CommandLineParser.java:55)
at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:153)
My job takes a configuration file (file.properties) as a parameter. This works fine in standalone mode but apparently the Kubernetes cluster cannot parse it
job-cluster-job.yaml:
args: ["job-cluster", "--job-classname", "com.test.Abcd", "-Djobmanager.rpc.address=flink-job-cluster",
"-Dparallelism.default=1", "-Dblob.server.port=6124", "-Dquery.server.ports=6125", "file.properties"]
How to fix this?
Update: The job was built for Apache 1.4.2 and this might be the issue, looking into it.
The job was built for 1.4.2, the class with the error (EntrypointClusterConfiguration.java) was added in 1.6.1 (https://github.com/apache/flink/commit/ab9bd87e521d19db7c7d783268a3532d2e876a5d#diff-d1169e00afa40576ea8e4f3c472cf858) it seems, so this caused the issue.
We updated the job's dependencies to point to new 1.6.1 release and the arguments are parsed correctly.
I have just started learning Apache Flink and found the guide link to start the development in EClipse IDE.
I followed the this to start off but getting the below error
00:20:26,993 INFO org.apache.flink.api.java.ExecutionEnvironment - The job has 0 registered types and 0 default Kryo serializers
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Traversable
at java.lang.ClassLoader.defineClass1(Native Method)
Here I have place the Error log log File ... Please let me know if your require more details. Thanks, Nyamath
java.util.zip.ZipException: invalid LOC header (bad signature)
Your Scala jar file provided by Maven seems to be corrupted. Please update your Maven dependencies by executing this from your project folder on the command line:
mvn -U clean install
In Eclipse, right click on your project and click on Update - - > Maven dependencies.
If that does not work you'll need to delete the corrupted Jar file in the .m2/repositories/ folder.