Disable logging for server-sent-events errors in RESTEasy - resteasy

I'm currently working with server-sent-events using RESTEasy in Wildfly. So far everything is working, except that sometimes the SSE implementation doesn't somehow recognize that the client(s) listening to events is/are already closed (also the close() method of the SseEventSourceon the client-side was called). In aspect of the program logic this isn't a problem at all.
But unfortunately org.jboss.resteasy.plugins.providers.sse.SseEventOutputImpl class which is used to send the events, does - in addition to reporting the exception back to org.jboss.resteasy.plugins.providers.sse.SseBroadcasterImpl - not only report the exception, but also logs it using the failedToWriteSseEvent(String, Throwable) method of the org.jboss.resteasy.resteasy_jaxrs.i18n.LogMessages (later class is based on the JBoss logging). So I get every now and then an unnecessary log messages on level ERROR telling me that the connection was closed by the client. And it get that entry in the log in addition to the onClose event I get from the SseBroadcaster.
Configuring the JBoss logging seems impossible as the log name is org.jboss.resteasy.resteasy_jaxrs.i18n which is also used for logging other errors (Means just configuring the logger in the log4j.xml of the deployment won't work / also turn off other errors).
2021-06-22 12:59:27 [ERROR] [org.jboss.resteasy.resteasy_jaxrs.i18n:272] - RESTEASY002030: Failed to write event org.jboss.resteasy.plugins.providers.sse.OutboundSseEventImpl#fd79b33
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(Unknown Source)
at sun.nio.ch.IOUtil.write(Unknown Source)
at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:162)
at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:647)
at io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(ChunkedStreamSinkConduit.java:166)
at io.undertow.conduits.ChunkedStreamSinkConduit.write(ChunkedStreamSinkConduit.java:128)
at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150)
at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:240)
at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2103)
at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:574)
at io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:489)
at io.undertow.servlet.spec.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:476)
at io.undertow.servlet.spec.HttpServletResponseImpl.flushBuffer(HttpServletResponseImpl.java:468)
at javax.servlet.ServletResponseWrapper.flushBuffer(ServletResponseWrapper.java:221)
at org.jboss.resteasy.plugins.server.servlet.HttpServletResponseWrapper.flushBuffer(HttpServletResponseWrapper.java:124)
at org.jboss.resteasy.plugins.providers.sse.SseEventOutputImpl.writeEvent(SseEventOutputImpl.java:264)
at org.jboss.resteasy.plugins.providers.sse.SseEventOutputImpl.send(SseEventOutputImpl.java:199)
at org.jboss.resteasy.plugins.providers.sse.SseBroadcasterImpl.lambda$null$4(SseBroadcasterImpl.java:150)
at java.lang.Iterable.forEach(Unknown Source)
at org.jboss.resteasy.plugins.providers.sse.SseBroadcasterImpl.lambda$broadcast$5(SseBroadcasterImpl.java:146)
at java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source)
at java.util.concurrent.CompletableFuture$AsyncRun.exec(Unknown Source)
at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Is there a way to control that logging within RESTEasy and disabling the logging of failedToWriteSseEvent(String, Throwable)? Like introducing / inject an own implementation of LogMessages (but as far as I understand it the interface is used as a proxy, so...)?

You might be hitting RESTEASY-1986. You can filter these out with a log filter though. In CLI something like:
/subsystem=logging/logger=org.jboss.resteasy.resteasy_jaxrs.i18n:add(filter-spec=not(match(".*RESTEASY002030.*")), level=INFO)

Related

Error log shows error about Software caused connection abort

I see the following error in the production log. Please advise what could be causing this error. These email logs are so abstract really not able to determine the issue.
secondly, where is the device information is enclosed here.
[Network Thread] 45:4:44,291 - The operation couldn’t be completed. Software caused connection abort
[Network Thread] 45:4:44,297 - Exception: java.io.IOException - The operation couldn’t be completed. Software caused connection abort
java.io.IOException
at com_codename1_io_ConnectionRequest.performOperation:884
at com_codename1_io_NetworkManager_NetworkThread.run:325
at com_codename1_impl_CodenameOneThread.run:176
at java_lang_Thread.runImpl:153
This is a "feature" of iOS 12: https://forums.developer.apple.com/thread/106838
It seems you're using the connection when the app went to the background. Apple disallows that and historically would randomly kill the app when you did that without a background task.

How to solve Too many open files exception in flink when using elastic search connector to send stream data to elastic search index?

I am using flink elastic search connector to send data to elastic search index.
But I am unable to run the job and getting this exception. How to solve this issue?.Is there any setting or configurations I am missing? Could anyone guide me?
io.netty.channel.ChannelException: The operating system does not offer enough file handles to open the network connection. Please increase the number of of available file handles.
at org.apache.flink.runtime.io.network.netty.NettyClient.connect(NettyClient.java:207)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:81)
at org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:60)
at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:112)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:419)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:441)
at org.apache.flink.streaming.runtime.io.BarrierTracker.getNextNonBlocked(BarrierTracker.java:85)
at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:261)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:665)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.ChannelException: Failed to open a socket.
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:62)
at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:72)
at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:453)
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:306)
at io.netty.bootstrap.Bootstrap.doConnect(Bootstrap.java:133)
at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:115)
at org.apache.flink.runtime.io.network.netty.NettyClient.connect(NettyClient.java:198)
... 11 more
Caused by: java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:411)
at sun.nio.ch.Net.socket(Net.java:404)
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:60)
... 21 more
In order to increase the number of file handles in your system you can set them via ulimit -n for a process if you have root privileges. If you want to change the number of file handles system-wide, then change the value in /proc/sys/fs/file-max (on Linux).

Does a 'Broken pipe' exception cancel my job?

Currently I am running a Flink program on a remote cluster of 4 machines using 144 TaskSlots. After running for around 30 minutes I received the following error:
INFO
org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet - Info
server for jobmanager: Failed to write json updates for job
b2eaff8539c8c9b696826e69fb40ca14, because
org.eclipse.jetty.io.RuntimeIOException:
org.eclipse.jetty.io.EofException at
org.eclipse.jetty.io.UncheckedPrintWriter.setError(UncheckedPrintWriter.java:107)
at
org.eclipse.jetty.io.UncheckedPrintWriter.write(UncheckedPrintWriter.java:280)
at
org.eclipse.jetty.io.UncheckedPrintWriter.write(UncheckedPrintWriter.java:295)
at
org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet.writeJsonUpdatesForJob(JobManagerInfoServlet.java:588)
at
org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet.doGet(JobManagerInfoServlet.java:209)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) at
javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
at org.eclipse.jetty.server.Server.handle(Server.java:352) at
org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
at
org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
at
org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.eclipse.jetty.io.EofException at
org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:905)
at
org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:427)
at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:78) at
org.eclipse.jetty.server.HttpConnection$Output.flush(HttpConnection.java:1139)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:159) at
org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:86) at
java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:154)
at org.eclipse.jetty.server.HttpWriter.write(HttpWriter.java:258) at
org.eclipse.jetty.server.HttpWriter.write(HttpWriter.java:107) at
org.eclipse.jetty.io.UncheckedPrintWriter.write(UncheckedPrintWriter.java:271)
... 24 more Caused by: java.io.IOException: Broken pipe at
sun.nio.ch.FileDispatcherImpl.write0(Native Method) at
sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at
sun.nio.ch.IOUtil.write(IOUtil.java:51) at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) at
org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:185)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:256)
at
org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:849)
... 33 more
I know that java.io.IOException: Broken pipe means that the JobManager lost some kind of connection so I guess the whole job failed and I have to restart it. Although I think the process is not running anymore the WebInterface still lists it as running. Additionally the JobManager is still present when I use jps to identify my running processes on the cluster. So my question is if my job is lost and whether this error is happening randomly sometimes or whether my program caused it.
EDIT: My TaskManagers still send Heartbeats every few seconds and seem to be running.
It's actually a problem of the JobManagerInfoServlet, Flink's web server, which cannot sent the latest JSON updates of the requested job to your browser because of the java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method). Thus, only the GET request to the server failed.
Such a failure should not affect the execution of the currently running Flink job. Simply refreshing your browser (with Flink's web UI) should send another GET request which then hopefully completes successfully.

solr dataimport handler jndi websphere

I am trying to use jndiName attribute in db-data-config.xml. This works great in tomcat. However having issues in websphere.
Following exception is thrown
"Make sure that a J2EE application does not execute JNDI operations on "java:" names within static code blocks or in threads created by that J2EE application. Such code does not necessarily run on the thread of a server application request and therefore is not supported by JNDI operations on "java:" names. [Root exception is javax.naming.NameNotFoundException: Name comp/env/jdbc not found in context "java:"."
It seems like websphere has issues accessing jndi resource from Static code. Has anyone experienced this ?
DataImporter E org.apache.solr.common.SolrException log Full Import failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: <REMOVE SQL from here>
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:238)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:596)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:268)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:187)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:359)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:427)
at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
Caused by: javax.naming.ConfigurationException: A JNDI operation on a "java:" name cannot be completed because the server runtime is not able to associate the operation's thread with any J2EE application component. This condition can occur when the JNDI client using the "java:" name is not executed on the thread of a server application request. Make sure that a J2EE application does not execute JNDI operations on "java:" names within static code blocks or in threads created by that J2EE application. Such code does not necessarily run on the thread of a server application request and therefore is not supported by JNDI operations on "java:" names. [Root exception is javax.naming.NameNotFoundException: Name comp/env/jdbc not found in context "java:".]
at com.ibm.ws.naming.java.javaURLContextImpl.throwConfigurationExceptionWithDefaultJavaNS(javaURLContextImpl.java:428)
at com.ibm.ws.naming.java.javaURLContextImpl.lookup(javaURLContextImpl.java:399)
at com.ibm.ws.naming.java.javaURLContextRoot.lookup(javaURLContextRoot.java:214)
at com.ibm.ws.naming.java.javaURLContextRoot.lookup(javaURLContextRoot.java:154)
at javax.naming.InitialContext.lookup(InitialContext.java:436)
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:140)
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:128)
at org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:363)
at org.apache.solr.handler.dataimport.JdbcDataSource.access$200(JdbcDataSource.java:39)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:240)
... 11 more
Caused by: javax.naming.NameNotFoundException: Name comp/env/jdbc not found in context "java:".
at com.ibm.ws.naming.ipbase.NameSpace.getParentCtxInternal(NameSpace.java:1837)
at com.ibm.ws.naming.ipbase.NameSpace.lookupInternal(NameSpace.java:1166)
at com.ibm.ws.naming.ipbase.NameSpace.lookup(NameSpace.java:1095)
at com.ibm.ws.naming.urlbase.UrlContextImpl.lookup(UrlContextImpl.java:1233)
at com.ibm.ws.naming.java.javaURLContextImpl.lookup(javaURLContextImpl.java:395)
... 19 more
Solr is using unmanaged threads here and is doing a JNDI lookup in the java: namespace in one of these threads. That is not supported in WebSphere because WebSphere identifies the java: namespace based on the thread performing the lookup. Tomcat does this based on the thread context class loader, which explains the difference in behavior. Note that unmanaged threads are not allowed by the J2EE specs, so that WebSphere's behavior is actually standards compliant.

JDBC connection hanging

One of our customers has a newish problem: the application grinds to halt. Thread dump shows that all threads are hanging on network IO in JDBC calls.
We/I have never seen these 'network IO' hangs. Typically a slow machine w/ DB problems has either a) one or two long-running queries or b) some type of lock/deadlock. In either of these cases the threads 'hang' on different methods. I have never seen all 30+ threads hanging on network IO.
Below I have included an excerpt from the thread dump. All HTTP threads are hanging on the same java.net.SocketInputStream.read call.
I talked to their dba and sysadmin. According to them 'nothing has changed' in the environment recently which would cause this problem.
db environment
MSSQL 2005 64-bit Service Pack 2
Driver: sqljdbc.jar : 1.0 809 102
Note: they are running an older jdbc driver. AFAIK they tried upgrading from 1.0 to the 1.2 driver but had some other problem.
other environment issues
They're running both the app server and the db server in VMWare VM's. I don't know how this setup affects network performance.
Apparently this is the only application with this problem. I don't know anything else about their network architecture.
Questions
* any insights on this problem?
* if it is network, any next steps for analyzing?
Appendix A: Excerpt from Thread dump
All HTTP connections are hanging on the same method:
"TP-Processor31" daemon prio=5 tid=0x04085b78 nid=0x970 runnable [0x0764d000..0x0764fd6c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.microsoft.sqlserver.jdbc.DBComms.receive(Unknown Source)
at com.microsoft.sqlserver.jdbc.IOBuffer.sendCommand(Unknown Source)
- locked (a com.microsoft.sqlserver.jdbc.DBComms)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.sendExecute(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteQuery(Unknown Source)
We've had similar issues, and traced them back to a buggy JDK update (1.6.29).
We downloaded 1.6.27 (http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html#jdk-6u27-oth-JPR), re-set the JAVA_HOME environment and we were back on track.
It is the changes to JSSE in version 1.6 u29. The change is to partially patch CVE-2011-3389 and CVE-2011-3560.
If are not deploying applets and using java web-start. You might be able to just use the 1.6 u27 jsse.jar. You're still going to have the vulnerability, but it will allow the sqljdbc.jar and sqljdbc4.jar to work.
The other options to migrate to Java 7, the sqljdbc4.jar does work in that environment.
The patch is not complete fix. So expect more changes in future patches.

Resources