CXF WebService client intermittent occurrence of java.net.SocketTimeoutException: Read timed out - cxf

We have a few CXF Web Services deployed on Websphere Application Server 7.0.0. Its a multi-server environment with load balancer.
We have a multi-threaded web service client (acts as a cache tool) which calls web services and service requests from caller. So essentially, request flow is like this -
caller -> CXF WebService client (multhithreaded) -> F5 load balancer -> Websphere ASs
This all works fine, but some times we see "java.net.SocketTimeoutException: Read timed out" on our WebService client. On one of the Websphere AS (which is handling this incoming request), we see "java.io.IOException: SRVE0080E: Invalid content length". We are not able to reproduce this issue in our test environment and randomly occurs in client's production environment. The occurrence is not very frequent and we verified that load is also not that much. Request doesn't contain too much of data, it just has an "id" of the object which client requested from server. Since server never received the full request data, so the webservice method never gets called.
Full stack trace at client's side -
[pool-1-thread-643] WARN phase.PhaseInterceptorChain Interceptor has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message.
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:472)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:302)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:254)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:123)
at $Proxy55.getObject(Unknown Source)
at com.***.****.***service.***Thread.retrieve(***Thread.java:344)
at com.***.****.***service.***Thread.handleMessage(***Thread.java:232)
at com.***.****.***service.***Thread.run(***Thread.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2046)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2022)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1947)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:632)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
... 13 more
Full stack trace Server side -
java.io.IOException: SRVE0080E: Invalid content length
at com.ibm.ws.webcontainer.srt.http.HttpInputStream.finish(HttpInputStream.java:184)
at com.ibm.ws.webcontainer.srt.http.HttpInputStream.close(HttpInputStream.java:532)
at com.ibm.ws.webcontainer.srt.SRTServletRequest.finish(SRTServletRequest.java:2617)
at com.ibm.ws.webcontainer.srt.SRTConnectionContext.finishConnection(SRTConnectionContext.java:80)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1048)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:516)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:307)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)
Does anyone have any guess what might be happening? Is there a possibility that the thread which is making webservice call is put on wait or suspension state by JVM before it could complete the request and later on socket timed out because JVM didnt reschedule this thread again within 1 min (we have set socket timeout to 1 min)? Is it possible load balancer not able to forward the entire request data to server?
Thanks in advance for your help.

Related

Lose headers when send to camel hazelcast seda endpoint

I use camel with hazelcast component and faced the problem with exchange headers, when exchange goes through hazelcast seda.
I need to poll files from directory and send exchanges to seda hazelcast queue (to processing exchanges in cluster).
There is a simple example of this situation.
There are two routes, which runs in other JVMs:
First for polling files:
from("file:someFromFolder")
.to("hazelcast:seda:queue?hazelcastInstance=#hazelcast-instance");
And second to processed this:
from("hazelcast:seda:queue?hazelcastInstance=#hazelcast-instance")
.to("file:someToFolder");
When I tried to read headers like Exchange.FILE_NAME or others, i realized they are empty.
Is it possible to save headers while transferring exchanges through hazelcast seda component?
EDIT
When I set transferExchange=true as an option on the producer route endpoint, I catch exception:
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.RuntimeExchangeException: Message body of type org.apache.camel.component.file.GenericFile is not supported by this marshaller. on the exchange: Exchange[ID-DF-240815-MS12-60568-1487853045890-0-12][file.txt]
at org.apache.camel.impl.DefaultExchangeHolder.marshal(DefaultExchangeHolder.java:99)
at org.apache.camel.impl.DefaultExchangeHolder.marshal(DefaultExchangeHolder.java:83)
at org.apache.camel.component.hazelcast.seda.HazelcastSedaProducer.checkAndStore(HazelcastSedaProducer.java:55)
at org.apache.camel.component.hazelcast.seda.HazelcastSedaProducer.process(HazelcastSedaProducer.java:42)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:442)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:214)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:178)
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
So could it be marshaller try to serialize all file data?
You need to set transferExchange=true as an option on the endpoint. By default its only the message body that is transferred. You can find this information in the documentation: http://camel.apache.org/hazelcast-component.html#HazelcastComponent-seda

spring security admin gateway and angularjs exception

I am using gateway based authentication in spring security. A user is authenticated from the database. Through zuul, the authenticated user is directed to the UI microservice. A successfully authenticates user at the admin gateway service when routed to the UI microservice gives the following exception.
There was an unexpected error (type=Internal Server Error, status=500).
Cannot deserialize; nested exception is org.springframework.core.serializer.support.SerializationFailedException: Failed to deserialize payload. Is the byte array a result of corresponding serialization for DefaultDeserializer?; nested exception is org.springframework.core.NestedIOException: Failed to deserialize object type; nested exception is java.lang.ClassNotFoundException: mysecurity.CustomUserDetails
Look at your exception:
java.lang.ClassNotFoundException: mysecurity.CustomUserDetails
Just make sure that you have mysecurity.CustomUserDetails class in your classpath

Use Google hosted email to send email from dotnetnuke site hosted on hostgator

I am attempting to configure our dotnetnuke site to be able to send basic messages (things like lost password, newsletters, etc) to our users.
According to all the documentation I have found I have set up the site correctly
SMTP Server and port:smtp.gmail.com:587
Connection Limit:1
Max Idle Time:0
SMTP Authentication:Basic
SMTP Username:username#weburl.com
SMTP Password:password
I also created a smtp relay in google apps with comprehensive mail storage.
Is there something I am missing?
Below is the error message I receive from logs, if I understand what I am seeing this is the hosting service I am using correct?
2014-10-21 19:20:54,310 [PSS16][Thread:16][ERROR] DotNetNuke.Services.Exceptions.Exceptions - System.Net.Mail.SmtpException: Failure sending mail. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions 74.125.198.109:587
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.ServicePoint.GetConnection(PooledStream PooledStream, Object owner, Boolean async, IPAddress& address, Socket& abortSocket, Socket& abortSocket6)
at System.Net.PooledStream.Activate(Object owningObject, Boolean async, GeneralAsyncDelegate asyncCallback)
at System.Net.PooledStream.Activate(Object owningObject, GeneralAsyncDelegate asyncCallback)
at System.Net.ConnectionPool.GetConnection(Object owningObject, GeneralAsyncDelegate asyncCallback, Int32 creationTimeout)
at System.Net.Mail.SmtpConnection.GetConnection(ServicePoint servicePoint)
at System.Net.Mail.SmtpTransport.GetConnection(ServicePoint servicePoint)
at System.Net.Mail.SmtpClient.GetConnection()
at System.Net.Mail.SmtpClient.Send(MailMessage message)
--- End of inner exception stack trace ---
at System.Net.Mail.SmtpClient.Send(MailMessage message)
at DotNetNuke.Services.Mail.Mail.SendMailInternal(MailMessage mailMessage, String subject, String body, MailPriority priority, MailFormat bodyFormat, Encoding bodyEncoding, IEnumerable`1 attachments, String smtpServer, String smtpAuthentication, String smtpUsername, String smtpPassword, Boolean smtpEnableSSL)
I have my settings at:
smtp.gmail.com:587
1
0
Basic
SSL Enabled
username
password
That seems to work fine for my sites using Google's mail services.
It sounds like you need to setup a SMTP relay service. To confirm, you've configured the SMTP relay service as per this, correct?
If you've done everything there, then it's likely a connection configuration issue on your server. Although there's no specific settings mentioned for your provider, it looks like you should be using the FQDN, smtp-relay.gmail.com:587, rather than smtp.gmail.com:587.
Hope this helps!
Are you allowed to open up sockets from your environment? It looks like it might be an OS restriction for the process. Do you have shell access to the machine where this deployment is running?
Login, and see if you can open a connection (telnet) to the host and port you're trying to connect to. Don't look for any usable output, since it's expected to start a SSL handshake. Look for 'connection established' or some other message that indicates a tcp connection is open.

Processing SAML 2.0 response with Apache CXF RACS

I am trying to implement a SAML Request Assertion Consumer Service (RACS) with Apache CXF 2.7.7
Unfortunately the response of my Identity Provider does not include a keyInfo (which is defined optional in the SAML specification).
This leads to an exception when processing the response, because CXF tries to load a DOM for the keyInfo.
java.lang.NullPointerException
at org.apache.ws.security.saml.ext.AssertionWrapper.verifySignature(AssertionWrapper.java:536)
I have a valid keystore.properties file as well as the certificate on my RACS site, but this does not chage the behaviour. Is this a bug in CXF or did I miss something to set up for my RACS?
After discussing this problem in the CXF Mailing List, we found out, that it is a bug in CXF Framework. A new issue was created here
Sending the KeyInfo in a SAML Request is optional (see Specification Section 5.4.5).
Currently there is no easy workaround available.

Processing of multipart/form-data request failed

File upload mechanism is added to my existing application which is deployed to WebSphere 7. I use commons-fileupload-1.2.2.jar for the file upload. During testing, I encountered below exception:
[5/2/13 18:00:00:773 CST] 0000001b SystemOut O 18:00:00,773 INFO AccountsController:156 - Session is valid.
[5/2/13 18:04:42:299 CST] 0000001e servlet E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: Uncaught exception created in one of the service methods of the servlet dispatcher in application SunLinkOnline_ST. Exception created : org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Async operation timed out
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:583)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at ph.com.sunlife.sunlink.filter.MFSanitizerFilter.doFilter(MFSanitizerFilter.java:38)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:445)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:504)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:301)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:275)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1563)
Caused by: org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Async operation timed out
at org.springframework.web.multipart.commons.CommonsMultipartResolver.parseRequest(CommonsMultipartResolver.java:172)
at org.springframework.web.multipart.commons.CommonsMultipartResolver.resolveMultipart(CommonsMultipartResolver.java:149)
at org.springframework.web.servlet.DispatcherServlet.checkMultipart(DispatcherServlet.java:1006)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:851)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
... 34 more
Caused by: org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Async operation timed out
at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:371)
at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126)
at org.springframework.web.multipart.commons.CommonsMultipartResolver.parseRequest(CommonsMultipartResolver.java:165)
... 39 more
Caused by: java.net.SocketTimeoutException: Async operation timed out
Could someone please share some insights what is causing above exception. Though the exception may point to server long response or due to handling of timeout in server but is there a way to resolve this? Like handling of exception in code side? Your help is much appreciated. Please let me know if I need to include other information to resolve this exceptions. Thanks!
It seems like file upload time out need to be increased. I came across same situation once but what i needed was to upload my application to websphere via admin console. Since my application is a large one i have to increase time out for admin console.
Since your scenario is for file upload in application hosted in websphere you have to change the time out of different queue.
Please find following URLs
IBM Websphere timeout custom properties
IBM changing time out in HTTP queues
what i did in my scenario

Resources