Camel-Jetty http proxy + large response data casue issue: Buffering capacity exceeded - apache-camel

Question

jetty expert, I got error when getting back large data via camel-jetty as http proxy, any experience/idea/suggestion are welcome, thanks in advance. RouteBuilder code:
from("servlet:balancer?matchOnUriPrefix=true")
.to("jetty:http://vm-a38e-8f19.nam.nsroot.net:19999/balancer/?" + "bridgeEndpoint=true"
+ "&disableStreamCache=true"
+ "&throwExceptionOnFailure=false"
+ "&bufferSize=1024000000"
+ "&inboundBufferSize=1024000000"
+ "&outboundBufferSize=1024000000"
+ "&eagerCheckContentAvailable=true"
+ "&httpClient.responseBufferSize=1024000000&responseBufferSize=1024000000&chunked=false")
ENV:
<jdk.version>1.8</jdk.version>
<camel.version>2.17.0</camel.version>
<spring.version>4.2.4.RELEASE</spring.version>
<spring.security.version>4.0.4.RELEASE</spring.security.version>
Got error when http based rest-service return back 5000 records of json data, I have tried many ways to adjust the response buffer size, unfortunately still not working, it works when service side return <1000 records:
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 314] - Response content HttpResponse[HTTP/1.1 200 OK]#54e3183e
DirectByteBufferR#61e4dcc8[p=40262,l=47776,c=1024000000,r=7514]={umn12":"data4782"...4875","\r\n2000\r\n<<<column10":"data48...16":"data4892",>>>data3198","column...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 158] - Parsed false, remaining 0 HttpParser{s=CHUNK,2096474 of -1}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser 1232] - parseNext s=CHUNK DirectByteBuffer#26a1b8da[p=47776,l=47776,c=1024000000,r=0]={umn12":"data4782"...16":"data4892",<<<>>>data3198","column...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 158] - Parsed false, remaining 0 HttpParser{s=CHUNK,2096474 of -1}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint 142] - filled 31280 SelectChannelEndPoint#7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,3/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 113] - Read 31280 bytes DirectByteBuffer#26a1b8da[p=0,l=31280,c=1024000000,r=31280]={<<<"column17":"data4...ata4964","colum>>>54","column16":"d...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} from SelectChannelEndPoint#7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,0/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser 1232] - parseNext s=CHUNK DirectByteBuffer#26a1b8da[p=0,l=31280,c=1024000000,r=31280]={<<<"column17":"data4...ata4964","colum>>>54","column16":"d...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 314] - Response content HttpResponse[HTTP/1.1 200 OK]#54e3183e
DirectByteBufferR#5991d3e4[p=0,l=678,c=1024000000,r=678]={<<<"column17":"data4...a4894","column8>>>\r\n2000\r\n":"data48...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser 1587] - CHUNK --> CHUNKED_CONTENT
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser 1587] - CHUNKED_CONTENT --> CHUNK_SIZE
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser 1587] - CHUNK_SIZE --> CHUNK
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 314] - Response content HttpResponse[HTTP/1.1 200 OK]#54e3183e
DirectByteBufferR#2b0b1218[p=686,l=8878,c=1024000000,r=8192]={"column17":"data4...column8\r\n2000\r\n<<<":"data4894","col..."column6":"data>>>\r\n2000\r\n4913","co...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange 228] - Failed HttpExchange#2e196e3f req=TERMINATED/null#null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded#35d21497: req=false/rsp=true java.lang.IllegalArgumentException: Buffering capacity exceeded
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser 1555] - close HttpParser{s=CHUNK,2105344 of -1}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser 1587] - CHUNK --> CLOSED
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 514] - Response failure HttpResponse[HTTP/1.1 200 OK]#54e3183e HttpExchange#2e196e3f req=TERMINATED/null#null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded#35d21497 on HttpChannelOverHTTP#4b2502b4(exchange=HttpExchange#2e196e3f req=TERMINATED/null#null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded#35d21497)[send=HttpSenderOverHTTP#325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]: java.lang.IllegalArgumentException: Buffering capacity exceeded
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 529] - Concurrent failure: response termination skipped, performed by helpers
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange 259] - Aborted (true) while active HttpExchange#2e196e3f req=TERMINATED/null#null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded#35d21497: java.lang.IllegalArgumentException: Buffering capacity exceeded
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 541] - State update failed: TRANSIENT -> CONTENT: FAILURE
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange 210] - Terminated response for HttpExchange#2e196e3f req=TERMINATED/null#null res=TERMINATED/java.lang.IllegalArgumentException: Buffering capacity exceeded#35d21497, result: Result[HttpRequest[GET /balancer/restservice/largedata HTTP/1.1]#66fc8f65 > HttpResponse[HTTP/1.1 200 OK]#54e3183e] java.lang.IllegalArgumentException: Buffering capacity exceeded
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 442] - Response complete HttpResponse[HTTP/1.1 200 OK]#54e3183e
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpChannel 89] - HttpExchange#2e196e3f req=TERMINATED/null#null res=TERMINATED/java.lang.IllegalArgumentException: Buffering capacity exceeded#35d21497 disassociated true from HttpChannelOverHTTP#4b2502b4(exchange=null)[send=HttpSenderOverHTTP#325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.ConnectionPool 257] - Connection removed HttpConnectionOverHTTP#383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP#4b2502b4(exchange=null)[send=HttpSenderOverHTTP#325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] - pooled: 0
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint 75] - oshut SelectChannelEndPoint#7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.http.HttpConnectionOverHTTP 154] - HttpConnectionOverHTTP#383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP#4b2502b4(exchange=null)[send=HttpSenderOverHTTP#325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] oshut
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractEndPoint 108] - onClose SelectChannelEndPoint#7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,in,OSHUT,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint 115] - close SelectChannelEndPoint#7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,in,OSHUT,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.SelectorManager 756] - Destroyed SelectChannelEndPoint#7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,ISHUT,OSHUT,-,-,5/0,HttpConnectionOverHTTP}{io=0,kio=-1,kro=-1}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractConnection 210] - onClose HttpConnectionOverHTTP#383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP#4b2502b4(exchange=null)[send=HttpSenderOverHTTP#325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]]
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractEndPoint 108] - onClose SelectChannelEndPoint#7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,ISHUT,OSHUT,-,-,5/0,HttpConnectionOverHTTP}{io=0,kio=-1,kro=-1}
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.http.HttpConnectionOverHTTP 157] - HttpConnectionOverHTTP#383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP#4b2502b4(exchange=null)[send=HttpSenderOverHTTP#325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] closed
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 450] - Request/Response failed: Result[HttpRequest[GET /balancer/restservice/largedata HTTP/1.1]#66fc8f65 > HttpResponse[HTTP/1.1 200 OK]#54e3183e] java.lang.IllegalArgumentException: Buffering capacity exceeded
[2016/04/11 18:32:09 CST] [DEBUG] [org.apache.camel.processor.DefaultErrorHandler 153] - Failed delivery for (MessageId: ID-APACCNDLBIW2480-51289-1460370708686-0-7 on ExchangeId: ID-APACCNDLBIW2480-51289-1460370708686-0-8). On delivery attempt: 0 caught: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]
[2016/04/11 18:32:09 CST] [DEBUG] [org.apache.camel.processor.Pipeline 78] - Message exchange has failed: so breaking out of pipeline for exchange: Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8] Exception: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]
[2016/04/11 18:32:09 CST] [ERROR] [org.apache.camel.processor.DefaultErrorHandler 204] - Failed delivery for (MessageId: ID-APACCNDLBIW2480-51289-1460370708686-0-7 on ExchangeId: ID-APACCNDLBIW2480-51289-1460370708686-0-8). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route1 ] [route1 ] [servlet:/balancer?matchOnUriPrefix=true ] [ 3135]
[route1 ] [policy1 ] [policy[null] ] [ 3136]
[route1 ] [to1 ] [jetty:http://vm-a38e-8f19.nam.nsroot.net:19999/balancer/?bridgeEndpoint=true&d] [ 3133]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]
at org.apache.camel.component.jetty9.JettyContentExchange9.doTaskCompleted(JettyContentExchange9.java:155)
at org.apache.camel.component.jetty9.JettyContentExchange9$2.onComplete(JettyContentExchange9.java:221)
at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:193)
at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:185)
at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:453)
at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:434)
at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:360)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:254)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:38)
at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1499)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1272)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:156)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:117)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:89)
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:122)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Buffering capacity exceeded
at org.eclipse.jetty.client.util.BufferingResponseListener.onContent(BufferingResponseListener.java:114)
at org.eclipse.jetty.client.api.Response$Listener$Adapter.onContent(Response.java:245)
at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:124)
at org.eclipse.jetty.client.ResponseNotifier.access$100(ResponseNotifier.java:35)
at org.eclipse.jetty.client.ResponseNotifier$ContentCallback.process(ResponseNotifier.java:274)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:117)
at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:322)
... 13 more
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver 158] - Parsed true, remaining 22402 HttpParser{s=CLOSED,2105344 of -1}

Answer 1

The reason is: camel-jetty9-2.17.0.jar has a limitation, org.apache.camel.component.jetty9.JettyContentExchange9#send
BufferingResponseListener responseListener = new BufferingResponseListener() {
#Override
public void onComplete(Result result) {
if (result.isFailed()) {
doTaskCompleted(result.getFailure());
} else {
onResponseComplete(result, getContent(), getMediaType());
}
}
};
In BufferingResponseListener there is default maxLength is 2MB(2 * 1024 * 1024). And single parameter constructor: BufferingResponseListener(int maxLength) is only way to adjust maxLength. but camel use a default constructor, so all response larger than 2MB will get such error.