camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Willem Jiang <>
Subject Re: Stream-only reverse proxy with minimal memory footprint
Date Fri, 25 Jul 2014 08:29:41 GMT

You just bring up an interesting topic of avoiding extra copy of response input stream.
It could save us lots of the memory and time if the response is a big and chunked message.
I just created a JIRA[1] to track this issue. We may create some sub tasks as we need to work
on five different http client implement at the time.

I prefer to avoid close the input stream in a Synchronization callback. It could be OK in
a proxy route, as the input stream can be consumed when the exchange int the route is completed.
But if the input stream is used in the other route, it’s hard for camel to know if the stream
is in use unless we have a reference count for it.


Willem Jiang

Red Hat, Inc.
Blog: (English) (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem

On July 25, 2014 at 2:36:14 PM, rsteppac2 ( wrote:
> I am trying to implement a memory efficient http reverse proxy that is
> working on streams only to handle large HTTP requests and responses. This
> should be as easy as described here:
> The Jetty consumer places the socket input stream with the http POST body
> into the exchange and I can hook it up with a producer to forward the
> request. No problem here.
> The problem is the response. All http producers that I am aware of (Jetty,
> http4, netty-http) read the response stream into a byte-array,wrap it in an
> InputStream and place that in the exchange body. Thus a large http response
> blows my proxy out of the water. None of the producers seem to offer an
> option to make them put the original socket input stream into the exchange,
> just as the Jetty consumer does.
> http4: Everything happens in
> org.apache.camel.component.http4.HttpProducer::process() ->
> populateResponse(..) -> extractResponseBody(..) ->
> doExtractResponseBodyAsStream(); here the original stream is copied into an
> instance of CachedOutputStream, backed by a byte-array.
> Jetty: org.eclipse.jetty.client.AsyncHttpConnection::handle() ->
> org.eclipse.jetty.http.HttpParser::parseNext() will fill a byte-array in
> org.eclipse.jetty.client.ContentExchange which is a CachedExchange which is
> a HttpExchange.
> netty-http: Builds a pipeline that assembles the HttpResponse content as a
> composite ChannelBuffer. The wrapped channel buffers make up the complete
> response stream.
> Is there a way to configure any of the clients to place the socket input
> stream into the exchange body?
> This thread describes the same problem and proposes a patch, which I think
> has not made it into the Camel code base as of yet:
> The question is how to close the socket stream of the client. If I should
> have to replicate the suggested patch, would a Synchronization call-back
> registered with the exchange via Exchange::addOnCompletion(Synchronization)
> be the correct place to do it? I have tried with a FileInputStream and that
> seems to work. I.e. the call-back is triggered after the stream contents
> have been written out to the socket.
> Thanks!
> Ralf
> --
> View this message in context:
> Sent from the Camel - Users mailing list archive at

View raw message