camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rsteppac2 <r...@steppacher.name>
Subject Re: Stream-only reverse proxy with minimal memory footprint
Date Fri, 25 Jul 2014 09:46:17 GMT
Willem,

thanks for taking this up.

Ralf

On 25.07.14 10:30, Willem.Jiang [via Camel] wrote:
> Hi,
>
> You just bring up an interesting topic of avoiding extra copy of 
> response input stream.
> It could save us lots of the memory and time if the response is a big 
> and chunked message.
> I just created a JIRA[1] to track this issue. We may create some sub 
> tasks as we need to work on five different http client implement at 
> the time.
>
> I prefer to avoid close the input stream in a Synchronization 
> callback. It could be OK in a proxy route, as the input stream can be 
> consumed when the exchange int the route is completed. But if the 
> input stream is used in the other route, it’s hard for camel to know 
> if the stream is in use unless we have a reference count for it.
>
> [1]https://issues.apache.org/jira/browse/CAMEL-7638
>
> -- 
> Willem Jiang
>
> Red Hat, Inc.
> Web: http://www.redhat.com
> Blog: http://willemjiang.blogspot.com (English)
> http://jnn.iteye.com (Chinese)
> Twitter: willemjiang
> Weibo: 姜宁willem
>
>
>
> On July 25, 2014 at 2:36:14 PM, rsteppac2 ([hidden email] 
> </user/SendEmail.jtp?type=node&node=5754438&i=0>) wrote:
>
> > I am trying to implement a memory efficient http reverse proxy that is
> > working on streams only to handle large HTTP requests and responses. 
> This
> > should be as easy as described here:
> >
> > 
> http://camel.apache.org/how-to-use-camel-as-a-http-proxy-between-a-client-and-server.html

>
> >
> > The Jetty consumer places the socket input stream with the http POST 
> body
> > into the exchange and I can hook it up with a producer to forward the
> > request. No problem here.
> >
> > The problem is the response. All http producers that I am aware of 
> (Jetty,
> > http4, netty-http) read the response stream into a byte-array,wrap 
> it in an
> > InputStream and place that in the exchange body. Thus a large http 
> response
> > blows my proxy out of the water. None of the producers seem to offer an
> > option to make them put the original socket input stream into the 
> exchange,
> > just as the Jetty consumer does.
> >
> > http4: Everything happens in
> > org.apache.camel.component.http4.HttpProducer::process() ->
> > populateResponse(..) -> extractResponseBody(..) ->
> > doExtractResponseBodyAsStream(); here the original stream is copied 
> into an
> > instance of CachedOutputStream, backed by a byte-array.
> >
> > Jetty: org.eclipse.jetty.client.AsyncHttpConnection::handle() ->
> > org.eclipse.jetty.http.HttpParser::parseNext() will fill a 
> byte-array in
> > org.eclipse.jetty.client.ContentExchange which is a CachedExchange 
> which is
> > a HttpExchange.
> >
> > netty-http: Builds a pipeline that assembles the HttpResponse 
> content as a
> > composite ChannelBuffer. The wrapped channel buffers make up the 
> complete
> > response stream.
> >
> > Is there a way to configure any of the clients to place the socket 
> input
> > stream into the exchange body?
> >
> > This thread describes the same problem and proposes a patch, which I 
> think
> > has not made it into the Camel code base as of yet:
> > 
> http://camel.465427.n5.nabble.com/Chunking-issue-with-http-producer-td5735075.html 
>
> >
> > The question is how to close the socket stream of the client. If I 
> should
> > have to replicate the suggested patch, would a Synchronization 
> call-back
> > registered with the exchange via 
> Exchange::addOnCompletion(Synchronization)
> > be the correct place to do it? I have tried with a FileInputStream 
> and that
> > seems to work. I.e. the call-back is triggered after the stream 
> contents
> > have been written out to the socket.
> >
> >
> > Thanks!
> > Ralf
> >
> >
> >
> > --
> > View this message in context: 
> http://camel.465427.n5.nabble.com/Stream-only-reverse-proxy-with-minimal-memory-footprint-tp5754424.html

>
> > Sent from the Camel - Users mailing list archive at Nabble.com.
> >
>
>
>
> ------------------------------------------------------------------------
> If you reply to this email, your message will be added to the 
> discussion below:
> http://camel.465427.n5.nabble.com/Stream-only-reverse-proxy-with-minimal-memory-footprint-tp5754424p5754438.html

>
> To unsubscribe from Stream-only reverse proxy with minimal memory 
> footprint, click here 
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5754424&code=cmFsZkBzdGVwcGFjaGVyLm5hbWV8NTc1NDQyNHwtMjc3MjkzMDM=>.
> NAML 
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

>





--
View this message in context: http://camel.465427.n5.nabble.com/Stream-only-reverse-proxy-with-minimal-memory-footprint-tp5754424p5754440.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message