cxf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <>
Subject Re: Async http client experiments....
Date Sun, 29 Jul 2012 10:59:15 GMT
On Sat, 2012-07-28 at 16:47 -0400, Daniel Kulp wrote:
> On Saturday, July 28, 2012 03:14:16 PM Oleg Kalnichevski wrote:
> > > 4) Apache HTTP Components (HC)- this was the first one I tried, ran into
> > > performance issues, abandoned it to test the others, then came back to
> > > it
> > > and figured out the performance issue.  :-)   I had this "working", but
> > > a
> > > simple "hello world" echo in a loop resulted in VERY VERY slow
> > > operation,
> > > about 20x slower than the URLConnection in the JDK.   Couldn't figure
> > > out
> > > what was going on which is why I started looking at the others.   I came
> > > back to it and started doing wireshark captures and discovered that it
> > > was waiting for ACK packets whereas the other clients were not.   The
> > > main issue was that the docs for how to set the TCP_NO_DELAY flag
> > > (which, to me, should be the default) seem to be more geared toward the
> > > 3.x or non-NIO versions. Anyway, once I managed to get that set, things
> > > improved significantly.  For non-chunked data, it seems to be working
> > > very well.  For chunked data, it seems to work well 99% of the time.  
> > > It's that last 1% that's going to drive me nuts.  :-(   It's
> > > occassionally writing out bad chunk headers, and I have no idea why.  
> > > A raw wireshark look certainly shows bad chunk headers heading out.   I
> > > don't know if it's something I'm doing or a bug in their stuff.  Don't
> > > really know yet.
> > 
> > Hi Daniel
> > 
> > If my memory serves me well, we have not had a single confirmed case of
> > message corruption I could think of for many years. I took a cursory
> > look at your code and could not spot anything obviously wrong. I am
> > attaching a patch that adds wire and i/o event logging to your HTTP
> > conduit. If you set 'org.apache.http' category to DEBUG priority you
> > should be able to see what kind of stuff gets written to and read from
> > the underlying NIO channel and compare it with what you see with
> > Wireshark.
> I'll take a look at the patch a bit on monday.   BTW:  the CXF sandbox where 
> I'm working on this is open to all Apache committers.  Feel free to commit 
> anything that may help.  :-) 

Hi Daniel

All right. Then, with your permission, I'll try to commit some of my
changes. You can always revert or re-work them if anything is not to
your liking.

> > If you let me know how to reproduce the issue I'll happily investigate
> > and try to find out what causes it. If there is anything wrong with
> > HttpCore I'll get it fixed.
> I just committed a quick update to the testcase to make the test messages 
> large enough to start chunking.   If you run the 
> AsyncHTTPConduitTest.testCalls  
> test in the unit test dir, it will USUALLY blow up after a few hundred 
> requests.    It will likely just hang if run from maven (I don't have any 
> exception handling in place yet so it's waiting for a response that just 
> won't be there).   The Jetty server it starts up has bailed due to an 
> invalid chunk size.   If you run from with Eclipse or similar, you will 
> likely see the log output from Jetty which will say something like:
> Caused by: bad chunk char: 112
> 	at org.eclipse.jetty.http.HttpParser.parseNext(
> 	at 
> org.eclipse.jetty.http.HttpParser.blockForContent(
> 	at
> 	at
> From my wireshark trace, I see what was sent as:
> POST http://localhost:9001/SoapContext/SoapPort HTTP/1.1
> Accept: */*
> User-Agent: Apache CXF 2.7.0-SNAPSHOT
> SOAPAction: ""
> Transfer-Encoding: chunked
> Content-Type: text/xml; charset=UTF-8
> Host: localhost:9001
> Connection: Keep-Alive
> <soap:Envelope..........
> Since there are two lines between the "Keep-Alive" line and the 
> <soap:env..., I assume some space was reserved in the chunk for the header, 
> but it wasn't written there.   Not really sure.   

I think I have found the problem. It appears to be a classic
thread-safety issue. Content codecs (ContentEncoder / ContentDecoder
instances) are not thread-safe and are not meant to be accessed by
worker threads directly. Worker threads are expected to use shared
buffers to produce and consume message bodies and let the I/O dispatcher
take care of transferring data from and to the underlying channel
(through a content codec) instead of trying to force data in or out of
the content codec directly. The reason why content codecs are not meant
to be shared is to minimize the chances of blocking the I/O dispatch
thread and all other connections on the same I/O selector along with

I'll see if I can re-work your code a little and fix the threading

> Another note is that all the calls are occuring one after another.   With 
> the keep-alives, I kind of would have expected just a single connection, but 
> I'm seeing additional connections being established.  Might be something I'm 
> doing (not marking a connection complete in time or something).   On my todo 
> list to look at as well.

I'll look into the connection pooling issue as soon as the basic
transport works reliably. 

> > > In anycase, I'm likely going to pursue option #4 a bit more and see if I
> > > can figure out the last issue with it.
> > > 
> > > >From a performance standpoint, for synchronous request/response, none
> > > >of
> > > 
> > > them perform as well as the in-jdk HttpURLConnection for what we do.  
> > > Netty came the closest at about 5% slower.  HC was about 10%, Jetty
> > > about 12%. Gave up on Ning before running benchmarks.
> > 
> > I am quite surprised the difference compared to HttpURLConnection is
> > relatively small. In my experience a decent blocking HTTP client
> > outperforms a decent non-blocking HTTP client by as much as 50% as long
> > as the number of concurrent connections is moderate (<500). NIO starts
> > paying off only with concurrency well over 2000 or when connections stay
> > idle most of the time.
> Oh, cool.  I was actually disappointed with the numbers.  If they look good 
> to you, I'm happy.  :-)
> Most of the overhead in this case is not so much with the http library but 
> with CXF.   CXF is generating the XML during the write process and that's 
> not exactly "cheap".   Thus, it's not like "normal" benchmarks that are 
> trying to blast pre-generated content back and forth as byte[].  

That makes sense.



View raw message