cxf-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Kulp <dk...@apache.org>
Subject Re: mysterious exception when hitting cxf service via load balancer with cxf client
Date Wed, 06 Jun 2012 17:34:45 GMT

No idea.  Strange.

I see a filter in there:
net.bull.javamelody.MonitoringFilter

Would that filter be doing something with the stream?  Maybe write a very 
simple servlet that just grabs the InputStream from the request and tried to 
read it to see if that works?   Not really sure what else to suggest.

Dan





On Tuesday, June 05, 2012 11:32:12 AM Jared Shapiro wrote:
> We have a service set up running in tomcat 7.0.26 with cxf version 2.5.2.
> 
> currently it is set up on two hosts behind a hardware load balancer.
> it is working flawlessly when hit by a php client using the standard
> php soap library and when each server is hit directly with a java cxf
> client.
> when a java cxf client hits the service via the load balancer, we
> intermittently see very long lags and the many times, the following
> exception:
> 
> java.lang.RuntimeException: Couldn't parse stream.
>       at
> org.apache.cxf.staxutils.StaxUtils.createXMLStreamReader(StaxUtils.java:1
> 181) at
> org.apache.cxf.interceptor.StaxInInterceptor.handleMessage(StaxInIntercep
> tor.java:104) at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorCh
> ain.java:263) at
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiatio
> nObserver.java:123) at
> org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTP
> Destination.java:207) at
> org.apache.cxf.transport.servlet.ServletController.invokeDestination(Serv
> letController.java:213) at
> org.apache.cxf.transport.servlet.ServletController.invoke(ServletControll
> er.java:193) at
> org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringS
> ervlet.java:126) at
> org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(Abstra
> ctHTTPServlet.java:185) at
> org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPS
> ervlet.java:108) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:641) at
> org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTP
> Servlet.java:164) at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicat
> ionFilterChain.java:305) at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilte
> rChain.java:210) at
> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:197)
> at
> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:171)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicat
> ionFilterChain.java:243) at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilte
> rChain.java:210) at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve
> .java:224) at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve
> .java:169) at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorB
> ase.java:472) at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
> 168) at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:
> 98) at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.j
> ava:118) at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:40
> 7) at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Pr
> ocessor.java:987) at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abst
> ractProtocol.java:579) at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.ja
> va:307) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
> :1110) at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
> a:603) at java.lang.Thread.run(Thread.java:679)
> Caused by: com.ctc.wstx.exc.WstxIOException: Read timed out
>       at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:536)
> at com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:585)
> at com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:610)
> at
> com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory
> .java:316) at
> org.apache.cxf.staxutils.StaxUtils.createXMLStreamReader(StaxUtils.java:1
> 179) ... 31 more
> Caused by: java.net.SocketTimeoutException: Read timed out
>       at java.net.SocketInputStream.socketRead0(Native Method)
>       at java.net.SocketInputStream.read(SocketInputStream.java:146)
>       at
> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.jav
> a:532) at
> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.jav
> a:501) at
> org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRea
> d(InternalInputBuffer.java:563) at
> org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInput
> Filter.java:118) at
> org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.j
> ava:326) at org.apache.coyote.Request.doRead(Request.java:422)
>       at
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:
> 290) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:431)
> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315)
> at
> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.ja
> va:200) at java.io.FilterInputStream.read(FilterInputStream.java:133) at
> com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155) at
> com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:368) at
> com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:111)
>       at
> com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:25
> 0) at
> com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java
> :133) at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:531)
> ... 35 more
> 
> we have checked the contents of the request by comparing a tcp dump
> that the client sends vs what is received on the server on the other
> side of the load balancer and they match.
> 
> again, the cxf client works perfectly when hitting each machine
> directly. it is only when going through the hardware load balancer
> that we see the issue, yet we've verified that the actual contents of
> the request passes through the load balancer unchanged.
> 
> any advice on what may be going on here and how to resolve it would be
> very helpful.
> 
> thanks
-- 
Daniel Kulp
dkulp@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com


Mime
View raw message