tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antoine Bonavita <anto...@stickyads.tv>
Subject NullPointerException in NioEndpoint under load (but not only)
Date Wed, 22 Jul 2015 14:54:22 GMT
Hello,

Before I start with my question and stack trace, let me give you a bit 
of context:
* Tomcat 7.0.61
* OpenJDK1.7.0_79
* Debian7
* Tomcat is running "behind" an nginx server (reverse proxying).
* As we have pretty stringent constraints on response time, nginx is 
configured to reply to the browser after 2s if it has not heard back 
from Tomcat.
* As implied in the email subject, we are using the http-nio connector. 
This is because our code is calling many webservices in parallel and we 
don't want to be stuck waiting for one of them (after 300ms we timeout 
the request and consider they will not answer).

Now, my problem is that once in a while (with a higher probability under 
load), I get this exception:
java.lang.NullPointerException
         at 
org.apache.tomcat.util.net.NioEndpoint.processSocketAsync(NioEndpoint.java:730) 
~[tomcat-coyote.jar:7.0.61]
         at 
org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.java:84) 
~[tomcat-coyote.jar:7.0.61]
         at 
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:764)

~[tomcat-coyote.jar:7.0.61]
         at org.apache.coyote.Response.action(Response.java:174) 
~[tomcat-coyote.jar:7.0.61]
         at org.apache.coyote.Response.finish(Response.java:274) 
~[tomcat-coyote.jar:7.0.61]
         at 
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319) 
~[catalina.jar:7.0.61]
         at 
org.apache.catalina.connector.CoyoteWriter.close(CoyoteWriter.java:112) 
~[catalina.jar:7.0.61]
         at 
networkComm.commands.HttpCommand.sendResponse(HttpCommand.java:239) 
~[HttpCommand.class:?]
         at 
com.stickyadstv.adex.AuctioneerResponseWriter.respondToClient(AuctioneerResponseWriter.java:323)

~[AuctioneerResponseWriter.class:?]
         at 
com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)

~[BidSerializationListener.class:?]
         at 
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)

~[BidSerializationListener.class:?]
         at 
com.stickyadstv.adex.bidder.marketplace.MarketPlaceBidSerializationWriter.respondToClient(MarketPlaceBidSerializationWriter.java:92)

~[MarketPlaceBidSerializationWriter.class:?]
         at 
com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)

~[BidSerializationListener.class:?]
         at 
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)

~[BidSerializationListener.class:?]
         at 
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:24)

~[BidSerializationListener.class:?]
         at 
org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) 
~[httpcore-4.4.1.jar:4.4.1]
         at 
com.stickyadstv.adex.bidder.openrtb.OpenRTBBid.serializeAdm(OpenRTBBid.java:166) 
~[OpenRTBBid.class:?]
         at 
com.stickyadstv.adex.bidder.openrtb.AdmReceived.completed(AdmReceived.java:111) 
~[AdmReceived.class:?]
         at 
com.stickyadstv.adex.bidder.openrtb.AdmReceived.completed(AdmReceived.java:32) 
~[AdmReceived.class:?]
         at 
org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) 
~[httpcore-4.4.1.jar:4.4.1]
         at 
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)

~[httpasyncclient-4.1.jar:4.1]
         at 
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:412)

~[httpcore-nio-4.4.1.jar:4.4.1]
         at 
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:305)

~[httpcore-nio-4.4.1.jar:4.4.1]
         at 
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)

~[httpcore-nio-4.4.1.jar:4.4.1]
         at 
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)

~[httpasyncclient-4.1.jar:4.1]
         at 
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)

~[httpasyncclient-4.1.jar:4.1]
         at 
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:116)

~[httpcore-nio-4.4.1.jar:4.4.1]
         at 
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164) 
~[httpcore-nio-4.4.1.jar:4.4.1]
         at 
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)

~[httpcore-nio-4.4.1.jar:4.4.1]
         at 
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317)

~[httpcore-nio-4.4.1.jar:4.4.1]
         at 
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278) 
~[httpcore-nio-4.4.1.jar:4.4.1]
         at 
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) 
~[httpcore-nio-4.4.1.jar:4.4.1]
         at 
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)

~[httpcore-nio-4.4.1.jar:4.4.1]
         at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]

I looked a bit at the NioEndpoint and AbstractProcessor code and it 
looks to me like the only possible cause is that 
AbstractProcessor.socketWrapper is null at this point. Can someone 
confirm my analysis ?

Now, I tried to figure out how this can happen and that's where I got 
lost in the code. So, if someone could help me figure out what in our 
code could lead to this, that would be very much appreciated.

Last, but not least, there seems to be a correlation between those 
exceptions and spikes with the LimitLatch that never go down. My 
question is : could this NPE result in the LimitLatch not going down ?

Thanks for your help and obviously, do not hesitate to ask me questions 
if you need more information I did not provide.

A.

-- 
Antoine Bonavita (antoine@stickyads.tv) - CTO StickyADS.tv
Tel: +33 6 34 33 47 36/+33 9 50 68 21 32
NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN | MADRID

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message