On 13/09/2011 13:19, Darius D. wrote:
>=20
> Hi,
>=20
> well the problem is that we have the same problem, real strange NPEs in=
> servlets.
>=20
> It is more pronounced when using dedicated thread executor, moving to
> threads provided by connector itself (with maxThreads=3D"500" option us=
ing NIO
> ) seems to be lowering counts of errors. You could try commenting out t=
he
> executor pool and using thread pool provided by connector to see if it =
helps
> your case.
>=20
>=20
> Atm we are seeing 3 modes of failure in our tomcat setup ( nginx revers=
e
> proxy -> tomcat with NIO connector )
>=20
>=20
> 1)=20
> SEVERE: Error finishing response
> java.lang.NullPointerException
>=20
> and/or
>=20
> SEVERE: Error processing request
> java.lang.NullPointerException
>=20
> in logs. ( could be some app problem, but still shady, as no stack trac=
e is
> given, like if exception object is gone itself??? )
>=20
> 2) Sometimes we get 502 like Rakesh in servlets that should be fine ( r=
esult
> of NPE in real strange places )
>=20
> 3) And most scary ones are the following:
>=20
> 2011 09 13 14:18:20.568 WARN Servlet - error response: error=3Dno act=
ion
> query: [type=3DgroupsChangedAction&v=3D1315912033175] content type:null=
> protocol: HTTP/1.0 length:-1 character encoding: null GET
> 2011 09 13 14:18:20.937 WARN Servlet - error response: error=3Dno act=
ion
> query: [type=3DgroupsChangedAction&v=3D1315912699092] content type:null=
> protocol: HTTP/1.0 length:-1 character encoding: null GET
>=20
> In tomcat access log they are shown as:
> 127.0.0.1 - - [13/Sep/2011:14:18:20 +0300] "GET
> /ourapp/servlet/Servlet?type=3DgroupsChangedAction&v=3D1315912033175 HT=
TP/1.0"
> 200 15
> 127.0.0.1 - - [13/Sep/2011:14:18:20 +0300] "GET
> /ourapp/servlet/Servlet?type=3DgroupsChangedAction&v=3D1315912699092 HT=
TP/1.0"
> 200 15
>=20
> So as you can see, tomcat got that GET reqest with type and v params li=
ke it
> gets and processes thousands others.
>=20
> Servlet code looks like:
> private void doProcess(HttpServletRequest request, HttpServletResponse
> response) throws IOException {
> HttpSession session =3D request.getSession();
> if (session !=3D null) {
>=20
> String action =3D request.getParameter("type");
> logger.debug("action: " + action);
> =09
> if (action =3D=3D null) {=20
>=20
> printError(request, response,"error=3Dno action");
> ....
>=20
> private void printError(HttpServletRequest request, HttpServletRespons=
e
> response, String error) throws IOException {
> logger.warn("error response: " + error + " ip: " +
> request.getRemoteAddr() + " query: [" + request.getQueryString() + "]" =
+ "
> content type:" + request.getContentType() + " protocol: " +
> request.getProtocol() + " length:" + request.getContentLength() + "
> character encoding: " + request.getCharacterEncoding() + " " +
> request.getMethod());
> =20
> response.setContentType("text/plain");
> PrintWriter out =3D response.getWriter();
> out.print(error);
> out.flush();
> out.close();
> }
>=20
> It happens several times in a day in bursts ( GC collecting something?)=
=2E
>=20
>=20
> So really puzzling and random error, tomcat got proper request as shown=
in
> access logs (and even request.getQueryString() is valid :) ) , processe=
d
> request from same user 5s ago, will process one more 5s later, but some=
how
> request is NULL??? Now we handle it by retrying, but such random NULL i=
s
> fatal for other servlets and services that throw NPEs and it seems that=
some
> tomcat code is affected too with (1).
>=20
>=20
> I'd grateful if some smart guys could look into this problem and willin=
g to
> provide any other info.
>=20
> BR,
>=20
> Darius.
>=20
>=20
> P.S. this is kinda thread hijack, but i feel the root cause of these Y
> errors in XXXX requests are the same.
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
> rakesh k-3 wrote:
>>
>> Hi Darius
>>
>> Thanks for reviewing the configurations.
>>
>> in our catalina.out we had the stack trace which you had menitoned.
>>
>> please find the stack trace information which has been requested.
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess=
orImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java=
:194)
>> at
>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(=
RPCMessageReceiver.java:102)
>> at
>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusiness=
Logic(AbstractInOutMessageReceiver.java:40)
>> at
>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMes=
sageReceiver.java:114)
>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)
>> at
>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequ=
est(HTTPTransportUtils.java:167)
>> at
>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:14=
2)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli=
cationFilterChain.java:290)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi=
lterChain.java:206)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa=
lve.java:233)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa=
lve.java:191)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja=
va:128)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja=
va:102)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv=
e.java:109)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java=
:286)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:=
845)
>> at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces=
s(Http11Protocol.java:583)
>> at
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint=
=2Ejava:354)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu=
tor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.=
java:908)
>> at java.lang.Thread.run(Thread.java:619)
>>
>>
>> Darius, if you require more information, i ca provide you the details.=
>>
>> Regards
>> Rakesh
>>
>>
>> On Mon, Sep 12, 2011 at 5:49 PM, Darius D. <darius.ski@gmail.com> wrot=
e:
>>
>>>
>>>
>>> rakesh k-3 wrote:
>>>>
>>>> Hi Christopher
>>>>
>>>> thanks for viewing m configuration ..as mentioned we are using http
>>>> connectors in my configuration not AJP.
>>>>
>>>> and here are my configuration details.
>>>>
>>>> Apache tomcat configuration (server.xml)
>>>>
>>>> <Executor name=3D"tomcatThreadPool" namePrefix=3D"catalina-exec-"
>>>> maxThreads=3D"150" minSpareThreads=3D"4"/>
>>>> <Connector executor=3D"tomcatThreadPool"
>>>> port=3D"8080" protocol=3D"HTTP/1.1"
>>>> connectionTimeout=3D"20000"
>>>> redirectPort=3D"8443" />
>>>>
>>>> Apache httpd configuration Httpd.conf.
>>>>
>>>> ProxyPass / http://localhost:8080 route=3Dmsf1
>>>>
>>>> Regards
>>>> Rakesh
>>>>
>>>>
>>>>
>>>
>>>
>>> Hi,
>>>
>>> can you check your catalina.out to see if there is something like:
>>>
>>> org.apache.coyote.http11.AbstractHttp11Processor endRequest
>>> SEVERE: Error finishing response
>>> java.lang.NullPointerException
>>>
>>> or
>>>
>>> SEVERE: Error processing request
>>> java.lang.NullPointerException
>>>
>>> without any further details / stack traces ?
>>>
>>> Darius.
You should start another thread if you also have a problem.
p
|