tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pid <...@pidster.com>
Subject Re: Apache Tomcat unable to send the request. tomcat throwing 502 Gate Way error
Date Tue, 13 Sep 2011 21:43:41 GMT
On 13/09/2011 13:19, Darius D. wrote:
> 
> Hi,
> 
> well the problem is that we have the same problem, real strange NPEs in
> servlets.
> 
> It is more pronounced when using dedicated thread executor, moving to
> threads provided by connector itself (with maxThreads="500" option using NIO
> ) seems to be lowering counts of errors. You could try commenting out the
> executor pool and using thread pool provided by connector to see if it helps
> your case.
> 
> 
> Atm we are seeing 3 modes of failure in our tomcat setup ( nginx reverse
> proxy -> tomcat with NIO connector )
> 
> 
> 1) 
> SEVERE: Error finishing response
> java.lang.NullPointerException
> 
> and/or
> 
> SEVERE: Error processing request
> java.lang.NullPointerException
> 
> in logs. ( could be some app problem, but still shady, as no stack trace is
> given, like if exception object is gone itself??? )
> 
> 2) Sometimes we get 502 like Rakesh in servlets that should be fine ( result
> of NPE in real strange places )
> 
> 3) And most scary ones are the following:
> 
> 2011 09 13 14:18:20.568 WARN  Servlet  - error response: error=no action
> query: [type=groupsChangedAction&v=1315912033175] content type:null
> protocol: HTTP/1.0 length:-1 character encoding: null  GET
> 2011 09 13 14:18:20.937 WARN  Servlet  - error response: error=no action
> query: [type=groupsChangedAction&v=1315912699092] content type:null
> protocol: HTTP/1.0 length:-1 character encoding: null  GET
> 
> In tomcat access log they are shown as:
> 127.0.0.1 - - [13/Sep/2011:14:18:20 +0300] "GET
> /ourapp/servlet/Servlet?type=groupsChangedAction&v=1315912033175 HTTP/1.0"
> 200 15
> 127.0.0.1 - - [13/Sep/2011:14:18:20 +0300] "GET
> /ourapp/servlet/Servlet?type=groupsChangedAction&v=1315912699092 HTTP/1.0"
> 200 15
> 
> So as you can see, tomcat got that GET reqest with type and v params like it
> gets and processes thousands others.
> 
> Servlet code looks like:
> private void doProcess(HttpServletRequest request, HttpServletResponse
> response) throws IOException {
> 		HttpSession session = request.getSession();
> 		if (session != null) {
> 
> 			String action = request.getParameter("type");
> 			logger.debug("action: " + action);
> 			
> 			if (action == null) { 
> 
> 				printError(request, response,"error=no action");
> ....
> 
> 	private void printError(HttpServletRequest request, HttpServletResponse
> response, String error) throws IOException {
> 		logger.warn("error response: " + error + " ip: "  +
> request.getRemoteAddr() + " query: [" + request.getQueryString() + "]" + "
> content type:" + request.getContentType() + " protocol: " +
> request.getProtocol() +  " length:" + request.getContentLength() + "
> character encoding: " + request.getCharacterEncoding() + "  " +
> request.getMethod());
> 	    
> 		response.setContentType("text/plain");
> 		PrintWriter out = response.getWriter();
> 		out.print(error);
> 		out.flush();
> 		out.close();
> 	}
> 
> It happens several times in a day in bursts ( GC collecting something?).
> 
> 
> So really puzzling and random error, tomcat got proper request as shown in
> access logs (and even request.getQueryString() is valid :) ) , processed
> request from same user 5s ago, will process one more 5s later, but somehow
> request is NULL??? Now we handle it by retrying, but such random NULL is
> fatal for other servlets and services that throw NPEs and it seems that some
> tomcat code is affected too with (1).
> 
> 
> I'd grateful if some smart guys could look into this problem and willing to
> provide any other info.
> 
> BR,
> 
> Darius.
> 
> 
> P.S. this is kinda thread hijack, but i feel the root cause of these Y
> errors in XXXX requests are the same.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> rakesh k-3 wrote:
>>
>> Hi Darius
>>
>> Thanks for reviewing the configurations.
>>
>> in our catalina.out we had the stack trace which you had menitoned.
>>
>> please find the stack trace information which has been requested.
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:194)
>> at
>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:102)
>> at
>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
>> at
>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)
>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)
>> at
>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)
>> at
>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
>> at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>> at
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:354)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>>
>>
>> Darius, if you require more information, i ca provide you the details.
>>
>> Regards
>> Rakesh
>>
>>
>> On Mon, Sep 12, 2011 at 5:49 PM, Darius D. <darius.ski@gmail.com> wrote:
>>
>>>
>>>
>>> rakesh k-3 wrote:
>>>>
>>>> Hi Christopher
>>>>
>>>> thanks for viewing m configuration ..as mentioned we are using http
>>>> connectors in my configuration not AJP.
>>>>
>>>> and here are my  configuration details.
>>>>
>>>> Apache tomcat configuration (server.xml)
>>>>
>>>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>>>         maxThreads="150" minSpareThreads="4"/>
>>>>     <Connector executor="tomcatThreadPool"
>>>>                port="8080" protocol="HTTP/1.1"
>>>>                connectionTimeout="20000"
>>>>                redirectPort="8443" />
>>>>
>>>> Apache httpd configuration Httpd.conf.
>>>>
>>>> ProxyPass / http://localhost:8080 route=msf1
>>>>
>>>> Regards
>>>> Rakesh
>>>>
>>>>
>>>>
>>>
>>>
>>> Hi,
>>>
>>> can you check your catalina.out to see if there is something like:
>>>
>>> org.apache.coyote.http11.AbstractHttp11Processor endRequest
>>> SEVERE: Error finishing response
>>> java.lang.NullPointerException
>>>
>>> or
>>>
>>> SEVERE: Error processing request
>>> java.lang.NullPointerException
>>>
>>> without any further details / stack traces ?
>>>
>>> Darius.

You should start another thread if you also have a problem.


p



Mime
View raw message