tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier ...@ice-sa.com>
Subject Re: Packet misses in Tomcat
Date Tue, 07 Jan 2014 10:09:24 GMT
Divyaprakash Y wrote:
>>>> Issue: Few packets do not reach the application servlet but tomcat receives
them. The missing packets reach the HTTP layer and thereafter they disappear.
>>>>
>>>> This issue is not frequent but occasionally consistent. For the POSTs of
missing packet I am not able to find the entry in localhost_access_log.
>>>> How do you know that the packet reaches tomcat if there is no matching entry
in localhost_access.log? Does all other access appear in this file?
>>> I could see the packet in the wireshark capture and it has reached HTTP layer
so I thought it has reached Tomcat; I may be wrong here. Also, as far as my observation, every
hit to the application URLs was getting logged in local access log.
>>>
>> And is there anything in the Tomcat error logs ?
>> (Or the Windows Event logs)
>>
>> A HTTP request might be discarded by Tomcat for various reasons (*)
>> before it is ever mapped to an application.
>> In such a case, there is probably also no log of the request in the Access log.
>> But I would expect some error message in the Tomcat error logs.
>>
>> (*) Invalid HTTP request, incomplete request, etc..
>> If there are really packets being lost somewhere, then for POST
>> requests the request size would not match the Content-length header, and that may
be one of these cases.
>>
>> I must say that the "packets lost" hypothesis sound a bit iffy to me.
>> This is TCP, which should detect missing packets and cause a client
>> connection abort if it was the case.  The invalid HTTP request being rejected by
Tomcat sounds more probable to me.
>> i
>> I could not find anything (Exception) in tomcat std error log file or any file in
the log folder. But I  have not checked the Windows event logs. Will do that.
>>
>> Regarding the content length, there is no mismatch as I have validated using Wireshark
and also the same packets(In terms of structure) have reached the application previously.
>>
>> W.r.t. the last point, I could see TCP ACK for the received packet in the wireshark
and the packet has reached HTTP layer which goes missing thereafter.
>>
> 
>> What do you mean exactly by "the packet has reached HTTP layer" ?
> 
> Are you using the word "packet" in the TCP/IP sense here, or do you mean "HTTP request"
> (composed of one or more TCP/IP packets) ?
> Does one complete HTTP POST request (headers and body) fit into one TCP packet ?
> 
> And how exactly do you recognise that some particular packet (or HTTP request) has not
been processed by the application ?
> Do these disappearing requests have some special characteristic that allows you to distinguish
them from other requests to the same application ?
> Or is it so that all the requests for that same application "disappear" ?
> Do all the ones that disappear have something in common, that other requests (which do
not
> disappear) do not have ?
> 
> 
> Suggestion : paste the content of your Tomcat's "server.xml" file in your next message,
after removing any private information such as host name or IP, passwords etc.
> (Do not send it as attachment, this list often removes them).
> 
> Actually, I meant "HTTP request has reached HTTP Layer".
> 
> I could recognise the miss as some noticeable activity will happen upon the reception
of the request in the application because of which it is easy to identify the misses in this
case when compared to other requests.
> 
> For the query regarding "All requests", all requests do not disappear. More importantly,
sometimes all requests  reach the application when I POST same set of requests. To give a
rough picture, 1-2 requests fail in a set of 45-50 requests and this behaviour varies [The
request which failed in my one test cycle succeeds in another cycle].
> 
> I could see this in Tomcat 7.0 and Tomcat 7.0.42.
> 
> Here is the server.xml details:
> 
> <?xml version='1.0' encoding='utf-8'?>
> <!--
>   Licensed to the Apache Software Foundation (ASF) under one or more
>   contributor license agreements.  See the NOTICE file distributed with
>   this work for additional information regarding copyright ownership.
>   The ASF licenses this file to You under the Apache License, Version 2.0
>   (the "License"); you may not use this file except in compliance with
>   the License.  You may obtain a copy of the License at
> 
>       http://www.apache.org/licenses/LICENSE-2.0
> 
>   Unless required by applicable law or agreed to in writing, software
>   distributed under the License is distributed on an "AS IS" BASIS,
>   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>   See the License for the specific language governing permissions and
>   limitations under the License.
> -->
> <!-- Note:  A "Server" is not itself a "Container", so you may not
>      define subcomponents such as "Valves" at this level.
>      Documentation at /docs/config/server.html
>  -->
> <Server port="8105" shutdown="SHUTDOWN">
>   <!-- Security listener. Documentation at /docs/config/listeners.html
>   <Listener className="org.apache.catalina.security.SecurityListener" />
>   -->
>   <!--APR library loader. Documentation at /docs/apr.html -->
>   <!---->
>   <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"
/>
> 
> 
>   <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html
-->
>   <Listener className="org.apache.catalina.core.JasperListener" />
>   <!-- Prevent memory leaks due to use of particular java/javax APIs-->
>   <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
>   <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
/>
>   <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
/>
> 
>   <!-- Global JNDI resources
>        Documentation at /docs/jndi-resources-howto.html
>   -->
>   <GlobalNamingResources>
>     <!-- Editable user database that can also be used by
>          UserDatabaseRealm to authenticate users
>     -->
>     <Resource name="UserDatabase" auth="Container"
>               type="org.apache.catalina.UserDatabase"
>               description="User database that can be updated and saved"
>               factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>               pathname="conf/tomcat-users.xml" />
>   </GlobalNamingResources>
> 
>   <!-- A "Service" is a collection of one or more "Connectors" that share
>        a single "Container" Note:  A "Service" is not itself a "Container",
>        so you may not define subcomponents such as "Valves" at this level.
>        Documentation at /docs/config/service.html
>    -->
>   <Service name="Catalina">
> 
>     <!--The connectors can use a shared executor, you can define one or more named
thread pools-->
>     <!--
>     <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>         maxThreads="150" minSpareThreads="4"/>
>     -->
> 
> 
>     <!-- A "Connector" represents an endpoint by which requests are received
>          and responses are returned. Documentation at :
>          Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
>          Java AJP  Connector: /docs/config/ajp.html
>          APR (HTTP/AJP) Connector: /docs/apr.html
>          Define a non-SSL HTTP/1.1 Connector on port 8080
>     -->
>  <!--  -->
>     <Connector port="8080" protocol="HTTP/1.1"
>                connectionTimeout="20000"
>                redirectPort="8443" allowTrace="true"/>
> 
> 
>     <!-- A "Connector" using the shared thread pool-->
>     <!--
>     <Connector executor="tomcatThreadPool"
>                port="8080" protocol="HTTP/1.1"
>                connectionTimeout="20000"
>                redirectPort="8443" />
>     -->
> 
> <!-- Define an AJP 1.3 Connector on port 8009 -->
>     <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
> 
> 
>     <!-- An Engine represents the entry point (within Catalina) that processes
>          every request.  The Engine implementation for Tomcat stand alone
>          analyzes the HTTP headers included with the request, and passes them
>          on to the appropriate Host (virtual host).
>          Documentation at /docs/config/engine.html -->
> 
>     <!-- You should set jvmRoute to support load-balancing via AJP ie :
>     <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
>     -->
>     <Engine name="Catalina" defaultHost="localhost">
> 
>       <!--For clustering, please take a look at documentation at:
>           /docs/cluster-howto.html  (simple how to)
>           /docs/config/cluster.html (reference documentation) -->
>       <!--
>       <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>       -->
> 
>       <!-- Use the LockOutRealm to prevent attempts to guess user passwords
>            via a brute-force attack -->
>       <Realm className="org.apache.catalina.realm.LockOutRealm">
>         <!-- This Realm uses the UserDatabase configured in the global JNDI
>              resources under the key "UserDatabase".  Any edits
>              that are performed against this UserDatabase are immediately
>              available for use by the Realm.  -->
>         <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>                resourceName="UserDatabase"/>
>       </Realm>
> 
>       <Host name="localhost"  appBase="webapps"
>             unpackWARs="false" autoDeploy="true">
> 
>         <!-- SingleSignOn valve, share authentication between web applications
>              Documentation at: /docs/config/valve.html -->
>         <!--
>         <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
>         -->
> 
>         <!-- Access log processes all example.
>              Documentation at: /docs/config/valve.html
>              Note: The pattern used is equivalent to using pattern="common" -->
>         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
>                prefix="localhost_access_log." suffix=".txt"
>                pattern="%h %l %u %t &quot;%r&quot; %s %b" />
> 
>       </Host>
>     </Engine>
>   </Service>
> </Server>
> 

Hi.
There is nothing in the above server.xml that strikes me as particularly remarkable or wrong.

I do not pretend to know your system, nor your application, nor that the following is a 
definite explanation.  But on the base of the currently available data, I would say :
- it is quite unlikely that Tomcat 7 is randomly "dropping requests".  If it was, then I 
would imagine that this list would be overflowing with cries for help.
There is quite a bit of traffic on this list related to Tomcat 7, but I don't recall 
seeing any significant number of issues mentioning "dropped requests".
- it also doesn't seem, from your wireshark-related observations, that the requests are 
being lost outside of Tomcat.
- so I would say at this point, that the most likely place for requests to disappear is in

your own application.

I am far from being an expert in that area, but I remember seeing multiple threads in this

list that tended to indicate that this kind of thing can happen if you keep improper 
references to Request/Response objects in your applications.

Maybe someone else more qualified could take over at this point, to give you tips about 
how you could diagnose this kind of thing.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message