Mailing-List: contact dev-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Developers List" <dev@tomcat.apache.org>
Message-ID: <4D6415B3.8070709@apache.org>
Date: Tue, 22 Feb 2011 19:59:47 +0000
From: Mark Thomas <markt@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB;
 rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7
MIME-Version: 1.0
To: Tomcat Developers List <dev@tomcat.apache.org>
Subject: Re: Reaching ulimit values for open files can generate huge log files
References: <4D6282FC.2090602@apache.org> <4D640FBA.5000806@hanik.com>
In-Reply-To: <4D640FBA.5000806@hanik.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 22/02/2011 19:34, Filip Hanik - Dev Lists wrote:
> Wouldn't you build this into the logging framework, instead of one
> specific component?

You could, if you can find an efficient way to spot duplicate log
messages and then differentiate correctly between when you want the
duplicates and when you don't. I'm not sure how easy that would be.
Suggestions welcome.

The other issue with the current code is that it is likely to chew up a
large chunk of CPU time. Fixing the log mechanism won't fix that.

Of course, the real solution is fix the ulimit issue in the first place
but Tomcat's behaviour in this error condition seems sufficiently bad to
justify some changes to handle it more gracefully.

Mark

> 
> best
> Filip
> 
> On 02/21/2011 08:21 AM, Mark Thomas wrote:
>> The ASF Sonar installation managed to generate 46GB of identical log
>> messages [1] today in the 8 hours it took to notice it was down.
>>
>> While better monitoring would/should have identified the problem sooner,
>> it does demonstrate a problem with the acceptor threads in all three
>> endpoints. If there is a system-level issue that causes the accept()
>> call to always fail (such as hitting the ulimit) then the endpoint
>> essentially enters a loop where it logs an error message for every
>> iteration of the loop. This will result in many log messages per second.
>>
>> I'd like to do something about this. I was thinking of something along
>> the lines of the following for each endpoint.
>>
>> Index: java/org/apache/tomcat/util/net/JIoEndpoint.java
>> ===================================================================
>> --- java/org/apache/tomcat/util/net/JIoEndpoint.java    (revision
>> 1072939)
>> +++ java/org/apache/tomcat/util/net/JIoEndpoint.java    (working copy)
>> @@ -183,9 +183,19 @@
>>           @Override
>>           public void run() {
>>
>> +            int errorDelay = 0;
>> +
>>               // Loop until we receive a shutdown command
>>               while (running) {
>>
>> +                if (errorDelay>  0) {
>> +                    try {
>> +                        Thread.sleep(errorDelay);
>> +                    } catch (InterruptedException e) {
>> +                        // Ignore
>> +                    }
>> +                }
>> +
>>                   // Loop if endpoint is paused
>>                   while (paused&&  running) {
>>                       try {
>> @@ -225,9 +235,15 @@
>>                               // Ignore
>>                           }
>>                       }
>> +                    errorDelay = 0;
>>                   } catch (IOException x) {
>>                       if (running) {
>>                          
>> log.error(sm.getString("endpoint.accept.fail"), x);
>> +                        if (errorDelay == 0) {
>> +                            errorDelay = 50;
>> +                        } else if (errorDelay<  1600) {
>> +                            errorDelay = errorDelay * 2;
>> +                        }
>>                       }
>>                   } catch (NullPointerException npe) {
>>                       if (running) {
>>
>>
>>
>> Thoughts / comments?
>>
>> Mark
>>
>>
>> [1] http://pastebin.com/CrsujeW4
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>
>>
>>    
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org