Mailing-List: contact dev-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Developers List" <dev@tomcat.apache.org>
Received-SPF: pass (athena.apache.org: domain of ian@darwinsys.com designates
 74.220.163.16 as permitted sender)
Message-ID: <4D628433.4090809@darwinsys.com>
Date: Mon, 21 Feb 2011 10:26:43 -0500
From: Ian Darwin <ian@darwinsys.com>
User-Agent: Mozilla/5.0 (X11; U; OpenBSD amd64; en-US;
 rv:1.9.2.13) Gecko/20110108 Thunderbird/3.1.7
MIME-Version: 1.0
To: dev@tomcat.apache.org
Subject: Re: Reaching ulimit values for open files can generate huge log files
References: <4D6282FC.2090602@apache.org>
In-Reply-To: <4D6282FC.2090602@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 02/21/11 10:21, Mark Thomas wrote:
> The ASF Sonar installation managed to generate 46GB of identical log
> messages [1] today in the 8 hours it took to notice it was down.

Continuing to drive down the cost of disk storage :-)

> While better monitoring would/should have identified the problem sooner,
> it does demonstrate a problem with the acceptor threads in all three
> endpoints. If there is a system-level issue that causes the accept()
> call to always fail (such as hitting the ulimit) then the endpoint
> essentially enters a loop where it logs an error message for every
> iteration of the loop. This will result in many log messages per second.
> 
> I'd like to do something about this. I was thinking of something along
> the lines of the following for each endpoint.
> 
> Index: java/org/apache/tomcat/util/net/JIoEndpoint.java
> ===================================================================
> --- java/org/apache/tomcat/util/net/JIoEndpoint.java	(revision 1072939)
> +++ java/org/apache/tomcat/util/net/JIoEndpoint.java	(working copy)
> @@ -183,9 +183,19 @@
>          @Override
>          public void run() {
> 
> +            int errorDelay = 0;
> +
>              // Loop until we receive a shutdown command
>              while (running) {
> 
> +                if (errorDelay > 0) {
> +                    try {
> +                        Thread.sleep(errorDelay);
> +                    } catch (InterruptedException e) {
> +                        // Ignore
> +                    }
> +                }
> +
>                  // Loop if endpoint is paused
>                  while (paused && running) {
>                      try {
> @@ -225,9 +235,15 @@
>                              // Ignore
>                          }
>                      }
> +                    errorDelay = 0;
>                  } catch (IOException x) {
>                      if (running) {
>                          log.error(sm.getString("endpoint.accept.fail"), x);
> +                        if (errorDelay == 0) {
> +                            errorDelay = 50;
> +                        } else if (errorDelay < 1600) {
> +                            errorDelay = errorDelay * 2;
> +                        }
>                      }
>                  } catch (NullPointerException npe) {
>                      if (running) {
> 
> 
> 
> Thoughts / comments?

+1 - a bit of smarts in reducing redundant logging is usually a good thing.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org