Mailing-List: contact dev-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Developers List" <dev@tomcat.apache.org>
Received-SPF: pass (nike.apache.org: local policy)
Date: Wed, 6 Jan 2010 15:28:47 +1300 (NZDT)
From: Tim Whittington <tim.whittington@orionhealth.com>
To: Tomcat Developers List <dev@tomcat.apache.org>
Message-ID: <1782699379.6391262744922311.JavaMail.timw@duckbook>
In-Reply-To: <4B2B6F31.1040003@kippdata.de>
Subject: Re: [PROPOSAL] Log rotation in Tomcat Connector
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_541_1560088686.1262744922310"

------=_Part_541_1560088686.1262744922310
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

For consistency (with Apache at least) I've gone down the road of mirroring the functionality of rotatelogs, which was fairly easy. 

I'm doing some long running tests (I'm seeing some weird DLL unloading/crashing behaviour in IIS 5.1 that I want to replicate and account for), but if that works out I'll port the changes to trunk (I'm developing on JK_1_2_28 at the moment for some reason) and submit a patch. 

The downside of the rotatelogs approach is that it's not easy to do log pruning (since the filenames are based on the time the log rotates, rather than some kind of backup suffix scheme), so I may do nothing about that for now - the best I could do is keep a list in memory and prune, which is sub-optimal. 

cheers 
tim 
----- Original Message ----- 
From: "Rainer Jung" <rainer.jung@kippdata.de> 
To: "Tomcat Developers List" <dev@tomcat.apache.org> 
Sent: Saturday, 19 December, 2009 1:01:53 AM GMT +12:00 New Zealand 
Subject: Re: [PROPOSAL] Log rotation in Tomcat Connector 

Hi Tim, 

On 18.12.2009 03:26, Tim Whittington wrote: 
> Hi all 
> 
> We're experiencing issues with the Tomcat Connector log in some IIS production sites where the log file grows to a very large size (8GB on one site). 
> This is almost entirely due to connection errors between the front end and back end produced when the back-end Tomcat is restarted after crashes or for maintenance. 
> (we have some very high request volume sites - some with hundreds of concurrent requests -, so you can get thousands of these errors on each restart). 
> 
> I'm thinking about implementing log file rotation in the IIS connector as a solution to this, and wanted to canvas opinions on alternatives and implementation first. 

Good thing, that was open for a long time for the isapi redirector. 

> What I intend implementing is: 
> * Daily log file rolling (not configurable) as a clone of the log_to_file function (check rotation, close and reopen log file, continue) 
> * Very simple configuration for a daily rolling file appender - basically 'on/off' and 'number of days to keep', possibly a single rotateable=bool/int parameter. 
> * The log file name suffix will be appended/inserted to the name of the file specified in the connector config with a static pattern e.g. YYYY_MM_dd 
> * The log method(s) will be guarded with a mutex to protect the rollover operation 
> * I'll only be doing this for the ISAPI redirector at present (so won't need to refactor jk_mt.h to make the locking primitives available on prefork builds) 
> * On rollover, the file will be closed and a new one opened. Older files will be pruned. 
> * Rollover checking can be a simple mod operation on the timestamp (since I'm not intending to handle log formats) 
> 
> Questions I have: 
> * Short of silencing the logging, is there an alternative to rolling the file? (perhaps rolling up of repeated connection errors?) 
> * Are there any performance concerns about using a mutex on each log? (log4c uses this approach, so I presume it's not prohibitive) 
> * Does anyone have any other input into config/behaviour? 

Mutex 
===== 

- log rotation for mod_jk with Apache is done using an external process 
and logging through a pipe. The external process often used with Apache 
is rotatelogs, that comes with apache and thus is also available under 
ASL 2.0 

- log path for Apache should not change seriously, e.g. no additional 
mutex when running mod_jk inside Apache. 

- for the redirector I'd say adding a mutex is OK, if it is acquired 
after the log line has been formatted and immediately before the line 
gets written to the file. So you'll only lock the access to the file. 

In apache-2.0/mod_jk.c you will find for example jk_log_to_file(), which 
does 

rv = apr_global_mutex_lock(jk_log_lock); 
if (rv != APR_SUCCESS) { 
... log error via Apache ... 
} 
rv = apr_file_write(p->jklogfp, what, &wrote); 
if (rv != APR_SUCCESS) { 
... log error via Apache ... 
} 
rv = apr_global_mutex_unlock(jk_log_lock); 
if (rv != APR_SUCCESS) { 
... log error via Apache ... 
} 

Rolling 
======= 

You could also think about doing the rolling externally like we do with 
Apache and pipes in case the user configured a pipe as the log file. I'm 
not sure about ISAPI and the IIS process management, whether this is 
feasable, e.g. all IIS processes sharing the file descriptor to the pipe 
etc. 

Apart from that I think having a simple daily solution, which can be 
activated or deactivated would already be a very useful start. We can 
always add more features, once the basics are there. Obvious possible 
extensions are more fine grained rolling intervals, alternatively basing 
the rolling on file size, flexible file name pattern, etc. 

Pruning 
======= 

I'm not a big fan of pruning the old files directly from the server 
process. I'd say pruning is part of some external log file management, 
which usually does compression, transferral to a log server and removing 
old files once they are old enough or have been relocated. 

Automatic pruning is nice for people to get a quick start withour caring 
about those details, but it should at least be possible to turn it off. 

Log message dedup 
================= 

Reducing the log volume is not easy to do. Of course we could keep track 
of state and not log problems we already noted, but then you would also 
like to log when the state changes to something better. That means 
comparing state also in case something worked correctly and will make 
our cluttered code even more difficult to understand. 

I remember Mladen once mentioning, that he also has some ideas or 
possibly code for log rotation in the isapi redirector, so he might want 
to chime in as well. 

Thanks for taking the initiative! 

Regards, 

Rainer 

--------------------------------------------------------------------- 
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org 
For additional commands, e-mail: dev-help@tomcat.apache.org 


------=_Part_541_1560088686.1262744922310--