Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <48598F1A.4010202@ice-sa.com>
Date: Thu, 19 Jun 2008 00:41:30 +0200
From: =?UTF-8?B?QW5kcsOpIFdhcm5pZXI=?= <aw@ice-sa.com>
User-Agent: Thunderbird 1.5 (Windows/20051201)
MIME-Version: 1.0
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: Logging for Dummies in Tomcat 5.5/6.0
References: <4857D96A.7090200@ice-sa.com>
 <485973A5.5010105@christopherschultz.net>
In-Reply-To: <485973A5.5010105@christopherschultz.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Chris,

don't give up yet, please.
There is a single question here.

While I still do not agree with some of your comments below, I *do* 
immensely appreciate the time you put into it, the civility with which 
you answered, and the clues it provides.
I also do understand that it is not easy to try to help someone, when 
you do not even know where things are, or how much someone may have 
mangled it.

My idea was, maybe naively, that Tomcat was like any other program, 
fairly logical, and that things in it happen for a reason.
Thus that when Tomcat starts, it knows, from some top-level 
configuration file, where to look for instructions as to what logging 
system to use, and has an idea about where the configuration for that 
stuff is.
And that this knowledge could be communicated to me somehow without 
breaking some official secrecy vow.

What really puzzled me however, was that going down the hierarchy of 
configuration files and directories, I never seemed to find a link 
between Tomcat and the logging it was doing.

I am starting to see the error of my ways.
What I am, ever so slowly, starting to think I understand (I hope), is 
that Tomcat /may/ not itself know; that your refusal to tell me may not 
after all have been motivated by a desire to keep the knowledge into 
your inner circle of initiates; but that instead, it is some other piece 
of software that "hooks" into Tomcat to "steal" the things to be logged, 
and that this other piece of thing is the (only) one that knows it's own 
configuration.
And thus that for instance Tomcat itself knows nothing about that file
/var/lib/tomcat5.5/conf/logging.properties
which I discovered long ago on my Linux Debian Etch system, but could 
not figure out how Tomcat found it or used it.

And this file contains stuff like

handlers = 1catalina.org.apache.juli.FileHandler, 
2localhost.org.apache.juli.FileHandler, 
3manager.org.apache.juli.FileHandler, 
4admin.org.apache.juli.FileHandler, 
5host-manager.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler

(the above being one single line), and to follow,

.handlers = 1catalina.org.apache.juli.FileHandler, 
java.util.logging.ConsoleHandler

############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################

1catalina.org.apache.juli.FileHandler.level = FINE
1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.FileHandler.prefix = catalina.

2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.FileHandler.prefix = localhost.

.. continued with similar stuff apparently related to the other apps.

I would thus hasard a wild guess that the logging mechanism used is 
"juli", rather than "log4j".
And that thus the bottom of the page at
http://tomcat.apache.org/tomcat-5.5-doc/logging.html
might be the applicable one for this case.

In that case, I would certainly not break it by attempting an uninformed 
switch to log4j, and would instead start twiddling with what is there.

However, my initial joy at finally discovering an initial thread to hang 
on to, gets severely dampened when I subsequently discover the following 
paragraph, which, would you believe it, reads to me like some highly 
sophisticated PSK code to which I do not have the key :

     *  A prefix may be added to handler names, so that multiple 
handlers of a single class may be instantiated. A prefix is a String 
which starts with a digit, and ends with '.'. For example, 22foobar. is 
a valid prefix.
     * As in Java 5.0, loggers can define a list of handlers using the 
loggerName.handlers property.
     * By default, loggers will not delegate to their parent if they 
have associated handlers. This may be changed per logger using the 
loggerName.useParentHandlers property, which accepts a boolean value.
     * The root logger can define its set of handlers using a .handlers 
property.
     * System property replacement for property values which start with 
${sytstemPropertyName}.

I mean, I do understand the part "A prefix is a String which starts with 
a digit, and ends with '.'. For example, 22foobar. is a valid prefix.", 
but the rest is a bit mysterious, and I am worried that I might have to 
understand it, before I start deleting lines and such.
Or not ?

Now, for example, if I were to delete all the references, in the 
logging.properties above, to "1catalina.org.apache.juli.FileHandler", 
would that then simply get rid of all the daily logfiles of the form
/var/log/tomcat5.5/catalina.yyyy-mm-dd.log

, or would it bring down my whole Tomcat server ?


I will venture another couple of guesses, even :

- that the link between the highest level of the Tomcat server (catalina 
  ?) and the specific files "catalina.yyyy-mm-dd.log", is made via the 
line above starting with ".handler", and that this is the very "root 
logger" mentioned in the ante-last line of the mysterious paragraph above.

- that the lines above starting with 
"2localhost.org.apache.juli.FileHandler" relate somehow to this tag and 
level in the "server.xml" :

       <Host name="localhost" appBase="webapps"
        unpackWARs="true" autoDeploy="true"
        xmlValidation="false" xmlNamespaceAware="false">

But, in that case, what is not clear to me is how the actual link is 
made.  That is because I find nothing within this <Host> section that 
would provide a link with "2localhost.org.apache.juli.FileHandler".
(There is actually nothing but comments between the <Host> above and 
</Host>)

- Or (another guess), if I added another virtual host introduced as such :

       <Host name="secondhost" appBase="webapps2"
        unpackWARs="true" autoDeploy="true"
        xmlValidation="false" xmlNamespaceAware="false">
	</Host>

and I added new references in the logging.properties file, e.g. to 
"7secondhost.org.apache.juli.FileHandler", would it then magically start 
creating logfiles like
/var/log/tomcat5.5/secondhost.yyyy-mm-dd.log
?

- or, if I deleted the references in logging.properties to 
"2localhost.org.apache.juli.FileHandler",
would that then result in :
- the files "localhost.yyyy-mm-dd.log" not being produced anymore
- but the corresponding logging entries would now be automatically 
intercepted by the higher-level "root logger", thus ending up in 
"catalina.yyyy-mm-dd.log" ?

(at this point, nothing would surprise me)

Of course I may be totally wrong with my guesses above, and become 
totally desperate as Chuck next shoots me down.
And there are still a lot of things I do not understand at all (such as 
what role the java.util.logging.ConsoleHandler gimmick plays in all 
this), but one thing at a time.


André

P.S.
As an aside, and without contradicting what I have been writing before, 
I must admit that it looks like the Debian packager of Tomcat 5.5 was 
particulary imaginative in terms of spreading things around, and in 
terms of symbolic links between directories and stuff.
Way above the Apache guys level anyway.
I will not pass judgement before he has a chance to explain however, 
because hey, the stuff works.  I installed it in 2 minutes flat, 
including download.  Tomcat runs fine, one can add and remove webapps 
and it still runs.  It's just that I want to change the way some 
logfiles are being created, and that doesn't seem easy to get started with.


Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> André,
> 
> Okay, I'll bite.
> 
> André Warnier wrote:
> | Assume that one has installed Tomcat 5.5 (or 6.0) on some system just by
> | using the standard software package management system for his Operating
> | System, and consequently finds himself with some Tomcat installation,
> | whose settings and layout have been chosen by a real guru (in both
> | Tomcat and in software packaging).
> 
> Note that everything starts to fall apart right about .... here. ^^^
> 
> | Assume that the Tomcat in question works fine, but that it writes
> | logfiles all over the place (or all over time), and that one would like
> | to understand where these logfiles come from, and either slightly change
> | which logfiles are being produced, or add a specific logfile for a
> | specific application, or something simple like that.
> 
> Okay. Sounds good.
> 
> | Assume that one has read the Tomcat logging page at
> | http://tomcat.apache.org/tomcat-6.0-doc/logging.html
> 
> Although the this is a good reference, you cannot follow any of the
> instructions on that page, because the logging has likely been set up in
> advance by the package manager. Since there is no standard way to
> configure logging (among package managers, that is), nobody on this list
> who doesn't have explicit experience with your
> OS/version/patchlevel/package manager can help you :(
> 
> | The kind of things one would like to know are :
> |
> | - where to start ?
> | In other words, here I have a Tomcat and it is working and it is writing
> | logfiles, but I do not have a clue which kind of logging mechanism it is
> | using, either directly or indirectly.  How do I find out ?
> 
> What would be great is if you could, say, identify a log file (say,
> catalina.out) and then say "where did this come from"? The answer is
> easy, if you use the standard package from tomcat.apache.org:
> 
> $ cd $TOMCAT_HOME
> $ grep catalina.out `find -type f`
> 
> If I run this command on my stock TC 5.5 install, I get these results:
> ./bin/catalina.sh
> ./webapps/jsp-examples/WEB-INF/classes/validators/DebugValidator.java
> ./webapps/tomcat-docs/printer/jndi-datasource-examples-howto.html
> ./webapps/tomcat-docs/printer/setup.html
> ./webapps/tomcat-docs/jndi-datasource-examples-howto.html
> ./webapps/tomcat-docs/setup.html
> 
> Ignoring the documentation and code results, I can see that
> bin/catalina.sh mentions catalina.out. Let's take a look, shall we?
> 
> Reading catalina.sh, you'll find that this line appears in several
> places in the script:
> 
> ~      >> "$CATALINA_BASE"/logs/catalina.out 2>&1 &
> 
> after certain commands. This indicates that catalina.out is redirected
> standard output.
> 
> Just got grins, let's come up with another one. I've got a file called
> localhost.2008-06-18.log in my "logs" directory. How'd it get there?
> Lessee...
> 
> $ grep -l 'localhost.*\.log' `find -type f `
> conf/logging.properties
> 
> Hey! Look at that! A file just for setting up logging properties? That's
> crazy!
> 
> ...rinse, repeat.
> 
> Unfortunately, if you installed your TC from a package manager, you
> probably don't have all the TC-related files in a single place. That
> wouldn't be such a big deal if it didn't take quite a while to grep
> every file on your system:
> 
> $ grep catalina.out `find / -type f`
> 
> ... snore ....
> 
> TC and package management gurus indeed.
> 
> The problem is that the UNIX philosophy has always been "there's one
> binary, a config directory somewhere under /etc, and the logs go under
> /var/log". It's bad enough that we have to have dozens of config files
> and deployment directories for Apache httpd... TC makes it even worse
> because it's more application-based than directory-based. ClassLoaders
> and system properties and endorsed libraries, oh my! Shoehorning TC into
> a semi-typical Linux (or UNIX) package management scheme often leads to
> confusion with people who read official documentation that suggests that
> the world is otherwise.
> 
> And we (on this list) get the fallout, instead of the wankers from the
> distros who muck everything up in the first place.
> 
> ...but I digress. Let's move on.
> 
> | - how does it work ?
> | In other words : it would seem that the kind of logging adopted in
> | Tomcat 5.5/6.0 is very powerful and flexible, allowing one to decide "at
> | the top" which mechanism is being used, and then define either some
> | overall generic logging settings for the whole Tomcat and valid for all
> | components and applications, and/or refine this at just about each
> | hierarchical level of Tomcat, Engine, Connector, Host, application and
> | whatnot, at whatever level of detail one needs between CRASH and CHATTY.
> |  Great.
> 
> Yup. See the aforementioned logging.properties file. And read the TC
> logging documentation, which you assumed was read. If you look through
> the logging.properties file, you may gain some insight into how the
> output from each level/application/host/etc is routed into separate files.
> 
> | - how does one set up a really simple logging configuration, but one
> | that will allow in the future some gradual tailoring and refinement
> | without complete redesign ?
> 
> Use the one that came configured with your TC installation. Locate the
> logging.properties file, and "simplify" it, whatever that means. Take
> out loggers you don't want. Run everything into /var/log/all-tomcat.log.
> Do whatever you want. Examples come with the stock TC install from
> tomcat.apache.org. Your distro doesn't? Hmm. Complain to them. Our
> documentation is short, and following the instructions ends up working.
> 
> | I would like to have, for the whole of Tomcat :
> | - one monthly file that shows the equivalent of an Apache "error log"
> 
> I don't see a way to make juli rotate logfiles on a different schedule
> than once per day. I /do/ know that log4j can do it by using a
> DailyRollingFileAppender (with rolling set to 'monthly'). If I were
> doing this myself, I would follow the instructions on TC's logging page
> to switch from using juli to using log4j, and then configure log4j to
> emit the log you describe above. This is easier for me, because I have
> much more familiarity with log4j than with juli/jdk-logging/whatever. If
> I were adamant about using juli, I would probably read all about how to
> configure java 1.4 logging.
> 
> | - one monthly file that shows the equivalent of an Apache "access log"
> 
> This is done using an AccessLogValve, which is somewhat orthogonal to
> the issue of "logging" as it is being covered, here. This has been
> pretty consistent across TC versions.
> 
> | b) when the above is achieved, then I'd like to re-introduce a separate
> | access log for each Host.  How do I do that ?
> 
> Different <Valve> under each host.
> http://www.google.com/search?q=tomcat%20access%20log
> 
> | c) when the above is achieved, then I'd like to re-introduce a separate
> | log for just the "MyApp" application.  How do I do that ?
> 
> That depends. Do you want to just log the Tomcat-specific log messages,
> or those from your own application? I'm guessing that, given yout lack
> of familiarity of logging systems in webapps, you are not using
> commons-logging in your own web application. That pretty much means that
> you're not going to get everything into one log file unless you can
> convince TC to hook into your application-specific logging system.
> 
> If you just want the former, and you've already switched to log4j (as I
> suggest above), you'll need to add further configuration to log4j.
> You'll need an appender (which writes log messages to something --
> file/xmtp/whatever) and you will need to identify the logger that dumps
> to that appender.
> 
> Given the examples provided in logging.properties, I'd do something like
> this and see if it works:
> 
> log4j.appender.MyApp=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.MyApp.file=/path/to/your/logs/MyApp.log
> # Set DRFA to roll on a monthly basis
> log4j.appender.MyApp.DatePattern='.'yyyy-MM
> log4j.appender.MyApp.layout = org.apache.log4j.PatternLayout
> log4j.appender.MyApp.layout.conversionPattern= %d [%t] %-5p %c- %m%n
> # Do not overwrite files
> log4j.appender.MyApp.append=true
> 
> # Configure this 'category' (logger) to go to MyApp's logfile
> log4j.category.org.apache.catalina.code.ContainerBase.[Catalina].[localhost].[/MyApp]=INFO,MyApp 
> 
> 
> | And no, the page at http://tomcat.apache.org/tomcat-6.0-doc/logging.html
> | is not enough to answer the above.  It assumes from the reader a greater
> | knowledge of Java and Tomcat than most Tomcat users have, and it
> | presupposes, I guess, that the installed Tomcat has been built
> | on-the-spot from the original Tomcat distribution
> 
> How can we document a distribution we not only do not distribute, but
> have no control over?
> 
> | and I believe that this is not the case for the majority of
> subscribers to this "Tomcat
> | users" list.
> 
> The majority of posts on this list either do not mention their use of
> package management, or end up being solved by removing the
> package-managed TC installation and starting from scratch (and then
> following the correct documentation).
> 
> Okay, that's the last thing I'm saying about logging for a while without
> getting paid. :p
> 
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iEYEARECAAYFAkhZc6UACgkQ9CaO5/Lv0PAVlwCeJ8gYvMaE3nVmpZdrYMDnMuTF
> dmYAn1akmjKdv2hhBDrxiNihKhQeCvY3
> =ChqB
> -----END PGP SIGNATURE-----
> 
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org