logging-log4net-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon E. Devlin" <simon.dev...@7layerforensics.com>
Subject RE: Multi-app centralized logging vs. per app logging and background process consolidation
Date Sun, 10 Feb 2008 03:03:17 GMT
Log4net isn't reliable -it's a best efforts delivery.  If you want
certain message delivery (or at least, certain notification of delivery
failure) you'll need to hand off to some other message based middleware
such as MSMQ.  Having said that, some appenders are more reliable than
others.  You'd know for example that an ADOAppender write failed.  A
remote syslog message could potentially get lost anywhere in the network
as there's no remote acknowledgement despite successfully 'delivery' by
the local host.


I'm sure you've already considered it, but 


a)      How much are you needing to log?  Minimal performance impact is
always on people's list, but in absolute terms it's often a function of
log volume.  Maybe one method takes 23ms longer than another, but if
you're logging 6 messages an hour, does it really matter?  If there's
heaps to log, is it all necessary?  We adopted a never log debug's
remotely policy.

b)      Network traffic.  A decentralized system is going to increase
traffic - it's a fact of life.  And, whether you use a local text file
then sync to a master through a timed copy or deliver immediately to a
database via ADOAppender or a  central Syslog server (which is what we
do for many of the reasons in Dean's message), that data is going to
have to get there somehow.  Again, this is a function of message volume.
Most of the text based appenders could have payload compression added if
you have very low bandwidth links I should imagine.  People often
overestimate the network impact (we had a single box log 85GB in one day
recently due to poor config, network complained not at all.  Syslog
server on the otherhand...)

c)       I'm not sure if you have industry compliance requirements but
you might find logging to a central DB easier to sell.  Let it deal with
concurrency issues, timestamping  and the merge process.


Hope this helps.





From: Mike Baker [mailto:mike.baker@harmonicinc.com] 
Sent: Saturday, February 09, 2008 9:46 AM
To: Log4NET User
Subject: RE: Multi-app centralized logging vs. per app logging and
background process consolidation


Thanks, Dean, for your ideas.  Unfortunately, one of the constraints on
our logging solution is to minimize additional network traffic.
OutputDebugString( ) is an elegant and efficient single-pc solution, as
it uses shared memory (see


I am not sure I agree that the RollingFileAppender approach is so bad.
Typically the logs are write-only (nobody has need to look at them), and
if they do have need examination, there is a specific time window of
interest (just before/after some anomaly).  An alternative to having a
background process to merging is to instead do it on-demand.  This
approach would involving writing a simple merge app with GUI that allows
the user to specify a time range (and perhaps select which log files) of
interest to merge.  


The other two goals of our logging solution cause minimal performance
hit when logging and is reliable. 


Any other thoughts on this?


Thanks again,






From: Dean Fiala [mailto:dfiala@celadonlabs.com] 
Sent: Thursday, February 07, 2008 9:44 PM
To: log4net-user@logging.apache.org
Subject: re: Multi-app centralized logging vs. per app logging and
background process consolidation


The RollingFile merge idea sounds like a lot of extra work and overhead
for little obvious gain.

The Publish-Subscribe sounds like a much better direction, but if you
are putting everything in a central repository, then there is only one
subscriber and there is no need for the overhead of who wants what.
Just publish everything to the central place.  

A couple of 2x4 solutions that require little more than configuration...
1) AdoNetAppenders to to log everything to a centralized database 
2) SmtpAppenders to log everything to a single email account

The advantage of the database is everything is neatly stored in typed
columns, timestamped, and easily sortable and analyzed. Triggers and
alerts will let you do some fancy things if needed. Lots of built in
tools for back-ups, archiving, etc. The disadvantage is that it might
not work with your database/server. The application logging load might
overwhelm the database. Probably not the best answer if the systems are
widely distributed (ie not on the same LAN). You'll probably also want
to build an app to look at the log info in, so there will be some build

The Smtp option isn't as flexible, but gosh it's cheap and easy and
works not matter how widely flung the servers are.  Throw Outlook (or
Outlook Express) on a box and your have a centralized logging client,
with a modicum of search and sorting capability already built in.  No UI
to build. Heck you could even just use a web mail client. Of course, if
you need to send each log event a single message you might make your
mail server (or administrator) angry, and parsing through the data will
require an export and another app.  You might also have delivery issues
that throw off time synching. 

Another 2x4 solution is to use LocalSysLogAppender/RemoteSyslogAppender
to ultimately get the messages to a single server's syslog service.
There are some free/cheap client tools for working with the syslog
service.  Everything comes in a known log format. There might be some
voodoo involved in getting the syslog services all talking together
and/or using the RemoteSyslogAppender.

If you need to get fancier, an approach based on the RemotingAppender
could work.  You'd need to build a Remoting server application (hosted
as a Windows Service or on  IIS), and a client app to query it.  If
necessary you'll also need to figure out how to persist the log
information and build the appropriate goodies to handle all that.  This
would work on a locally or widely distributed set of servers.  One
possible issue is that buffering could throw off the time synching .  

If you have the infrastructure, you might want to check out the
MSMQAppender.  Would give you very robust message delivery. You'd have
to build similar helper apps as you would in the Remoting solution.

Fun problem to think about.  Good luck.


Dean Fiala


A previous generation application suite simply uses OutputDebugString( )
to achieve centralized logging for a multi-app suite running on a single
host. The next generation replacement is in the design stage, and will
be distributed. It is desired to maintain an efficient and reliable
means of centralized logging.

One idea is some sort of publish-subscribe model, whereby each app
writes to a publisher, with the publisher responsible for delivering the
messages to each subscriber.

Another idea is to have each app use a RollingFileAppender, and have a
background process examine the time tags to assemble a centralized log
from the individual files. Obviously time-syncing will be necessary
between hosts, and a means to determine any offsets. 

Both of these approaches may involve more work than we would like to
budget for, and may have inherent time-order ambiguity for messages that
are close together due to OS context switching. 

Any comments on these approaches? Any other ideas on how better to do


View raw message