logging-log4j-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bruno Melloni" <Bruno.Mell...@wnco.com>
Subject RE: network logging performance
Date Thu, 04 Sep 2008 13:23:04 GMT
**Theoretically**, a logging event should be a 'fire and forget'
activity from the perspective of the application.  That means that you
should be able to submit the logging event to the 'logging system'
without blocking, and then the 'logging system' should be able to
deliver those logging events to the final destination on its own
thread(s) and at its own speed.  As long as the average volume of
logging does not exceed the capacity of the actual logging mechanism
(i.e.: over the network) then there ought to be no impact whatsoever to
the application (except perhaps for the cpu overhead of the physical
logging thread(s)).

I don't know log4j well enough to say whether it is already doing this
isolation of 'log event submission' vs 'log event processing'.  If it
doesn't, you should be able to have your application submit all logging
requests through a class written by you that puts all requests into a
memory cache, ensuring no blocking.  Then have one or more separate
threads handling the 'log event processing' independently from the rest
of the app.  For safety, you should probably add some code that monitors
memory, disk and network utilization by this approach and notify you
(perhaps by email) when things get backed up - that way you'll detect
any volume-caused issues before they have a chance to impact the app.

If you really wanted to be bulletproof, implement the cache to use
memory first (up to some limit) and then overflow to a local file cache.
That way if your network gets bogged down by issues other than your
app's you would still be able to continue functioning while the network
recovers.

I know that did not answer your immediate question, but it should help
some in making your choices.

bruno

-----Original Message-----
From: Paul Smith [mailto:psmith@aconex.com] 
Sent: Wednesday, September 03, 2008 8:31 PM
To: Log4J Users List
Subject: network logging performance

If anyone out there is using "Logging over the network" of any form
(socket, JMS, Multicast, syslog appenders etc), this topic is for you.  
I'm wondering whether people could comment on their experience setting
up high performance logging of application data over the network.  The
cost of shipping an event over the wire compared with writing it to a
local file is obviously higher, and so one can notice user-visible
impact if the logging volume is high when using network-based logging
(unless one wraps it in AsyncAppenders).

Just curious to hear peoples experiences, strategies, and thoughts.   
Perhaps people can relate to the flow rate they've manage to achieve
under different configurations.

The driver to my question is that for my Apache Lab project (Pinpoint)
I'm building a central logging repository server to allow data mining of
the generated logging events.  In a high performance site one can have
many logging events shipped from multiple threads in, say, a web- app,
being shipped serially over the wire, so slowing down the logging can
slow down the response time.

cheers,

Paul Smith

---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org


Mime
View raw message