logging-log4net-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Parrish, Ken" <KParr...@gomez.com>
Subject RE: Throttling/Rollup feature in EventLogAppender
Date Sun, 04 May 2008 21:38:16 GMT
Pranav,

 

I recently posted a code snippet for a throttled / caching SMTP
appender.  I've copied it here.  It might serve as the basis for what
you need:

 

Ken Parrish

Gomez, Inc.

__________________________

 

Ole,

 

I recently solved a very similar problem.  I had a situation where in
the event of a serious error in our server infrastructure, I needed to
'throttle' the volume of e-mail messages sent to our operations team.
In our case, regardless of the number of serious error messages
generated, we wanted to limit to 1 e-mail per 5 minutes, and restrict
the size of each e-mail to the most recent 10 events plus some context
information (e.g. INFO log events).

 

I created a log4net extension which is an extension to the SMTPAppender
class named SMTPCachingAppender.  This appender has a number of control
parameters which establish it's behavior:  interval between flush, event
count between flush, whether or not to include context information,
maximum buffer size.

 

This appender might not exactly suit your needs, but it does provide a
skeleton for creating a similar appender that is fine-tuned for your
application.

 

Below is a copy of the source code and a sample configuration.

 

Hope this helps.

Ken Parrish

Gomez, Inc.

 

__________________________

 

Sample log4net configuration:

 

<!-- Caching SMTP appender for operational notifications. -->

<appender name="OperationsLogSmtpAppender"
type="log4net.Extensions.SmtpCachingAppender, Utilities">

      <to value="ops-notifications@myplace.com" />

      <from value=" ops-notifications@myplace.com " />

      <subject value="Buffered Operational Notification" />

      <smtpHost value="mySMTPServer.com" />

      <bufferSize value="0" />

      <lossy value="false" />

      <priority value="high" />

      <FlushInterval value="00:05:00" />

      <FlushCount value="0" />

      <MaxBufferSize value="10" />

      <layout type="log4net.Layout.PatternLayout">

            <param name="ConversionPattern" value="%date{yyyy-MM-dd
HH:mm:ss.fff} [%-2thread] [%-5level] [%logger]
%message%newline%exception" />

      </layout>

</appender>

 

__________________________

 

Source code:

 

 

using System;

using System.Collections.Generic;

 

using log4net.Appender;

using log4net.Core;

 

namespace log4net.Extensions

{

      /// <summary>

      ///         Caching version of the standard log4net SmtpAppender.
This appender will

      ///         cache log events that are to be sent out via Smtp and
send them in block.

      /// 

      ///         Configuration options:

      /// 

      ///         <FlushInterval value="hh:mm:ss" />

      ///               Indicates the periodic interval for log events
flushing (e.g. sending and e-mail), specified

      ///               as a time span.  If the value of FlushInterval
is 0:0:0 (zero), periodic interval flushing

      ///               is surpressed.  Default value:  0:5:0 (5
minutes).

      /// 

      ///         <FlushCount value="x" />

      ///               Indicates the number of log events received by
the appender which will trigger flushing

      ///               (e.g. sending and e-mail).  If the value
FlushCount is 0, buffer flush triggering based

      ///               the number of log events received is surpressed.
Default value:  20 events.

      /// 

      ///         <IncludeContextLogEvents value="[true/false]"/>"

      ///               True if context log events are included in the
buffered log event cache.  Cache events are

      ///               those that are of a lower level than that
specified for triggering the SmtpCachingAppender.

      ///               False if no context log events are included.
Default value:  true.

      /// 

      ///         <MaxBufferSize value="x"/>

      ///               Maximum number of log events, both
SmtpCachingAppender events and context events to save

      ///               in the cache buffer.  If more than MaxBufferSize
events are received before the flushing

      ///               criteria is met, only the newest MaxBufferSize
events are saved in the buffer.  A value

      ///               of 0 indicates no limit in the size of the
cache.  Default value:  0 (no limit)

      /// 

      ///         Sample SmtpCachingAppender configuration (in addition
to all standard SmtpAppender options).

      ///               Note the namespace and assembly name for the
appender.

      /// 

      ///         <appender name="SmtpCachingAppender"
type="log4net.Extensions.SmtpCachingAppender, Utilities">

      ///               . . .

      ///               <FlushInterval value="00:05:00" />

      ///               <FlushCount value="20" />

      ///               <IncludeContextLogEvents value="false"/>"

      ///               <MaxBufferSize value="3"/>

      ///         </appender>

      ///

      /// </summary>

      public class SmtpCachingAppender : SmtpAppender

      {

            // Configuration options

            int _flushCount = 20;

            TimeSpan _flushInterval = new TimeSpan(0, 5, 0);

            bool _includeContextLogEvents = true;

            int _maxBufferSize = 0;

 

            // Appender state data

            System.Threading.Timer _timer;

            bool _timedFlush = false;

            int _numCachedMessages = 0;

            List<LoggingEvent> _loggingEvents = new
List<LoggingEvent>();

 

            /// <summary>

            ///         TimeSpan indicating the maximum period of time
elapsed before sending

            ///         any cached SMTP log events.

            /// </summary>

            public TimeSpan FlushInterval

            {

                  get { return this._flushInterval; }

                  set { this._flushInterval = value; }

            }

 

            /// <summary>

            ///         Maximium number of SMTP events to cache before
sending via SMTP.

            /// </summary>

            public int FlushCount

            {

                  get { return this._flushCount; }

                  set { this._flushCount = value; }

            }

 

            public int MaxBufferSize

            {

                  get { return this._maxBufferSize; }

                  set { this._maxBufferSize = value; }

            }

 

            public bool IncludeContextLogEvents

            {

                  get { return this._includeContextLogEvents; }

                  set { this._includeContextLogEvents = value; }

            }

 

            /// <summary>

            ///         Create a timer that fires to force flushing
cached log events

            ///         via SMTP at a specified interval.

            /// </summary>

            public override void ActivateOptions()

            {

                  if (this._flushInterval > TimeSpan.Zero)

                  {

                        this._timer = new System.Threading.Timer(

                              new
System.Threading.TimerCallback(OnTimer),

                              null,

                              this._flushInterval,

                              this._flushInterval);

                  }

                  base.ActivateOptions();

            }

 

            void OnTimer(Object stateInfo)

            {

                  this._timedFlush = true;

                  Flush(true);

            }

 

            protected override void SendBuffer(LoggingEvent[] events)

            {

                  if (this._includeContextLogEvents)

                  {

                        this._loggingEvents.AddRange(events);

                  }

                  else

                  {

                        this._loggingEvents.Add(events[events.Length -
1]);

                  }

 

                  if (this._maxBufferSize != 0)

                  {

                        int numRemoved = this._loggingEvents.Count -
this._maxBufferSize;

                        if ((numRemoved > 0) && (numRemoved <=
this._loggingEvents.Count))

                        {

                              this._loggingEvents.RemoveRange(0,
numRemoved);

                        }

                  }

 

                  this._numCachedMessages++;

                  if (((this._flushCount != 0) &&
(this._numCachedMessages >= this._flushCount))

                        || this._timedFlush)

                  {

                        if (this._loggingEvents.Count > 0)

                        {

                              LoggingEvent[] bufferedEvents =
this._loggingEvents.ToArray();

 

                              base.SendBuffer(bufferedEvents);

 

                              this._loggingEvents.Clear();

                        }

                        // Reset cache buffer conditions.

                        this._numCachedMessages = 0;

                        this._timedFlush = false;

                  }

            }

      }

}

 

__________________________

 

-----Original Message-----
From: heinztomato [mailto:ole@nocarrier.de] 
Sent: Tuesday, April 22, 2008 5:32 AM
To: log4net-user@logging.apache.org
Subject: Counting number of Errors or Emails sent

 

 

Hi there,

 

I got a task that SHOULD be quite easy but I did not find any solution
yet.

Maybe the reason is that google finds a lot of stuff when searching for

count... (impossible to get a non-sql-related site :-D)

 

But to my problem:

log4net works fine, sending emails when an Error occurs. But sometimes
it

happens that because of a major bug thousands of Mails are sent because
of

thousands of error messages. That ist NOT very nice especially because
those

mails are sent to an recipientgroup: My collegues nearly cut my head off

last time they received 10.000 mails in a day B-)

 

So I want to create a Watchdog which stops my service (which sends those

mails) if a number of Errors had occured in a specific time (or as an

alternative if a number of Mails had been sent)

 

Any Hints how to achieve that goal? (A really good link would be
helpful,

too)

 

Thanks in advance.

Ole

-- 

View this message in context:
http://www.nabble.com/Counting-number-of-Errors-or-Emails-sent-tp1682374
5p16823745.html

Sent from the Log4net - Users mailing list archive at Nabble.com.

 

 

 

 

 

 

 

________________________________

From: Pranav Varia [mailto:Pranav.Varia@microsoft.com] 
Sent: Saturday, May 03, 2008 9:58 PM
To: Log4NET User
Subject: RE: Throttling/Rollup feature in EventLogAppender

 

Thanx. 

 

Dupe filtering sounds very close. Is there a way to "append" a summary
to the message itself ? Something like

 

Event id 1, Msg: Invalid input value : 45 [2 in last 5 sec]

 

Also I am thinking it would be nicer (though  more complicated) to
collapse dupe across a time window. If we check just for the last event
then it might still cause event log flooding. For example if an error
causes 2 different events (different ids) to be logged and if this error
happens too often (say per request) then the dupe checker that checks
only with previous error will allow these messages and end up with log
flooding.

 

So here is an example  of what I have in mind.

 

Example:

 

Input:

Event id 1, Msg: Invalid input value : 45

Event id 1, Msg: Invalid input value : 45

Event id 1, Msg: Invalid input value : 800                 <= Message is
different though event id is same

Event id 2, Msg: Permission denied for user: xyz     <= Different event
id

Event id 2, Msg: Permission denied for user: xyz

 

Output:

Event id 1, Msg: Invalid input value : 45 [2 in last 5 sec]

Event id 1, Msg: Invalid input value : 800 [1 in last  5 sec]

Event id 2, Msg: Permission denied for user: xyz [2 in last 5 sec]

 

 

If I can get help on how to add summary to a message, I can implement
the dupe check part using a hash.

 

Thanx

 

Regards

Pranav

 

From: Ron Grabowski [mailto:rongrabowski@yahoo.com] 
Sent: Saturday, May 03, 2008 11:52 AM
To: Log4NET User
Subject: Re: Throttling/Rollup feature in EventLogAppender

 

It looks like that class was written a while ago so you may already know
this...you should be able to extend the built-in ForwardingAppender and
let it handle the IAppenderAttachable stuff so the Append methods would
look something like this:

public class DupeFilteringForwardingAppender : ForwardingAppender
{
    protected override void Append(LoggingEvent loggingEvent)
    {
        base.Append(CheckForDupe(loggingEvent));
    }
   
    protected override void Append(LoggingEvent[] loggingEvents)
    {
        List<LoggingEvent> filteredLoggingEventList = new
List<LoggingEvent>();

        foreach (LoggingEvent loggingEvent in loggingEvents)
        {
 
filteredLoggingEventList.AddRange(CheckForDupe(loggingEvent));
        }
        
        base.Append(filteredLoggingEventList.ToArray());
    }

    // snip
}

----- Original Message ----
From: Jerry Shea <jerry.shea@gmail.com>
To: log4net-user@logging.apache.org
Sent: Saturday, May 3, 2008 12:50:13 AM
Subject: Re: Throttling/Rollup feature in EventLogAppender

Hi Pranav,

 

James Wilkinson and I worked up some code to do this. See
http://markmail.org/message/pahnqti7yq4xoz7m 

 

I am currently the bottleneck on this having not responded to James'
last message :(. I'll get moving and hopefully we can contribute this
code to the community soon.

 

Cheers, Jerry
 

---------- Forwarded message ----------
From: Pranav Varia <Pranav.Varia@microsoft.com>
To: "log4net-user@logging.apache.org" <log4net-user@logging.apache.org>
Date: Fri, 2 May 2008 14:00:08 -0700
Subject: Throttling/Rollup feature in EventLogAppender
 

	Is there a Throttling feature available or planned in
EventLogAppender.

	If we log to event log a message that could potentially be
logged at frequency of one per incoming request to my web application,
it could flood the event log with millions of messages if/when things go
wrong.  If there was a = way to roll them up every x seconds into a
single event log message (with a count of how many messages where
rolledup), it would be useful to me.

	Is there any such feature? Or workaround?

	Regards

	Pranav

	 

	 

 


Mime
View raw message