logging-log4j-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Duane <nic...@msn.com>
Subject Re: approach for defining loggers
Date Mon, 17 Oct 2016 23:27:23 GMT
Sorry to revive this old thread.  However, we're in the process of adding support for other
"categories" of events and thus I wanted to first take a step back and try to ensure we're
not convoluting things.


There was a requirement to log a "compliance" event under certain conditions.  We did not
want to write our own logging framework and instead decided to use existing off-the-shelf
logging frameworks.  We have applications on both Linux/Java, Windows/.NET and Windows/Java.
 Initially we chose log4net for Windows/.NET and log4j2 for Windows/Java and Linux/Java. 
For these logging frameworks we wrote artifacts, appenders basically, to help facilitate getting
these events to our system.  By the way, our system will get the events centrally, possibly
put them into a relational database and also hand them off to another system which will get
them eventually to an HDFS backend.  We also exposed methods for creating this compliance
event.  The compliance event is basically a map.  We chose a map so that the event could also
be extended by the application team in case they needed to add additional properties which
made sense for them.


We chose to create a custom level for this "compliance" event such that we could filter out
only these events and get them into our system.  The configuration example we created had
our custom unix domain socket appender added to the root logger.  It also contained a filter
which filtered out any events that weren't compliance events.  The level we chose for "compliance"
was less critical than off and more critical than fatal as we wanted to ensure that as long
as logging wasn't turned off altogether our events would get logged.


I want to go over a few suggestions that were made and explain why I didn't make use of those
suggestions.


1. Since our "compliance" level didn't fit within the "vernacular" of the existing levels
we should not define this custom level.  Instead we should look at using markers.


I am not that familiar with markers but did look into them when they were suggested.  While
I don't have anything against markers in general there were some downsides as I saw it.


a. Markers are not available, as far as I know, in log4net so we'd still have to figure out
something there.

b. A bigger problem, at least I thought it was a bigger problem, was that there would be confusion
about what level to log the event at.  I would certainly not want to give an example as follows:


logger.debug(COMPLIANCE_MARKER, evnt);


or


logger.info(COMPLIANCE_MARKER, evnt);


or


logger.error(COMPLIANCE_MARKER, evnt);


...


That just screams confusion to me.


2. Use a dedicated logger to log all compliance events.


There were, as far as I could tell, a couple problems with this approach also.


a. If everyone is using a single "well known" logger to log a specific event category then
I lose the logger "context" of who's logging the event.  As it stands now we're copying the
logger name into a property we call "eventSource".


b. You cannot turn on/off logging for a specific set of code.  If it turns out that we have
some errant code which is using this well known logger then we can't just turn off that code
from logging as turning off the well know logger would turn it off for everyone using it.


I did look into the EventLogger and initially that seemed promising as I guess it logs any
event you give it at the "all" level.  However, as a well known logger it suffers from the
same issues above.


Now we're looking to add Business events.  My initial thinking is that I can do the same thing
we did before.  Add an additional custom level called "Business" and expose methods for creating
a business event.  Though unlike the compliance event, the application teams would be defining
the schema more so than our framework team.  Thus any method we expose would just be used
as a starting point for setting the common properties.  You would use another instance of
our unix domain socket appender for these business events and forward them to a different
location as business events would most likely have a different retention period than compliance
events.  Plus you might also want them in a different store as you may never need to query
for both categories of events and thus no need to query against a larger set of data.


In addition we're planning to capture centrally what we refer to as diagnostic events: error,
info, warn, debug, trace, etc.  However, we may need to separate these out into two different
categories: critical-diagnostic and noncritical-diagnostic.  The reason is that we don't want
the potential of a critical diagnostic event, let's say an error, queued up behind potentially
thousands of non-critical diagnostic events.  So you see, the category also defines aspects
on how we handle events at the source.   We separate at the source based on category as it
seems a reasonable place to do so.  Also, you may want different flush times for different
categories.  We have a process which buffers, compresses and sends events centrally so we
have the notion of flush time.  The buffers are flushed when they become full or the flush
time elapses.  Errors, since they are more critical in monitoring systems, we'll most likely
want to flush more often than say debug and trace events.


Sorry for the long winded explanation.  Initially I was thinking that when we create an event
we'd set its category.  However, now I'm thinking the category should be set by the act of
logging the event at a level.  In some cases we have a 1:1 mapping from level to category,
eg. compliance level -> compliance category.  In some cases we have a many:1 mapping from
level to category, eg. error, info, warn -> critical-diagnostic.


We could also just define a single custom level, say "always_on", or something like that.
 Then we provide some helper method to log our "new" event categories (eg. business and compliance)
at this level and have the user specify the category, I guess similar to a marker.


logEvent(Logger logger, String category, object evnt);


I guess it's similar to the EventLogger except that we're not using a single well known logger
and thus don't have the downsides of that which I pointed out earlier.


Any thoughts/suggestions would be appreciated.


Thanks,

Nick

________________________________
From: Mikael Ståldal <mikael.staldal@magine.com>
Sent: Wednesday, September 9, 2015 3:47 AM
To: Log4J Users List
Subject: Re: approach for defining loggers

Then perhaps you should create your own facade for doing business event
logging, which could then forward them to Log4j in an appropriate way.

On Wed, Sep 9, 2015 at 4:49 AM, Nicholas Duane <nickdu@msn.com> wrote:

> I was just about to reply to your previous email about using a single
> "business" logger, or some hierarchy of business loggers, to log business
> events and say that we might go that route.  However, now that you brought
> up the post from Ralph, which I just replied to, I'm thinking a logger
> won't work either for the same reason I listed in my reply to Ralph's post.
>
> You could do:
>
> logger.info("Hello");
> logger.fatal("Hello");
> logger.error("Hello");
> ...
>
> It's confusing as there are n ways to log a business event that way and
> they will all do the same thing.  Which one should a developer choose.
> Should I say pick any one, it doesn't matter?
>
> Thanks,
> Nick
>
> > Date: Tue, 8 Sep 2015 19:28:21 -0700
> > Subject: Re: approach for defining loggers
> > From: garydgregory@gmail.com
> > To: log4j-user@logging.apache.org
> >
> > Or
> > Logger logger = LogManager.getLogger("Business");
> > ...
> > logger.info("Hello");
> >
> > Gary
> >
> > On Tue, Sep 8, 2015 at 7:24 PM, Ralph Goers <ralph.goers@dslextreme.com>
> > wrote:
> >
> > > Can you please clarify, “If we had some way to know an event is a
> business
> > > event we wouldn’t need level”?  I do not understand how you can code
> > > logger.log(BUSINESS, msg)  but you cannot code logger.info(BUSINESS,
> msg).
> > >
> > > Ralph
> > >
> > > > On Sep 8, 2015, at 6:09 PM, Nicholas Duane <nickdu@msn.com> wrote:
> > > >
> > > >
> > > >
> > > >
> > > > I looked over that stackoverflow post and I'm still not seeing a good
> > > match as a way for us to log our business events.
> > > >
> > > > A business event I guess is an event which extends whatever schema we
> > > come up with for a business event.  While an instance of this schema
> could
> > > be logged at any level, that really doesn't make sense in our scenario,
> > > regardless of whether some marker was supplied.  If we had some way to
> know
> > > an event is a business event we wouldn't need level.  We could of
> course
> > > add some property to our schema which indicates the 'category' of the
> > > event, 'business' being one such category.  Instead we were thinking we
> > > could just use level to indicate that an event is a business event.
> > > >
> > > > As I mentioned, we're looking to capture 'trace' level events to one
> > > store, 'info' - 'fatal' level events to another store, and 'business'
> > > events to yet another store.  For 'trace' and 'info' - 'fatal' it seems
> > > reasonable to filter on level within the appender to get those events
> to
> > > the appropriate location.  It seemed reasonable to do something
> similar for
> > > 'business'.
> > > >
> > > > I also looked into the EventLogger but not sure that's appropriate.
> For
> > > one we lose the granularity to control a specific piece of code from
> > > generating business events.  This is most likely a non-issue as I have
> > > mentioned that we don't want to turn business logging off.  The other
> is
> > > that we lose the name of the logger as it would be the same for
> everyone.
> > > Not sure this is that big a deal either as I guess you might be able to
> > > capture component name, though I would rather distinguish using logger
> name.
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > > >> From: ralph.goers@dslextreme.com
> > > >> Subject: Re: approach for defining loggers
> > > >> Date: Mon, 7 Sep 2015 20:39:11 -0700
> > > >> To: log4j-user@logging.apache.org
> > > >>
> > > >> I still don’t understand why you don’t want to use Markers. They
> were
> > > designed exactly for the use case you are describing.
> > > >>
> > > >> You might set retention policies for debug vs info, error and fatal,
> > > but a BUSINESS marker could cross-cut them all.  That is exactly why
> it is
> > > NOT a level. IOW, it gives you a second dimension for filtering. Ceki
> > > invented Markers when he created SLF4J. For his point of view see
> > >
> http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them
[http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded&a]<http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them>

What is markers in Java Logging frameworks and that is a ...<http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them>
stackoverflow.com
This is a rehashed version my answer to the question "Best practices for using Markers in
SLF4J/Logback". Markers can be used to color or mark a single log statement.


> > > <
> > >
> http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them
[http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded&a]<http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them>

What is markers in Java Logging frameworks and that is a ...<http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them>
stackoverflow.com
This is a rehashed version my answer to the question "Best practices for using Markers in
SLF4J/Logback". Markers can be used to color or mark a single log statement.


> > > >.
> > > >>
> > > >> Ralph
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> On Sep 7, 2015, at 5:54 PM, Nicholas Duane <nickdu@msn.com>
wrote:
> > > >>>
> > > >>> If I'm attempting to control all the logging from the configuration
> > > and I don't know the complete set of loggers in my application as there
> > > could be 100's or 1000's, wouldn't it be hard to separate events based
> on
> > > loggers?  It would seem much easier to separate events based on
> level.  In
> > > addition, level might be a more reasonable approach for separating.
> For
> > > example, if I want to send all events to some big-data backend I might
> want
> > > to separate out traces and debug from info to fatal as traces and
> debug are
> > > most likely less important from a systems management aspect.  My
> retention
> > > period for traces and debug might be just a couple days.  The retention
> > > period for info to fatal could be 30 days.  Business level might be 2
> > > years.  Any system management notifications would probably be driven
> off of
> > > info to fatal events and not trace and debug events, which is another
> > > reason you might want to separate by level.
> > > >>>
> > > >>> Thanks,
> > > >>> Nick
> > > >>>
> > > >>>> Subject: Re: approach for defining loggers
> > > >>>> From: ralph.goers@dslextreme.com
> > > >>>> Date: Mon, 31 Aug 2015 08:50:58 -0700
> > > >>>> To: log4j-user@logging.apache.org
> > > >>>>
> > > >>>> A logging “Level” is a level of importance. That is why
there is a
> > > hierarchy. If you want informational messages then you also would want
> > > warnings and errors.
> > > >>>>
> > > >>>> “BUSINESS” does not convey the same meaning.  Rather,
it is some
> sort
> > > of category, which is what Markers are for.
> > > >>>>
> > > >>>> Using the class name as the logger name is a convention. If
you
> > > really want the class name, method name or line number then you should
> be
> > > specifying that you want those from the logging event, rather than the
> > > logger name.  Unless location information is disabled you always have
> > > access to that information.
> > > >>>>
> > > >>>> In short, different loggers are used primarily as a way of
> grouping
> > > sets of messages - for example all org.hibernate events can be routed
> to a
> > > specific appender or turned off en masse. Levels are used to filter out
> > > noise across a set of logging events. Markers are used to categorize
> > > logging events by arbitrary attributes.
> > > >>>>
> > > >>>> Ralph
> > > >>>>
> > > >>>>
> > > >>>>> On Aug 31, 2015, at 8:10 AM, Nicholas Duane <nickdu@msn.com>
> wrote:
> > > >>>>>
> > > >>>>> Thanks for the feedback.  I will look into Markers and
MDC.
> > > >>>>>
> > > >>>>> With respect to using a separate logger, it would seem
I would
> lose
> > > the information about what application code, eg. the class logger, is
> > > sourcing the event.  We would like to have this information.  On top of
> > > that, it seems odd, maybe to me only, that for this new level we have
> our
> > > own logger.  It seemed reasonable to me that this new event we want to
> > > capture is just a new level.  Just like a DEBUG event is different
> from an
> > > INFO event.  If I define a BUSINESS level why would that not follow the
> > > same design as the current levels?  You wouldn't suggest having
> different
> > > loggers for TRACE DEBUG INFO WARN ERROR FATAL, would you?  I think one
> of
> > > the reasons someone on our side is suggesting I have separate loggers
> is
> > > that they think the overhead of filtering at the appender is going to
> have
> > > a noticeable impact.  Our plan, at least the one I have now in my
> head, is
> > > that we'll have some number of appenders in the root.  We'll then
> filter x
> > > < INFO events to a tracing appender, INFO <= x <= FATAL to a logging
> > > appender, and our custom level will go to another appender.  Thoughts?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Nick
> > > >>>>>
> > > >>>>>> Subject: Re: approach for defining loggers
> > > >>>>>> From: ralph.goers@dslextreme.com
> > > >>>>>> Date: Sat, 29 Aug 2015 20:59:36 -0700
> > > >>>>>> To: log4j-user@logging.apache.org
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>> On Aug 29, 2015, at 7:44 PM, Nicholas Duane <nickdu@msn.com>
> > > wrote:
> > > >>>>>>>
> > > >>>>>>> I'm curious if there is a prescribed approach
to defining
> > > loggers.  Let me state what my assumption is.  I assume that normally
> if
> > > some piece of code wants to log events/messages that it should create a
> > > logger for itself.  I guess a reasonable name to use is the class name
> > > itself.  In terms of logger configuration I would expect that no
> loggers
> > > are specified in the log4j configuration UNLESS is needs settings other
> > > than the default.  The root logger would specify the default settings,
> eg.
> > > level and appenders.  If some piece of code tied to a logger needs to
> > > enable tracing in order to debug an issue then you would add that
> logger to
> > > the configuration and set the level less specific for that logger.  Is
> this
> > > a typical and reasonable approach?
> > > >>>>>>
> > > >>>>>> What you describe here is the common convention. It
is a
> reasonable
> > > approach.
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> I asked because we have the need for a new type
of event.  To
> have
> > > this event flow to where we want it to flow the plan is to have a
> custom
> > > level and have all events at that level captured by a specific
> appender.
> > > My assumption was that for existing applications we'd just need to add
> our
> > > appender to the root and add our custom level.  The app would need to
> be
> > > modified to log our new event at the custom level.  However, someone
> > > suggested that we could also create a separate logger for this event.
> My
> > > thinking is that while we don't ever want to turn off logging of this
> > > event, loggers represent "event sources", e.g the code raising the
> events
> > > and thus having multiple different pieces of code use the same logger
> > > wouldn't allow you to turn on/off logging from those different
> sections of
> > > code independently.  I think the current configuration includes all the
> > > loggers.  Normally I would expect there to be many, on the order of
> 10's or
> > > 100's, loggers within an application.  However, in the case I was given
> > > there were only a handful because I think this handful is shared.  So
> as I
> > > mentioned, this doesn't sound like an ideal design as you have less
> > > granularity on what you can turn on/off.
> > > >>>>>>
> > > >>>>>> You have a few options. Using a CustomLevel would
not be the
> option
> > > I would choose.  Creating a custom Logger will certainly work and makes
> > > routing the message to the appropriate appender rather easy.  Another
> > > approach is to use Markers.  Markers are somewhat hierarchical so you
> can
> > > use them for a variety of purposes.  If you look at how Log4j handles
> event
> > > logging it actually does both - it specifies EventLogger as the name
> of the
> > > logger to use and it uses Markers to identify the kind of event.
> > > >>>>>>
> > > >>>>>> A third option is to use the MDC or Logger properties.
If you do
> > > that then you can have information included in the actual logging event
> > > that can affect how it is routed. I also built a system that uses the
> > > RFC5424 format so that the event could have lots of key/value pairs to
> > > identify the events.
> > > >>>>>>
> > > >>>>>> Unfortunately, without knowing more details I don’t
know that I
> can
> > > give you a better idea on how I would implement it.
> > > >>>>>>
> > > >>>>>> Ralph
> > > >>>>>>
> > > >>>>>>
> > > ---------------------------------------------------------------------
> > > >>>>>> To unsubscribe, e-mail:
> log4j-user-unsubscribe@logging.apache.org
> > > >>>>>> For additional commands, e-mail:
> log4j-user-help@logging.apache.org
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> > > >>>> For additional commands, e-mail:
> log4j-user-help@logging.apache.org
> > > >>>>
> > > >>>
> > > >>
> > > >
> > > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> > > For additional commands, e-mail: log4j-user-help@logging.apache.org
> > >
> > >
> >
> >
> > --
> > E-Mail: garydgregory@gmail.com | ggregory@apache.org
> > Java Persistence with Hibernate, Second Edition
> > <http://www.manning.com/bauer3/>
> > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
> > Spring Batch in Action <http://www.manning.com/templier/>
> > Blog: http://garygregory.wordpress.com
[https://s0.wp.com/i/blank.jpg]<http://garygregory.wordpress.com/>

Gary Gregory<http://garygregory.wordpress.com/>
garygregory.wordpress.com
Software construction, the web, and other techs


> > Home: http://garygregory.com/
Gary Gregory<http://garygregory.com/>
garygregory.com
Rocket | Seagull . I am a Software Architect for Seagull Software, a division of Rocket Software.
Rocket Seagull specializes in tools and expertise to modernize ...


> > Tweet! http://twitter.com/GaryGregory
Gary Gregory (@GaryGregory) | Twitter<http://twitter.com/GaryGregory>
twitter.com
The latest Tweets from Gary Gregory (@GaryGregory). Principal Software Engineer, author: Java
Persistence Hibernate https://t.co/3F8sYxc0oq, JUnit https://t.co/yXU1DqAMDG, Spring Batch
https://t.co/XwoMNoBxh7. U.S.A.


>
>



--
[image: MagineTV]

*Mikael Ståldal*
Senior software developer

*Magine TV*
mikael.staldal@magine.com
Regeringsgatan 25  | 111 53 Stockholm, Sweden  |   www.magine.com<http://www.magine.com>
[https://de.magine.com/content/uploads/2016/09/magine_global_social.png]<http://www.magine.com/>

TV online with Magine TV<http://www.magine.com/>
www.magine.com
Watch the TV you love, on any device, anywhere in Germany and Sweden and find out more about
our global OTT B2B solutions. Get started today.



Privileged and/or Confidential Information may be contained in this
message. If you are not the addressee indicated in this message
(or responsible for delivery of the message to such a person), you may not
copy or deliver this message to anyone. In such case,
you should destroy this message and kindly notify the sender by reply
email.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message