flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ralph.goers @dslextreme.com" <ralph.go...@dslextreme.com>
Subject Re: Log4j Appender Pattern Layout
Date Tue, 16 Aug 2011 19:00:17 GMT
I'm working on Log4J 2.0. I've written a Flume appender there that I would
like feedback on. The url is
https://svn.apache.org/repos/asf/logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers/log4j2-core/src/main/java/org/apache/logging/log4j/core/appender/flume/.
 As the URL implies, this is still experimental code so I don't recommend it
for production, but if you are interested in helping out It would be most
welcome.

Ralph

On Mon, Aug 15, 2011 at 5:40 AM, Chris Jansen <
chris.jansen@cognitomobile.com> wrote:

> Thanks very much for the feedback Eric, I'm joining up to the developer
> mailing list and I've created a JIRA account so I can contribute my changes.
>
> I'll start enquiring on how the tail source copes with displaying multiline
> events so I can get that into the changes too.
>
> Rgds
>
> Chris
>
> -----Original Message-----
> From: Eric Sammer [mailto:esammer@cloudera.com]
> Sent: 12 August 2011 19:29
> To: flume-user@incubator.apache.org
> Subject: Re: Log4j Appender Pattern Layout
>
> Chris:
>
> On Fri, Aug 12, 2011 at 7:26 AM, Chris Jansen <
> chris.jansen@cognitomobile.com> wrote:
> > Hello all,
> >
> > I've spent a few hours today playing with the new Log4j avro appender,
> > thanks to Jonathan et al for getting that in the latest release. It's
> > seems to work very reliably, however it only seems to take the message
> > body of a log4j event and ignores the other elements. This is a bit of
> > a problem for me as our software tracing relies on displaying the
> > class, thread, MDC and stacktrace entries for debugging purposes.
> > Essentially flume is ignoring the log4j pattern.
> >
> > I had a look in the source and found that the Log4JEventAdaptor class
> > and found that only the message of the event was being used, and the
> > layout ignored. So I created a new constructor to allow me to pass in
> > a Log4j layout to the event adaptor, I also modified the getBody
> > method to use the layout if the layout object is not null and I added
> > some code to print the stacktrace if it is present. See my hack below:
> >
> >     public Log4JEventAdaptor(LoggingEvent evt, Layout layout) {
> >
> >         super();
> >
> >         this.evt = evt;
> >
> >         this.layout = layout;
> >
> >         // This is needed to differentiate between events at the same
> > millisecond.
> >
> >         this.nanos = System.nanoTime();
> >
> >     }
> >
> >
> >
> >     public byte[] getBody() {
> >
> >         if (layout == null) {
> >
> >             return evt.getRenderedMessage().getBytes();
> >
> >         } else {
> >
> >
> >
> >             String body = layout.format(evt);
> >
> >
> >
> >             String[] s = evt.getThrowableStrRep();
> >
> >             if (s != null) {
> >
> >                 int len = s.length;
> >
> >                 for (int i = 0; i < len; i++) {
> >
> >                     body = body + s[i];
> >
> >                     body = body + Layout.LINE_SEP;
> >
> >                 }
> >
> >             }
> >
> >
> >
> >             return body.getBytes();
> >
> >         }
> >
> >     }
> >
> > So now I get formatted log4j entries in flume! My question is; was
> > there a reason for ignoring the log4j layout in the event adaptor and
> > might what I have done be useful to anyone else?
>
> It wasn't a choice to ignore the layout, more a lack of knowledge of the
> finer points of writing a log4j appender. I think this would definitely be
> useful to others. A few other folks have mentions things like the MDC / NDC
> data (which I know less about) as something else Flume doesn't carry
> properly. Definitely check out
> https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute if
> you're interested in getting these changes back into Flume proper.
>
> (If you do decide to contribute, check out StringBuilder. :))
>
> > Also stack traces are appearing all on one line with the line
> > separator escape character instead of a new line. From what I have
> > read the tail source can cope with stacktraces and make them into a
> > single event, is it possible to do this with the log4j/avro source?
>
> Ideally, a single log4j event becomes a single Flume event. If the content
> of that event is multiple lines, that's fine. Flume should not mutate the
> content of the message.
>
> Thanks for the comments. The changes sound great.
>
> >
> >
> >
> > Thanks
> >
> >
> >
> > Chris
> >
> >
> >
> >
> > NOTICE: Cognito Limited. Benham Valence, Newbury, Berkshire, RG20 8LU.
> UK.
> > Company number 02723032. This e-mail message and any attachment is
> > confidential. It may not be disclosed to or used by anyone other than
> > the intended recipient. If you have received this e-mail in error
> > please notify the sender immediately then delete it from your system.
> > Whilst every effort has been made to check this mail is virus free we
> > accept no responsibility for software viruses and you should check for
> > viruses before opening any attachments. Opinions, conclusions and
> > other information in this email and any attachments which do not
> > relate to the official business of the company are neither given by the
> company nor endorsed by it.
> >
> > This email message has been scanned for viruses by Mimecast
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>
>
>

Mime
View raw message