Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.220.225 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=OOl41WOZ4eUN+1z5U3zQEGJkl0elbnHnSrqq3U8xRI0daW4hCTsUvUybCGc2lcEc81
         PKQxtKz7JVxwKm0Yc6lMLlSRhl4lLU0OKB2OVN3jGFnpk6xGq+9Ileqp+7/HSDMNMbBD
         PsOt/kuOicqVZ1yRhDg3sWKjR5c0yfKWfYB7s=
MIME-Version: 1.0
In-Reply-To: <2986c2f30908200749y553337b2xf75f39ed764ac744@mail.gmail.com>
References: <2986c2f30908191632u74dcad9dt9073fc84eb7d849e@mail.gmail.com>
	 <480D7E03-6E9F-4917-BE31-A82D2A42D9C8@cse.unl.edu>
	 <cbbf4b570908200744l5e206ea1hbb8fe5d439fbfc77@mail.gmail.com>
	 <2986c2f30908200749y553337b2xf75f39ed764ac744@mail.gmail.com>
Date: Thu, 20 Aug 2009 11:16:32 -0400
Message-ID: <cbbf4b570908200816h493e7e04ue4c2679cb464d2cb@mail.gmail.com>
Subject: Re: syslog-ng and hadoop
From: Edward Capriolo <edlinuxguru@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Aug 20, 2009 at 10:49 AM, mike anderson<saidtherobot@gmail.com> wro=
te:
> Yeah, that is interesting Edward. I don't need syslog-ng for any particul=
ar
> reason, other than that I'm familiar with it. If there were another way t=
o
> get all my logs collated into one log file that would be great.
> mike
>
> On Thu, Aug 20, 2009 at 10:44 AM, Edward Capriolo <edlinuxguru@gmail.com>=
wrote:
>
>> On Wed, Aug 19, 2009 at 11:50 PM, Brian Bockelman<bbockelm@cse.unl.edu>
>> wrote:
>> > Hey Mike,
>> >
>> > Yup. =A0We find the stock log4j needs two things:
>> >
>> > 1) Set the rootLogger manually. =A0The way 0.19.x has the root logger =
set
>> up
>> > breaks when adding new appenders. =A0I.e., do:
>> >
>> > log4j.rootLogger=3DINFO,SYSLOG,console,DRFA,EventCounter
>> >
>> > 2) Add the headers; otherwise log4j is not compatible with syslog:
>> >
>> > log4j.appender.SYSLOG=3Dorg.apache.log4j.net.SyslogAppender
>> > log4j.appender.SYSLOG.facility=3Dlocal0
>> > log4j.appender.SYSLOG.layout=3Dorg.apache.log4j.PatternLayout
>> > log4j.appender.SYSLOG.layout.ConversionPattern=3D%p %c{2}: %m%n
>> > log4j.appender.SYSLOG.SyslogHost=3Dred
>> > log4j.appender.SYSLOG.threshold=3DERROR
>> > log4j.appender.SYSLOG.Header=3Dtrue
>> > log4j.appender.SYSLOG.FacilityPrinting=3Dtrue
>> >
>> > Brian
>> >
>> > On Aug 19, 2009, at 6:32 PM, Mike Anderson wrote:
>> >
>> >> Has anybody had any luck setting up the log4j.properties file to send
>> logs
>> >> to a syslog-ng server?
>> >> My log4j.properties excerpt:
>> >> log4j.appender.SYSLOG=3Dorg.apache.log4j.net.SyslogAppender
>> >> log4j.appender.SYSLOG.syslogHost=3D10.0.20.164
>> >> log4j.appender.SYSLOG.layout=3Dorg.apache.log4j.PatternLayout
>> >> log4j.appender.SYSLOG.layout.ConversionPattern=3D%d{ISO8601} %p %c: %=
m%n
>> >> log4j.appender.SYSLOG.Facility=3DHADOOP
>> >>
>> >> and my syslog-ng.conf file running on 10.0.20.164
>> >>
>> >> source s_hadoop {
>> >> =A0 =A0 =A0 # message generated by Syslog-NG
>> >> =A0 =A0 =A0 internal();
>> >> =A0 =A0 =A0 # standard Linux log source (this is the default place fo=
r the
>> >> syslog()
>> >> =A0 =A0 =A0 # function to send logs to)
>> >> =A0 =A0 =A0 unix-stream("/dev/log");
>> >> =A0 =A0 =A0 udp();
>> >> };
>> >> destination df_hadoop { file("/var/log/hadoop/hadoop.log");};
>> >> filter f_hadoop {facility(hadoop);};
>> >> log {
>> >> source(s_hadoop);
>> >> filter(f_hadoop);
>> >> destination(df_hadoop);
>> >> };
>> >>
>> >>
>> >> Thanks in advance,
>> >> Mike
>> >
>> >
>>
>> Mike slightly off topic but you can also run a Log 4J server which
>> perfectly transports the messages fired off by LOG4j. The
>> log4J->syslog loses/ changes some information. If anyone is interested
>> in this let me know and I will write up something about it.
>>
>

Mike,
I just put this up for you.
http://www.edwardcapriolo.com/wiki/en/Log4j_Server

All of the functionality is in the class
org.apache.log4j.net.SocketServer which ships as part of Log4j.

I pretty much followed this http://timarcher.com/node/10

I started with the syslog appender but it had some quirks. Mostly the
syslog appender can only write a syslog so it loses some information.
The Log4jserver transfers the log.error("whatever" ) as is and can
handle it on the server end though the servers logging properties.
Cool stuff.