Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CACa_Z=HB00KXhR4APm5j5_YOT=o9gZYcKpaKC1vOL2+PqopM4Q@mail.gmail.com>
References: 
 <CACa_Z=HB00KXhR4APm5j5_YOT=o9gZYcKpaKC1vOL2+PqopM4Q@mail.gmail.com>
Date: Tue, 1 Dec 2015 15:43:41 +0530
Message-ID: 
 <CACa_Z=G_bRCPDp1o3PA3DGZzP9S3g2RpD9Darffh31T7G+8rNw@mail.gmail.com>
Subject: Re: Flume log4j Appender issue
From: yogendra reddy <yogendra.60@gmail.com>
To: user@flume.apache.org
Content-Type: multipart/alternative; boundary=001a11c3b8a67a82350525d36969

--001a11c3b8a67a82350525d36969
Content-Type: text/plain; charset=UTF-8

update

I ran the flume agent first and then made changes to hadoop log4j
properties file and after the restart this started working fine. but now
Hive service is not coming up because of avro-ipc jar that I had to add to
hadoop-hdfs lib to get flume log4j appender working

I would like to know if anybody here has used flume to copy Hadoop
daemon/service logs?

Thanks,
Yogendra

On Wed, Nov 25, 2015 at 2:03 PM, yogendra reddy <yogendra.60@gmail.com>
wrote:

> Hi All,
>
> I'm trying to configure flume to write hadoop service logs to a common
> sink.
>
> Here's what I have added to hdfs log4j.properties
>
> # Define the root logger to the system property "hadoop.root.logger".
> log4j.rootLogger=${hadoop.root.logger}, flume
>
> #Flume Appender
> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
> log4j.appender.flume.Hostname = localhost
> log4j.appender.flume.Port = 41414
>
> and when I run sample pi job I get this error
>
> $ hadoop jar hadoop-mapreduce-examples.jar pi 10 10
> log4j:ERROR Could not find value for key log4j.appender.flume.layout
> 15/11/25 07:23:26 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
> log4j:ERROR RPC client creation failed! NettyAvroRpcClient { host:
> localhost, port: 41414 }: RPC connection error
> Exception in thread "main" java.lang.ExceptionInInitializerError
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:200)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.commons.logging.LogConfigurationException:
> User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
> cannot be found or is not useable.
>         at
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:804)
>         at
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
>         at
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
>         at
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
>         at
> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
>         at
> org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:44)
>         ... 2 more
>
> I have added these jars to hadoop-hdfs lib -
>
> avro-ipc-1.7.3.jar
> flume-avro-source-1.5.2.2.2.7.1-33.jar
> flume-ng-log4jappender-1.5.2.2.2.7.1-33.jar
> flume-hdfs-sink-1.5.2.2.2.7.1-33.jar
> flume-ng-sdk-1.5.2.2.2.7.1-33.jar
>
> and I do have the commons-logging( commons-logging-1.1.3.jar) and
> log4j(1.2.17) jars present in the hdfs lib. Any pointers to debug this
> issue?
>
> Thanks,
> Yogendra
>
>
>

--001a11c3b8a67a82350525d36969
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>update</div><div><br></div>I ran the flume agent firs=
t and then made changes to hadoop log4j properties file and after the resta=
rt this started working fine. but now Hive service is not coming up because=
 of avro-ipc jar that I had to add to hadoop-hdfs lib to get flume log4j ap=
pender working=C2=A0<div><br></div><div>I would like to know if anybody her=
e has used flume to copy Hadoop daemon/service logs?</div><div><br></div><d=
iv>Thanks,</div><div>Yogendra</div></div><div class=3D"gmail_extra"><br><di=
v class=3D"gmail_quote">On Wed, Nov 25, 2015 at 2:03 PM, yogendra reddy <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:yogendra.60@gmail.com" target=3D"_blan=
k">yogendra.60@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmai=
l_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left=
:1ex"><div dir=3D"ltr">Hi All,<div><br></div><div>I&#39;m trying to configu=
re flume to write hadoop service logs to a common sink.</div><div><br></div=
><div>Here&#39;s what I have added to hdfs log4j.properties=C2=A0</div><div=
><br></div><div><div># Define the root logger to the system property &quot;=
hadoop.root.logger&quot;.</div><div>log4j.rootLogger=3D${hadoop.root.logger=
}, flume</div></div><div><br></div><div><div>#Flume Appender</div><div>log4=
j.appender.flume =3D org.apache.flume.clients.log4jappender.Log4jAppender</=
div><div>log4j.appender.flume.Hostname =3D localhost</div><div>log4j.append=
er.flume.Port =3D 41414</div></div><div><br></div><div>and when I run sampl=
e pi job I get this error=C2=A0</div><div><br></div><div><div>$ hadoop jar =
hadoop-mapreduce-examples.jar pi 10 10</div><div>log4j:ERROR Could not find=
 value for key log4j.appender.flume.layout</div><div>15/11/25 07:23:26 WARN=
 api.NettyAvroRpcClient: Using default maxIOWorkers</div><div>log4j:ERROR R=
PC client creation failed! NettyAvroRpcClient { host: localhost, port: 4141=
4 }: RPC connection error</div><div>Exception in thread &quot;main&quot; ja=
va.lang.ExceptionInInitializerError</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 a=
t org.apache.hadoop.util.RunJar.run(RunJar.java:200)</div><div>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)</d=
iv><div>Caused by: org.apache.commons.logging.LogConfigurationException: Us=
er-specified log class &#39;org.apache.commons.logging.impl.Log4JLogger&#39=
; cannot be found or is not useable.</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =
at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation=
(LogFactoryImpl.java:804)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apac=
he.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)=
</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.commons.logging.impl.L=
ogFactoryImpl.getInstance(LogFactoryImpl.java:292)</div><div>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 at org.apache.commons.logging.impl.LogFactoryImpl.getInstance=
(LogFactoryImpl.java:269)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apac=
he.commons.logging.LogFactory.getLog(LogFactory.java:657)</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.util.ShutdownHookManager.&lt;clin=
it&gt;(ShutdownHookManager.java:44)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 .=
.. 2 more</div></div><div><br></div><div>I have added these jars to hadoop-=
hdfs lib -</div><div><br></div><div><div>avro-ipc-1.7.3.jar =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0=C2=A0</div><div>flume-avro-source-1.5.2.2.2.7.1-33.jar=
=C2=A0</div><div>flume-ng-log4jappender-1.5.2.2.2.7.1-33.jar =C2=A0</div><d=
iv>flume-hdfs-sink-1.5.2.2.2.7.1-33.jar =C2=A0=C2=A0</div><div>flume-ng-sdk=
-1.5.2.2.2.7.1-33.jar</div></div><div><br></div><div>and I do have the comm=
ons-logging( commons-logging-1.1.3.jar) and log4j(1.2.17) jars present in t=
he hdfs lib. Any pointers to debug this issue?</div><div><br></div><div>Tha=
nks,</div><div>Yogendra</div><div><br></div><div><br></div></div>
</blockquote></div><br></div>

--001a11c3b8a67a82350525d36969--