flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Flume log4j Appender issue
Date Tue, 01 Dec 2015 11:10:10 GMT
Ok, I see now.
If you want to collect logs what I would do is a directory spooling source
on the namenode instead of messing with hdfs, sounds too risky.



On 1 December 2015 at 11:04, yogendra reddy <yogendra.60@gmail.com> wrote:

> My usecase is to copy hadoop service logs and that is why I'm adding flume
> log4j appender to hadoop log4j properties.
>
> # Define the root logger to the system property "hadoop.root.logger".
> log4j.rootLogger=${hadoop.root.logger}, *flume*
>
> #Flume Appender
> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
> log4j.appender.flume.Hostname = localhost
> log4j.appender.flume.Port = 41414
>
>  now, for this to work I need to add the flume log4j appender related
> libraries to hadoop classpath right? If not this will throw
> NoSuchClass found expection org.apache.flume.clients.
> log4jappender.Log4jAppender.
>
>
> On Tue, Dec 1, 2015 at 4:24 PM, Gonzalo Herreros <gherreros@gmail.com>
> wrote:
>
>> That doesn't sound right:
>> -Flume should use it's own log4j.properties in the conf directory
>> -Never update the hdfs libs to add stuff you need for Flume, each product
>> has it's own library directories
>>
>> Regards,
>> Gonzalo
>>
>>
>> On 1 December 2015 at 10:48, yogendra reddy <yogendra.60@gmail.com>
>> wrote:
>>
>>> I din't follow. I'm adding flume libraries to hadoop classpath i.e
>>> hadoop-hdfs lib folder and this is causing the issue. I need these jars to
>>> be in hdfs lib as I have added log4j appender to hdfs log4j properties.
>>>
>>> On Tue, Dec 1, 2015 at 4:09 PM, Gonzalo Herreros <gherreros@gmail.com>
>>> wrote:
>>>
>>>> Adding a library to Flume shouldn't affect hive or any other tools.
>>>> You can add the jar to the lib or plugin.d directories.
>>>>
>>>> Regards,
>>>> Gonzalo
>>>>
>>>> On 1 December 2015 at 10:13, yogendra reddy <yogendra.60@gmail.com>
>>>> wrote:
>>>>
>>>>> update
>>>>>
>>>>> I ran the flume agent first and then made changes to hadoop log4j
>>>>> properties file and after the restart this started working fine. but
now
>>>>> Hive service is not coming up because of avro-ipc jar that I had to add
to
>>>>> hadoop-hdfs lib to get flume log4j appender working
>>>>>
>>>>> I would like to know if anybody here has used flume to copy Hadoop
>>>>> daemon/service logs?
>>>>>
>>>>> Thanks,
>>>>> Yogendra
>>>>>
>>>>> On Wed, Nov 25, 2015 at 2:03 PM, yogendra reddy <yogendra.60@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I'm trying to configure flume to write hadoop service logs to a
>>>>>> common sink.
>>>>>>
>>>>>> Here's what I have added to hdfs log4j.properties
>>>>>>
>>>>>> # Define the root logger to the system property "hadoop.root.logger".
>>>>>> log4j.rootLogger=${hadoop.root.logger}, flume
>>>>>>
>>>>>> #Flume Appender
>>>>>> log4j.appender.flume =
>>>>>> org.apache.flume.clients.log4jappender.Log4jAppender
>>>>>> log4j.appender.flume.Hostname = localhost
>>>>>> log4j.appender.flume.Port = 41414
>>>>>>
>>>>>> and when I run sample pi job I get this error
>>>>>>
>>>>>> $ hadoop jar hadoop-mapreduce-examples.jar pi 10 10
>>>>>> log4j:ERROR Could not find value for key log4j.appender.flume.layout
>>>>>> 15/11/25 07:23:26 WARN api.NettyAvroRpcClient: Using default
>>>>>> maxIOWorkers
>>>>>> log4j:ERROR RPC client creation failed! NettyAvroRpcClient { host:
>>>>>> localhost, port: 41414 }: RPC connection error
>>>>>> Exception in thread "main" java.lang.ExceptionInInitializerError
>>>>>>         at org.apache.hadoop.util.RunJar.run(RunJar.java:200)
>>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>>>>> Caused by: org.apache.commons.logging.LogConfigurationException:
>>>>>> User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
>>>>>> cannot be found or is not useable.
>>>>>>         at
>>>>>> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:804)
>>>>>>         at
>>>>>> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
>>>>>>         at
>>>>>> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
>>>>>>         at
>>>>>> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
>>>>>>         at
>>>>>> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
>>>>>>         at
>>>>>> org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:44)
>>>>>>         ... 2 more
>>>>>>
>>>>>> I have added these jars to hadoop-hdfs lib -
>>>>>>
>>>>>> avro-ipc-1.7.3.jar
>>>>>> flume-avro-source-1.5.2.2.2.7.1-33.jar
>>>>>> flume-ng-log4jappender-1.5.2.2.2.7.1-33.jar
>>>>>> flume-hdfs-sink-1.5.2.2.2.7.1-33.jar
>>>>>> flume-ng-sdk-1.5.2.2.2.7.1-33.jar
>>>>>>
>>>>>> and I do have the commons-logging( commons-logging-1.1.3.jar) and
>>>>>> log4j(1.2.17) jars present in the hdfs lib. Any pointers to debug
this
>>>>>> issue?
>>>>>>
>>>>>> Thanks,
>>>>>> Yogendra
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message