flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chhaya Vishwakarma <Chhaya.Vishwaka...@lntinfotech.com>
Subject RE: Could not instantiate class org.apache.flume.clients.log4jappender
Date Mon, 14 Jan 2013 07:09:24 GMT
Which version of slf4j you using ? for this error you need to put SLF4J 1.7.2 also in classpath
as org.slf4j.spi.LocationAwareLogger.log is available in 1.7.2 of slf4j.

From: Dongliang Sun [mailto:davidsundl@gmail.com]
Sent: Monday, January 14, 2013 11:54 AM
To: Connor Woodson; Chhaya Vishwakarma; user@flume.apache.org
Subject: Re: Could not instantiate class org.apache.flume.clients.log4jappender

It does not work, -Dpig.additional.jars={REALPATH}/lib/*.
Also tried Chhaya said, I put all the libraries from FLUME_HOME/lib/* to PIG_HOME/lib/, got
following error:

13/01/14 14:19:03 WARN api.NettyAvroRpcClient: Invalid value for batchSize: 0; Using default
value.
Exception in thread "main" java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
    at org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:211)
    at org.apache.pig.impl.util.LogUtils.writeLog(LogUtils.java:125)
    at org.apache.pig.impl.util.LogUtils.writeLog(LogUtils.java:79)
    at org.apache.pig.Main.run(Main.java:589)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

2013/1/14 Connor Woodson <cwoodson.dev@gmail.com<mailto:cwoodson.dev@gmail.com>>
Alright so it is just before you weren't adding them properly to the classpath. I am not quite
sure how to do that; a look at Pig's run script (bin/pig) might yield insight into an option
to add them. Putting the files in pig's lib's folder almost works; the issue is those libraries
have more dependencies that aren't in the classpath yet. Looking at their (log4jappender and
sdk) poms could tell you exactly which libs you need to include, but let's not do that.

I mistyped the -D command from before. Try this:

-Dpig.additional.jars=${FLUME_HOME}/lib/*

FLUME_HOME obviously needs to be declared beforehand, or just replace that with the real path
(and make sure the log4jappender and sdk jars, and all other jars, are in the libs folder)

On Sun, Jan 13, 2013 at 10:05 PM, Dongliang Sun <davidsundl@gmail.com<mailto:davidsundl@gmail.com>>
wrote:
REGISTER and -Dpig.additional.libraries both do not work, same error, could not find the class.
If put the particular jars in pig's lib, the following error:
Error before Pig is launched
----------------------------
ERROR 2998: Unhandled internal error. org/jboss/netty/channel/ChannelFactory

java.lang.NoClassDefFoundError: org/jboss/netty/channel/ChannelFactory
        at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:168)
        at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:128)
        at org.apache.flume.clients.log4jappender.Log4jAppender.activateOptions(Log4jAppender.java:184)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)

        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
        at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
        at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
        at org.apache.pig.Main.configureLog4J(Main.java:678)
        at org.apache.pig.Main.run(Main.java:337)
        at org.apache.pig.Main.main(Main.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: org.jboss.netty.channel.ChannelFactory

        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        ... 19 more

2013/1/14 Connor Woodson <cwoodson.dev@gmail.com<mailto:cwoodson.dev@gmail.com>>
You could try calling 'REGISTER <log4jappender jar>' at the start of your script, or
on the command line by adding -Dpig.additional.libraries=<log4jappender jar> to your
Pig call. I don't know if that will work or not though.

You could try placing it in pig's lib folder.

You also need to include the flume-ng-sdk jar the same way as the log4jappender one, as the
sdk is a dependency of the appender.

On Sun, Jan 13, 2013 at 9:24 PM, Dongliang Sun <davidsundl@gmail.com<mailto:davidsundl@gmail.com>>
wrote:
Actually I have append the FLUME_HOME/lib/ to my CLASSPATH.
Whether there is some problem in my PIG log4j.properties?
Do you have any experience using log4jappender on PIG?
Thanks,
Dongliang


2013/1/14 Chhaya Vishwakarma <Chhaya.Vishwakarma@lntinfotech.com<mailto:Chhaya.Vishwakarma@lntinfotech.com>>

Put flume-log4j-appender<https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-log4j-appender/0.9.4-cdh3u1/flume-log4j-appender-0.9.4-cdh3u1.jar>.jar
in CLASSPATH. Jar can be found in FLUME_HOME/lib/



From: 孙东亮 [mailto:davidsundl@gmail.com<mailto:davidsundl@gmail.com>]
Sent: Monday, January 14, 2013 8:55 AM
To: user@flume.apache.org<mailto:user@flume.apache.org>
Subject: Could not instantiate class org.apache.flume.clients.log4jappender



I'm a newbie for the Flume, and I just set up flume for a test.
I want to use the log4jappender to get log info from PIG scripts, the log4j.properties is:

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender

log4j.appender.flume.Hostname = localhost

log4j.appender.flume.Port = 41414

log4j.logger.org.apache.pig=DEBUG,flume

And for the flume.config:

a1.sources = r1

a1.sinks = k1

a1.channels = c1



# Describe/configure the source

a1.sources.r1.type = avro

a1.sources.r1.bind = localhost

a1.sources.r1.port = 41414



# Describe the sink

a1.sinks.k1.type = logger

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100



# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

But I got the following error when run a pig script:
java.lang.ClassNotFoundException: org.apache.flume.clients.log4jappender.Log4jAppender
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
at org.apache.pig.Main.configureLog4J(Main.java:678)
at org.apache.pig.Main.run(Main.java:337)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
log4j:ERROR Could not instantiate appender named "flume".

Could you please help me find what I miss or where is incorrect for the configuration.

Thanks a lot!
Dongliang

________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information
for the intended recipient(s). Unintended recipients are prohibited from taking action on
the basis of information in this e-mail and using or disseminating the information, and must
notify the sender and delete it from their system. L&T Infotech will not accept responsibility
or liability for the accuracy or completeness of, or the presence of any virus or disabling
code in this e-mail"





Mime
View raw message