incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kirk True <>
Subject Re: Chukwa can't find Demux class
Date Wed, 28 Apr 2010 20:16:34 GMT
Hi Eric,

On 4/28/10 10:23 AM, Eric Yang wrote:
> Hi Kirk,
> Is the ownership of the jar file setup correctly as the user that runs
> demux?

When browsing via the NameNode web UI, it lists permissions of 
"rw-r--r--" and "kirk" as the owner (which is also the user ID running 
the Hadoop and Chukwa processes).

>    You may find more information by looking at running mapper task or
> reducer task, and try to find out the task attempt shell script.

Where is the task attempt shell script located?

>    Make sure
> the files are downloaded correctly from distributed cache, and referenced in
> the locally generated jar file.  Hope this helps.

Sorry for asking such basic questions, but where is the locally 
generated JAR file found? I'm assuming under /tmp/hadoop-<user>, by 
default? I saw one file named job_<timstamp>.jar but it appeared to be a 
byte-for-byte copy of chukwa-core-0.4.0.jar, i.e. my "XmlBasedDemux" 
class was nowhere to be found.


> Regards,
> Eric
> On 4/28/10 9:37 AM, "Kirk True"<>  wrote:
>> Hi guys,
>> I have a custom Demux that I need to run to process my input, but I'm getting
>> ClassNotFoundException when running in Hadoop. This is with the released 0.4.0
>> build.
>> I've done the following:
>> 1. I put my Demux class in the correct package
>> (org.apache.hadoop.chukwa.extraction.demux.processor.mapper)
>> 2. I've added the JAR containing the Demux implementation to HDFS at
>> /chuka/demux
>> 3. I've added an alias to it in chukwa-demux-conf.xml
>> The map/reduce job is picking up on the fact that I have a custom Demux and is
>> trying to load it, but I get a ClassNotFoundException. The HDFS-based URL to
>> the JAR is showing up in the job configuration in Hadoop, which is another
>> evidence that Chukwa and Hadoop know where the JAR lives and that it's part of
>> the Chukwa-initiated job.
>> My Demux is very simple. I've stripped it down to a System.out.println with
>> dependencies on no other classes/JARs other than Chukwa, Hadoop, and the core
>> JDK. I've double-checked that my JAR is being built up correctly. I'm
>> completely flummoxed as to what I'm doing wrong.
>> Any ideas what I'm missing? What other information can I provide?
>> Thanks!
>> Kirk

View raw message