Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <10514476.1170359045833.JavaMail.jira@brutus>
Date: Thu, 1 Feb 2007 11:44:05 -0800 (PST)
From: "Dennis Kubes (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-964) ClassNotFoundException in
 ReduceTaskRunner
In-Reply-To: <23361462.1170287345512.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469560 ] 

Dennis Kubes commented on HADOOP-964:
-------------------------------------

Are we going to be able to move all comparator access int the child process.  This is occurring during the sorting and merging.  Would that process have to be moved to the child?

The short term fix for loading the classes into the ReduceTaskRunner is classpath2.path, should I change this to occur in TaskTracker?  It would essentially just be moving the method over to the TaskTracker and changing where it gets called.

I will create a unit test for the comparator in the jar file now , the new bug has already been created as HADOOP-968.

> ClassNotFoundException in ReduceTaskRunner
> ------------------------------------------
>
>                 Key: HADOOP-964
>                 URL: https://issues.apache.org/jira/browse/HADOOP-964
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>         Environment: windows xp and fedora core 6 linux, java 1.5.10...should affect all systems
>            Reporter: Dennis Kubes
>            Priority: Blocker
>             Fix For: 0.11.0
>
>         Attachments: classpath.patch, classpath2.path
>
>
> In the ReduceTaskRunner constructor lin 339 a sorter is created that attempts to get the map output key and value classes from the configuration object.  This is before the TaskTracker$Child process is spawned off into into own separate JVM so here the classpath for the configuration is the classpath that started the TaskTracker.  The current hadoop script includes the hadoop jars, meaning that any hadoop writable type will be found, but it doesn't include nutch jars  so any nutch writable type or any other writable type will not be found and will throw a ClassNotFoundException.
> I don't think it is a good idea to have a dependecy on specific Nutch jars in the Hadoop script but it is a good idea to allow jars to be included if they are in specific locations, such as the HADOOP_HOME where the nutch jar resides.  I have attached a patch that adds any jars in the HADOOP_HOME directory to the hadoop classpath.  This fixes the issues with getting ClassNotFoundExceptions inside of Nutch processes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.