hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nigel Daley <nda...@yahoo-inc.com>
Subject Re: [jira] Commented: (HADOOP-1622) Hadoop should provide a way to allow the user to specify jar file(s) the user job depends on
Date Fri, 20 Jul 2007 18:30:14 GMT
I'm not crazy about this semantics (later jar classes overwrite  
earlier jar classes) because it is the opposite of the Java class  
loader ordering.  If other's want this order, however, it should be  
clearly documented in all the right places (javadoc, script help msg,  

On Jul 20, 2007, at 10:55 AM, Dennis Kubes (JIRA) wrote:

>     [ https://issues.apache.org/jira/browse/HADOOP-1622? 
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment- 
> tabpanel#action_12514266 ]
> Dennis Kubes commented on HADOOP-1622:
> --------------------------------------
> For Nutch development at least (I don't know about others),  it is  
> more useful if classes in later jars overwrite classes in earlier  
> jars.  This will enable someone to do Nutch development, overriding  
> or reworking core classes, without touching the main Nutch source  
> code base.  For many of the Nutch programs, a NutchJob is created  
> that automatically sets the job jar file, which would now be the  
> first jar file.  We wanted to be able to override that when necessary.
>> Hadoop should provide a way to allow the user to specify jar file 
>> (s) the user job depends on
>> --------------------------------------------------------------------- 
>> -----------------------
>>                 Key: HADOOP-1622
>>                 URL: https://issues.apache.org/jira/browse/ 
>> HADOOP-1622
>>             Project: Hadoop
>>          Issue Type: Improvement
>>            Reporter: Runping Qi
>>         Attachments: multipleJobJars.patch
>> More likely than not, a user's job may depend on multiple jars.
>> Right now, when submitting a job through bin/hadoop, there is no  
>> way for the user to specify that.
>> A walk around for that is to re-package all the dependent jars  
>> into a new jar or put the dependent jar files in the lib dir of  
>> the new jar.
>> This walk around causes unnecessary inconvenience to the user.  
>> Furthermore, if the user does not own the main function
>> (like the case when the user uses Aggregate, or datajoin,  
>> streaming), the user has to re-package those system jar files too.
>> It is much desired that hadoop provides a clean and simple way for  
>> the user to specify a list of dependent jar files at the time
>> of job submission. Someting like:
>> bin/hadoop .... --depending_jars j1.jar:j2.jar
> -- 
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.

View raw message