hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: How do I include the newer version of Commons-lang in my jar?
Date Mon, 09 Apr 2012 20:53:29 GMT
Sky,

Yes you are right in your summary. Thanks also for reporting back on
the 0.205/1.x issue.

One ugly hack that just comes to mind, would be to manipulate the
classloader at runtime. I've not tried it out to know for sure if its
possible/will work, but just thought I'd note it down.

However, your statement "then, there is not possible for me to use an
external jar such as "commons-lang" from apache in my application. Any
external jars packaged within my jar under "lib" directory are not
captured." needs some refinement - You can plug and send dependent
jars to a job for sure - but if that jar is already present installed
on the cluster (i.e., hadoop uses it via its $HADOOP_HOME/lib/ paths),
then you can't override it unless you work around.

On Tue, Apr 10, 2012 at 2:16 AM, Sky USC <skyusc@hotmail.com> wrote:
>
>
>
>
> Thanks for the reply. I appreciate your helpfulness. I created Jars by following instructions
at "http://blog.mafr.de/2010/07/24/maven-hadoop-job/". So external Jars are stored in lib/
folder within a jar.
>
> Am I summarizing this correctly:
> 1. If hadoop version = 0.20.203 or lower - then, there is not possible for me to use
an external jar such as "commons-lang" from apache in my application. Any external jars packaged
within my jar under "lib" directory are not captured. This appears like a huge limitation
to me?
> 2. If hadoop version >  0.20.204 to 1.0.x - then use  "HADOOP_USER_CLASSPATH_FIRST=true"
environment variable before launching "hadoop jar" might help. I tried this for version 0.20.205
but it didnt work.
> 3. If hadoop version > 2.x or formerly 0.23.x - then this can be set via API?
>
> Is there a working version of testable jar that has these dependencies that I can try
to figure out if its my way of packaging jar or something else??
>
> Thx
>
>> From: harsh@cloudera.com
>> Date: Mon, 9 Apr 2012 13:50:37 +0530
>> Subject: Re: How do I include the newer version of Commons-lang in my jar?
>> To: common-user@hadoop.apache.org
>>
>> Answer is a bit messy.
>>
>> Perhaps you can set the environment variable "export
>> HADOOP_USER_CLASSPATH_FIRST=true" before you do a "hadoop jar …" to
>> launch your job. However, although this approach is present in
>> 0.20.204+ (0.20.205, and 1.0.x), am not sure if it makes an impact on
>> the tasks as well. I don't see it changing anything but for the driver
>> CP. I've not tested it - please let us know if it works in your
>> environment.
>>
>> In higher versions (2.x or formerly 0.23.x), this is doable from
>> within your job if you set "mapreduce.job.user.classpath.first" to
>> true inside your job, and ship your replacement jars along.
>>
>> Some versions would also let you set this via
>> "JobConf/Job.setUserClassesTakesPrecedence(true/false)" API calls.
>>
>> On Mon, Apr 9, 2012 at 11:14 AM, Sky <skyusc@hotmail.com> wrote:
>> > Hi.
>> >
>> > I am new to Hadoop and I am working on project on AWS Elastic MapReduce.
>> >
>> > The problem I am facing is:
>> > * org.apache.commons.lang.time.DateUtils: parseDate() works OK but
>> > parseDateStrictly() fails.
>> > I think parseDateStrictly might be new in lang 2.5. I thought I included all
>> > dependencies. However, for some reason, during runtime, my app is not
>> > picking up the newer commons-lang.
>> >
>> > Would love some help.
>> >
>> > Thx
>> > - sky
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Mime
View raw message