Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: 
 <CAN0bKpJNVQbdCuOro_WkOLP-6iCdSnWFhbNqimy3yDc1LBLeYg@mail.gmail.com>
References: 
 <CAELCxnyyVjTfVg+1hjWm4N6GK1ZWqLZFKS=KpCsJayk3C4+QPQ@mail.gmail.com>
	<CAN0bKpJNVQbdCuOro_WkOLP-6iCdSnWFhbNqimy3yDc1LBLeYg@mail.gmail.com>
Date: Thu, 28 Jan 2016 13:54:07 -0800
Message-ID: 
 <CAELCxnyX529C_i=7zNr5Rc5Py_=hNLLM+YCKhk3NyStTFG=Nkg@mail.gmail.com>
Subject: Re: Hadoop library version issues and conflict
From: Liren Ding <sky.gonna.bright@gmail.com>
To: Solomon Duskis <sduskis@gmail.com>
Cc: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a114389d4359397052a6bf5f5

--001a114389d4359397052a6bf5f5
Content-Type: text/plain; charset=UTF-8

I saw a few posts mentioned the property "mapreduce.job.user.classpath.first"
could be used to handle this situation. But it seems to have been
deprecated in yarn. Does anyone know any alternative?


On Thu, Jan 28, 2016 at 12:19 PM, Solomon Duskis <sduskis@gmail.com> wrote:

> I'm not sure about the solution here, but it would be nice to have much
> newer dependencies of everything.  There are a host of issues with newer
> libraries running in the same VM as hbase and hadoop.  Even the newer hbase
> libraries are 3+ year old.
>
> On Thu, Jan 28, 2016 at 2:33 PM, Liren Ding <sky.gonna.bright@gmail.com>
> wrote:
>
>> My MR job use following libs:
>>
>>    - commons-codec-1.9,
>>    - commons-compress-1.7,
>>    - guava-12.0.1,
>>    - httpclient-4.3.1
>>    - httpcore-4.
>>
>> But the versions in hadoop libs are:
>>
>>    - commons-codec-1.4
>>    - commons-compress-1.4.1
>>    - guava-11.0.2
>>    - httpclient-4.2.5
>>    - httpcore-4.2.5
>>
>> This conflict causes a "java.lang.NoSuchMethodError" error. One insane
>> solution is replacing all libs allover the cluster with new versions. But
>> the approach is not practical in production environment. Is there a decent
>> way to solve the hadoop libraries version conflict issue?
>>
>
>

--001a114389d4359397052a6bf5f5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I saw a few posts mentioned the property &quot;<span style=
=3D"color:rgb(0,0,255);font-family:Consolas,&#39;Bitstream Vera Sans Mono&#=
39;,&#39;Courier New&#39;,Courier,monospace;font-size:12px;line-height:13.2=
px;white-space:pre">mapreduce.job.user.classpath.first</span>&quot; could b=
e used to handle this situation. But it seems to have been deprecated in ya=
rn. Does anyone know any alternative?<div><br></div></div><div class=3D"gma=
il_extra"><br><div class=3D"gmail_quote">On Thu, Jan 28, 2016 at 12:19 PM, =
Solomon Duskis <span dir=3D"ltr">&lt;<a href=3D"mailto:sduskis@gmail.com" t=
arget=3D"_blank">sduskis@gmail.com</a>&gt;</span> wrote:<br><blockquote cla=
ss=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa=
dding-left:1ex"><div dir=3D"ltr">I&#39;m not sure about the solution here, =
but it would be nice to have much newer dependencies of everything.=C2=A0 T=
here are a host of issues with newer libraries running in the same VM as hb=
ase and hadoop.=C2=A0 Even the newer hbase libraries are 3+ year old.</div>=
<div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br><div=
 class=3D"gmail_quote">On Thu, Jan 28, 2016 at 2:33 PM, Liren Ding <span di=
r=3D"ltr">&lt;<a href=3D"mailto:sky.gonna.bright@gmail.com" target=3D"_blan=
k">sky.gonna.bright@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D=
"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding=
-left:1ex"><div dir=3D"ltr"><p style=3D"margin:0px 0px 1em;padding:0px;bord=
er:0px;font-size:15px;clear:both;color:rgb(34,36,38);font-family:Arial,&#39=
;Helvetica Neue&#39;,Helvetica,sans-serif;line-height:15.6px">My MR job use=
 following libs:</p><ul style=3D"margin:0px 0px 1em 30px;padding:0px;border=
:0px;font-size:15px;color:rgb(34,36,38);font-family:Arial,&#39;Helvetica Ne=
ue&#39;,Helvetica,sans-serif;line-height:15.6px"><li style=3D"margin:0px 0p=
x 0.5em;padding:0px;border:0px;word-wrap:break-word">commons-codec-1.9,</li=
><li style=3D"margin:0px 0px 0.5em;padding:0px;border:0px;word-wrap:break-w=
ord">commons-compress-1.7,</li><li style=3D"margin:0px 0px 0.5em;padding:0p=
x;border:0px;word-wrap:break-word">guava-12.0.1,</li><li style=3D"margin:0p=
x 0px 0.5em;padding:0px;border:0px;word-wrap:break-word">httpclient-4.3.1</=
li><li style=3D"margin:0px;padding:0px;border:0px;word-wrap:break-word">htt=
pcore-4.</li></ul><p style=3D"margin:0px 0px 1em;padding:0px;border:0px;fon=
t-size:15px;clear:both;color:rgb(34,36,38);font-family:Arial,&#39;Helvetica=
 Neue&#39;,Helvetica,sans-serif;line-height:15.6px">But the versions in had=
oop libs are:</p><ul style=3D"margin:0px 0px 1em 30px;padding:0px;border:0p=
x;font-size:15px;color:rgb(34,36,38);font-family:Arial,&#39;Helvetica Neue&=
#39;,Helvetica,sans-serif;line-height:15.6px"><li style=3D"margin:0px 0px 0=
.5em;padding:0px;border:0px;word-wrap:break-word">commons-codec-1.4</li><li=
 style=3D"margin:0px 0px 0.5em;padding:0px;border:0px;word-wrap:break-word"=
>commons-compress-1.4.1</li><li style=3D"margin:0px 0px 0.5em;padding:0px;b=
order:0px;word-wrap:break-word">guava-11.0.2</li><li style=3D"margin:0px 0p=
x 0.5em;padding:0px;border:0px;word-wrap:break-word">httpclient-4.2.5</li><=
li style=3D"margin:0px;padding:0px;border:0px;word-wrap:break-word">httpcor=
e-4.2.5</li></ul><p style=3D"margin:0px 0px 1em;padding:0px;border:0px;font=
-size:15px;clear:both;color:rgb(34,36,38);font-family:Arial,&#39;Helvetica =
Neue&#39;,Helvetica,sans-serif;line-height:15.6px">This conflict causes a &=
quot;java.lang.NoSuchMethodError&quot; error. One insane solution is replac=
ing all libs allover the cluster with new versions. But the approach is not=
 practical in production environment. Is there a decent way to solve the ha=
doop libraries version conflict issue?</p></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a114389d4359397052a6bf5f5--