reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Byung-Gon Chun <bgc...@gmail.com>
Subject Re: Need to update pom.xml
Date Tue, 04 Jul 2017 05:24:14 GMT
Hi Seokchan,

You can search some useful information in the mailing list archive. :)
I copied the email Sergiy sent out a while go.

I hope this helps.

===
Hi Markus, Gon,

These are very good questions. Here's what I observe:

1) REEF (including Unmanaged AM functionality) compiles with Hadoop 2.6
jars just fine.

2) All unit tests pass in local mode regardless of Hadoop version that REEF
is compiled with.

3) When compiled against Hadoop 2.6 (i.e. current default), REEF java apps
and unit tests run on YARN only when Hadoop 2.6 jars are in the classpath.
Note that it is still OK to run YARN 2.7+ RM, NM, and other services - we
just need 2.6 jars available to our app at runtime - on all nodes that run
REEF Java parts.

3a) Item #3 is why Todd's application did not run on cisl-slave-002. It was
quite hard to figure out - the app fails at runtime because it cannot find
some method that was available in Hadoop 2.6 but not in 2.7.

4) Our CISL cluster runs Hadoop 2.7.1. If we compile REEF against Hadoop
2.7.3, everything works fine on YARN.

5) I don't know what version of YARN HDInsight uses - I'll ask tomorrow.
What really matters, though, is what version of Hadoop jars we have
available to the apps on HDI.


In other words, it would be much easier to run and debug our apps on cisl
cluster if we compile against hadoop 2.7.3; that would also make Unmanaged
AM mode available without caveats.

Alternatively, we can keep compiling against hadoop 2.6, but we have to
write a warning in the README that the same hadoop 2.6 jars must be
available in the classpath on all YARN nodes, and that YARN 2.7.3+ RM is
required for the Unmanaged AM mode.

Cheers,
Sergiy.
===

On Sun, Jul 2, 2017 at 11:23 PM, 안석찬 <ahnsc93@gmail.com> wrote:

> Hi,
>
> Yarn tests of current REEF requires a method(org.apache.hadoop.
> tracing.SpanReceiverHost.get()) that hadoop 2.7.1~2.7.3 contains. But I
> just recognized pom.xml on the master branch uses hadoop 2.6.0 and it
> generates exceptions.
>
> Yarn tests were successful after I updated my hadoop version on pom.xml as
> 2.7.3.
>
> So, how about updating hadoop version on pom.xml?
>
> Best,
> Seokchan




-- 
Byung-Gon Chun

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message