Mailing-List: contact dev-help@giraph.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@giraph.apache.org
Content-Type: multipart/alternative;
 boundary="===============8986987612555292559=="
MIME-Version: 1.0
Subject: Re: Review Request 14575: Giraph Application Master: move to new and
 stable
 YARN API
From: "Mohammad Islam" <mislam77@yahoo.com>
To: "giraph" <dev@giraph.apache.org>, "Mohammad Islam" <mislam77@yahoo.com>,
 "Eli Reisman" <initialcontext@gmail.com>
Date: Sat, 12 Oct 2013 11:49:10 -0000
Message-ID: <20131012114910.29647.77144@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Mohammad Islam" <noreply@reviews.apache.org>
References: <20131011015242.28842.93533@reviews.apache.org>
In-Reply-To: <20131011015242.28842.93533@reviews.apache.org>
Reply-To: "Mohammad Islam" <mislam77@yahoo.com>

--===============8986987612555292559==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit


> On Oct. 11, 2013, 1:52 a.m., Eli Reisman wrote:
> > I'm +1 for moving to the new YARN APIs. I think with Hadoop 2 on the beta line now, supporting alpha is no longer such a priority.
> 
> Eli Reisman wrote:
>     While I'm +1 for moving forward on the upgrade, I'm having a bit of trouble getting this patch to build. Let me play with it a bit and post a more detailed review in a bit. Thanks Mohammad, looks great!
> 
> Eli Reisman wrote:
>     I'm running this:
>     
>     mvn -Dhadoop.version=2.1.1-SNAPSHOT -Phadoop_yarn install
>     
>     And getting an NPE on org.apache.giraph.io.TestFilters#testEdgeFilter that fails the test. I'm sure I'm doing something dumb wrong. For one thing, I'm on a Mac running Java 1.6_40? Anyway if I'm doing something wrong Mohammad let me know and I'll get this thing reviewed and committed ASAP. Great work!
>     
>     I'll leave a more detailed review once we have this build thing squared away. Thanks!

Eli thanks for looking into this.
The same test passed for me. But I got few non-consistent error. For TestYarnJob, I found exception few times. When add a sleep(20000) just before the run invocation. It worked fine.
I believe all are related to some sort of timing. MiniCluster has this type of inconsistency. 
Please let me know if you want me to check.


- Mohammad


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14575/#review26918
-----------------------------------------------------------


On Oct. 10, 2013, 7:50 a.m., Mohammad Islam wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14575/
> -----------------------------------------------------------
> 
> (Updated Oct. 10, 2013, 7:50 a.m.)
> 
> 
> Review request for giraph.
> 
> 
> Bugs: GIRAPH-737
>     https://issues.apache.org/jira/browse/GIRAPH-737
> 
> 
> Repository: giraph-git
> 
> 
> Description
> -------
> 
> WIP patch. Please give your early comments.
> 
> Key points:
> 
> * Giraph AM using new API and asynchronous/handler model.
> * Adding Kerberos support.
> 
> Copied from the JIRA:
> 
> Giraph was the early adopter of Hadoop YARN AM! Eli successfully wrote a Giraph AM based on Hadoop 2.0.x_alpha. However, in last few months, Yarn significantly overhauled its APIs and associated coding patterns. The new beta version is 2.1.x and I was told by Yarn-dev that current APIs will not change much.
> In the above circumstances, we need to substantially overhaul Giraph AM as well to accommodate with the new Yarn API. Moreover, in newer YARN API, supporting kerberos security in AM becomes easier and more transparent.
> Potential impact:
> The upcoming Girpah AM will not work with earlier alpha Hadoop versions such as 2.0.3. I'm not sure if anyone is using Giraph AM in production. However, the more prevalent way of Giraph processing (MR-based) should continue to work.
> 
> 
> Diffs
> -----
> 
>   giraph-core/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java 00a802f 
>   giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/SaslServerHandler.java 922f373 
>   giraph-core/src/main/java/org/apache/giraph/yarn/GiraphApplicationMaster.java c2b88a0 
>   giraph-core/src/main/java/org/apache/giraph/yarn/GiraphYarnClient.java 341db0e 
>   giraph-core/src/main/java/org/apache/giraph/yarn/YarnUtils.java aa042e8 
>   giraph-hive/src/main/java/org/apache/giraph/hive/output/package-info.java 65d87e3 
>   giraph-hive/src/test/java/org/apache/giraph/hive/input/package-info.java c2327ca 
>   pom.xml 41b6bb1 
> 
> Diff: https://reviews.apache.org/r/14575/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Mohammad Islam
> 
>


--===============8986987612555292559==--