Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Fri, 12 Feb 2016 00:17:18 +0000 (UTC)
From: "Vinod Kumar Vavilapalli (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12848899.1437745240000.31895.1455236238576@Atlassian.JIRA>
In-Reply-To: <JIRA.12848899.1437745240000@Atlassian.JIRA>
References: <JIRA.12848899.1437745240000@Atlassian.JIRA>
 <JIRA.12848899.1437745240695@arcas>
Subject: [jira] [Commented] (MAPREDUCE-6608) Work Preserving AM Restart for
 MapReduce
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143761#comment-15143761 ] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-6608:
----------------------------------------------------

bq. I agree that storing state in zookeeper may have scalability issues. I am just thinking that will it be ended up having too many small files in hdfs if we are planning to store AM information in HDFS.
A solution for this is already given at YARN-1489 by [~bikassaha]. See this comment: https://issues.apache.org/jira/browse/YARN-1489?focusedCommentId=13862359&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13862359.

The solution is essentially a combination of registry with YARN acting as a distributed readers solution: Registry owns the write path and storage, RM/NMs take care of providing scalable reads.

> Work Preserving AM Restart for MapReduce
> ----------------------------------------
>
>                 Key: MAPREDUCE-6608
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6608
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Srikanth Sampath
>            Assignee: Srikanth Sampath
>         Attachments: Patch1.patch, WorkPreservingMRAppMaster-1.pdf, WorkPreservingMRAppMaster-2.pdf, WorkPreservingMRAppMaster.pdf
>
>
> Providing a framework for work preserving AM is achieved in [YARN-1489|https://issues.apache.org/jira/browse/YARN-1489].  We would like to take advantage of this for MapReduce(MR) applications.  There are some challenges which have been described in the attached document and few options discussed.  We solicit feedback from the community.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)