Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@spark.apache.org
Date: Tue, 6 May 2014 00:17:15 +0000 (UTC)
From: "Tathagata Das (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12712458.1399333518519.233524.1399335435307@arcas>
In-Reply-To: <JIRA.12712458.1399333518519@arcas>
References: <JIRA.12712458.1399333518519@arcas>
Subject: [jira] [Commented] (SPARK-1730) Make receiver store data reliably
 to avoid data-loss on executor failures
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/SPARK-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990124#comment-13990124 ] 

Tathagata Das commented on SPARK-1730:
--------------------------------------

[~hshreedharan] This could also be something you can work on after 1729. The way this can be done is to add a set of new methods to the Receiver name storeReliably, which will return true or false based on whether the data was successfully saved or not. And for now the system will decide true or false after the receiver has reported the name of the block to the driver. This ignores driver failures and assumes reporting to the driver is sufficient for ensuring reliable receiving. Once we have a better idea of driver recovery, we can figure out how to extend this to make it more reliable.

> Make receiver store data reliably to avoid data-loss on executor failures
> -------------------------------------------------------------------------
>
>                 Key: SPARK-1730
>                 URL: https://issues.apache.org/jira/browse/SPARK-1730
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Streaming
>    Affects Versions: 1.0.0
>            Reporter: Tathagata Das
>


--
This message was sent by Atlassian JIRA
(v6.2#6252)