spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-1730) Make receiver store data reliably to avoid data-loss on executor failures
Date Tue, 06 May 2014 00:17:15 GMT

    [ https://issues.apache.org/jira/browse/SPARK-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990124#comment-13990124
] 

Tathagata Das commented on SPARK-1730:
--------------------------------------

[~hshreedharan] This could also be something you can work on after 1729. The way this can
be done is to add a set of new methods to the Receiver name storeReliably, which will return
true or false based on whether the data was successfully saved or not. And for now the system
will decide true or false after the receiver has reported the name of the block to the driver.
This ignores driver failures and assumes reporting to the driver is sufficient for ensuring
reliable receiving. Once we have a better idea of driver recovery, we can figure out how to
extend this to make it more reliable.

> Make receiver store data reliably to avoid data-loss on executor failures
> -------------------------------------------------------------------------
>
>                 Key: SPARK-1730
>                 URL: https://issues.apache.org/jira/browse/SPARK-1730
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Streaming
>    Affects Versions: 1.0.0
>            Reporter: Tathagata Das
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message