spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-11740) Fix DStream checkpointing logic to prevent failures during checkpoint recovery
Date Fri, 13 Nov 2015 22:49:11 GMT

     [ https://issues.apache.org/jira/browse/SPARK-11740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-11740:
------------------------------------

    Assignee: Apache Spark

> Fix DStream checkpointing logic to prevent failures during checkpoint recovery
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-11740
>                 URL: https://issues.apache.org/jira/browse/SPARK-11740
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>            Reporter: Shixiong Zhu
>            Assignee: Apache Spark
>
> We will do checkpoint when generating a batch and completing a batch. When the processing
time of a batch is greater than the batch interval, checkpointing for completing an old batch
may run after checkpointing of a new batch. If this happens, checkpoint of an old batch actually
has the latest information, but we won't recovery from it. Then we may see some RDD checkpoint
file missing exception during checkpoint recovery. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message