spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Davidson (JIRA)" <>
Subject [jira] [Created] (SPARK-1602) Cancelled jobs can lead to corrupted cached partitions
Date Thu, 24 Apr 2014 02:57:14 GMT
Aaron Davidson created SPARK-1602:

             Summary: Cancelled jobs can lead to corrupted cached partitions
                 Key: SPARK-1602
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.0.0, 0.9.1
            Reporter: Aaron Davidson
            Assignee: Reynold Xin
            Priority: Critical
             Fix For: 1.0.0

When jobs are cancelled, the InterruptibleIterator simply returns hasNext = false, which can
confuse the CacheManager into thinking that the RDD was fully computed, causing it to store
the incomplete result into the BlockManager.

This unfortunately will lead to incorrect results being returned on all future operations
containing this RDD while it's still cached.

This message was sent by Atlassian JIRA

View raw message