hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6638) Jobs with encrypted spills don't recover if the AM goes down
Date Wed, 07 Sep 2016 06:21:21 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15469707#comment-15469707

Karthik Kambatla commented on MAPREDUCE-6638:

We should have two JIRAs for this - (1) Avoid recovering an AM if encrypted spill is enabled,
and (2) Support recovering an AM even when encrypted spill is enabled. I am fine with using
this JIRA for either, but we should file the other one too. 

Comments on the patch:
# The comment for the variable can be simplified to be clear. For instance, "When intermediate
data is encrypted, recovering the job requires access to the key. Until the encryption key
is persisted, we should avoid recovery attempts."
# Can the boolean condition be simplified to {{numReduceTasks <= 0 || shuffleKeyValidForRecovery
|| !spillEncrypted}}? 
# I assume the new method {{recovered()}} is for tests only. Should we annotate it @VisibleForTesting?

> Jobs with encrypted spills don't recover if the AM goes down
> ------------------------------------------------------------
>                 Key: MAPREDUCE-6638
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 2.7.2
>            Reporter: Karthik Kambatla
>            Assignee: Haibo Chen
>            Priority: Critical
>         Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch
> Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be recovered
if the AM fails. We should store the key some place safe so they can actually be recovered.
If there is no "safe" place, at least we should restart the job by re-running all mappers/reducers.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message