hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6638) Do not attempt to recover jobs if encrypted spill is enabled
Date Sat, 17 Sep 2016 05:26:20 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498224#comment-15498224

Haibo Chen commented on MAPREDUCE-6638:

Thanks for your comments, Karthik!  Uploading a new patch to address your suggestions, which
does make the logging much more user-friendly. 

bq. One could store the encrypted key in KMS. Once stored, we could do one of the following:
1) Tasks have a way to fetch this key directly    2) Leave the tasks as is, but augment the
AM to recover this key as part of recovery.
I think option 2 is advantageous in that the change needed is minimal and only one query for
each job attempt to the safe store is needed instead of one for each task. 

Will create a follow up jira once this is done.

> Do not attempt to recover jobs if encrypted spill is enabled
> ------------------------------------------------------------
>                 Key: MAPREDUCE-6638
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster
>    Affects Versions: 2.7.2
>            Reporter: Karthik Kambatla
>            Assignee: Haibo Chen
>         Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, mapreduce6638.003.patch
> Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be recovered
if the AM fails. We should store the key some place safe so they can actually be recovered.
If there is no "safe" place, at least we should restart the job by re-running all mappers/reducers.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message