hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johan Gustavsson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6988) Let JHS support different file systems for intermediate_done and done
Date Tue, 24 Oct 2017 02:10:00 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Johan Gustavsson updated MAPREDUCE-6988:
    Attachment: MAPREDUCE-6988.001.patch

This should fix style and test issues

> Let JHS support different file systems for intermediate_done and done
> ---------------------------------------------------------------------
>                 Key: MAPREDUCE-6988
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6988
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobhistoryserver
>    Affects Versions: 2.7.4
>            Reporter: Johan Gustavsson
>            Priority: Minor
>             Fix For: 2.7.5
>         Attachments: MAPREDUCE-6988.000.patch, MAPREDUCE-6988.001.patch
> Currently JHS uses filecontext to move files from intermediate_done to done folder. Since
filecontext limits the use to 1 filesystem it makes it harder to use s3 as a storage for jhist
files. By moving this to filesystem interface we can set hdfs for intermediate storage and
s3 as long term storage therefore reducing the number of puts to s3 and removing the need
for all M/R containers to carry a s3 sdk.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message