hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4877) AM doesn't properly support multiple NNs
Date Wed, 16 Apr 2014 21:20:19 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Daryn Sharp updated MAPREDUCE-4877:

    Target Version/s: 3.0.0, 2.5.0  (was: 3.0.0, 0.23.11)

> AM doesn't properly support multiple NNs
> ----------------------------------------
>                 Key: MAPREDUCE-4877
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4877
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster, job submission
>    Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
> Yarn/MR clusters assume there's a 1-to-1 correspondence between itself and a NN.  Certain
internal paths like the staging dir, job history, intermediate/intermediate-done dirs are
resolved relative to the defaultFS.  The JT used the host's conf which ensured the correct/expected
NN.  However the AM uses the user's job conf, which means the user's defined defaultFS can
cause the job to use incorrect paths.
> Typically the output path's NN is also the yarn cluster's NN.  However problems occur
when a yarn cluster is servicing multiple NN's (ex. federated clusters).  The JHS is assuming
the AM will write to NN1, whereas the user's job conf may be using a defaultFS of NN2 or NN3
which influences where the AM writes.

This message was sent by Atlassian JIRA

View raw message