hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gera Shegalov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent conf across JobClient, MRAppMaster, and YarnChild
Date Fri, 13 Jun 2014 07:40:02 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14030349#comment-14030349

Gera Shegalov commented on MAPREDUCE-5875:

Hi Robert,

bq. We were trying out v3 of the patch some more and ran into a problem if you configure your
history files (i.e. the "done" directory) to go under /user instead of /tmp

This should be unrelated to the root location /tmp vs /user. It's related to the permissions/umask
used to create the history done dir:
  public static final FsPermission HISTORY_DONE_DIR_PERMISSION =
    FsPermission.createImmutable((short) 0770); 
   * Umask for the done dir and derivatives.
  public static final FsPermission HISTORY_DONE_DIR_UMASK = FsPermission
      .createImmutable((short) (0770 ^ 0777));

Presumably the goal is to prevent users from listing other users job files. Looks like we
can't support job-specified limits when dealing with history if the user does not share the
group with JHS user. 

> Make Counter limits consistent conf across JobClient, MRAppMaster, and YarnChild
> --------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5875
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster, client, task
>    Affects Versions: 2.4.0
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch, MAPREDUCE-5875.v03.patch,
> Currently, counter limits "mapreduce.job.counters.*" handled by {{org.apache.hadoop.mapreduce.counters.Limits}}
are initialized asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max counters/groups is
only increased when needed. With the current Limits implementation relying on static constants,
it's going to be challenging for tools that submit jobs concurrently  without resorting to
class loading isolation.
> The patch that I am uploading is not perfect but demonstrates the issue. 

This message was sent by Atlassian JIRA

View raw message