hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-15548) Randomize local dirs
Date Fri, 29 Jun 2018 20:50:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eric Payne updated HADOOP-15548:
--------------------------------
       Resolution: Fixed
    Fix Version/s: 3.0.4
                   2.8.5
                   2.9.2
                   3.1.1
                   3.2.0
                   2.10.0
           Status: Resolved  (was: Patch Available)

> Randomize local dirs
> --------------------
>
>                 Key: HADOOP-15548
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15548
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jim Brennan
>            Assignee: Jim Brennan
>            Priority: Minor
>             Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 2.8.5, 3.0.4
>
>         Attachments: HADOOP-15548-branch-2.001.patch, HADOOP-15548.001.patch, HADOOP-15548.002.patch
>
>
> shuffle LOCAL_DIRS, LOG_DIRS and LOCAL_USER_DIRS when launching container. Some applications
will process these in exactly the same way in every container (e.g. roundrobin) which can
cause disks to get unnecessarily overloaded (e.g. one output file written to first entry specified
in the environment variable).
> There are two paths for local dir allocation, depending on whether the size is unknown
or known.  The unknown path already uses a random algorithm.  The known path initializes
with a random starting point, and then goes round-robin after that.  When selecting a dir,
it increments the last used by one and then checks sequentially until it finds a dir that
satisfies the request.  Proposal is to increment by a random value of between 1 and num_dirs
- 1, and then check sequentially from there.  This should result in a more random selection
in all cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message