hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benoy Antony (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7461) Reduce impact of laggards on Mover
Date Tue, 02 Dec 2014 02:49:12 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Benoy Antony updated HDFS-7461:
-------------------------------
    Description: 
The current Mover logic is as follows :

{code}
 for (Path target : targetPaths) {
        hasRemaining |= processPath(target.toUri().getPath());
      }
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}

The _processPath_ will schedule moves, but it is bounded by the number of concurrent moves
(default is 5 per node} .  Once block moves are scheduled,  it will wait for ALL  scheduled
moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover  idle for a long time. 

It will be a  performance improvement to schedule the moves as soon as any {source , target}
slot is available instead of waiting for all the scheduled moves to finish. 



  was:
The current Mover logic is as follows :

{code}
 for (Path target : targetPaths) {
        hasRemaining |= processPath(target.toUri().getPath());
      }
      // wait for pending move to finish and retry the failed migration
      hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}

The processPath will schedule moves, but it is bounded by the number of concurrent moves (default
is 5 per node} .  Once block moves are scheduled,  it will wait for ALL  scheduled moves to
finish. 
One slow move could keep the Mover  idle for a long time. 

It will be a  performance improvement to schedule the moves as soon as any {source , target}
slot is available instead f waiting for all the scheduled moves to finish. 




> Reduce impact of laggards on Mover
> ----------------------------------
>
>                 Key: HDFS-7461
>                 URL: https://issues.apache.org/jira/browse/HDFS-7461
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer & mover
>    Affects Versions: 2.6.0
>            Reporter: Benoy Antony
>            Assignee: Benoy Antony
>
> The current Mover logic is as follows :
> {code}
>  for (Path target : targetPaths) {
>         hasRemaining |= processPath(target.toUri().getPath());
>       }
> // wait for pending move to finish and retry the failed migration
> hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
> {code}
> The _processPath_ will schedule moves, but it is bounded by the number of concurrent
moves (default is 5 per node} .  Once block moves are scheduled,  it will wait for ALL  scheduled
moves to finish in _waitForMoveCompletion_.
> One slow move could keep the Mover  idle for a long time. 
> It will be a  performance improvement to schedule the moves as soon as any {source ,
target} slot is available instead of waiting for all the scheduled moves to finish. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message