[ https://issues.apache.org/jira/browse/HDFS-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7461:
-------------------------------
Description:
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of concurrent moves
(default is 5 per node} . Once block moves are scheduled, it will wait for ALL scheduled
moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any (source , target)
slot is available instead of waiting for all the scheduled moves to finish.
was:
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of concurrent moves
(default is 5 per node} . Once block moves are scheduled, it will wait for ALL scheduled
moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any {source , target}
slot is available instead of waiting for all the scheduled moves to finish.
> Reduce impact of laggards on Mover
> ----------------------------------
>
> Key: HDFS-7461
> URL: https://issues.apache.org/jira/browse/HDFS-7461
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
> Affects Versions: 2.6.0
> Reporter: Benoy Antony
> Assignee: Benoy Antony
>
> The current Mover logic is as follows :
> {code}
> for (Path target : targetPaths) {
> hasRemaining |= processPath(target.toUri().getPath());
> }
> // wait for pending move to finish and retry the failed migration
> hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
> {code}
> The _processPath_ will schedule moves, but it is bounded by the number of concurrent
moves (default is 5 per node} . Once block moves are scheduled, it will wait for ALL scheduled
moves to finish in _waitForMoveCompletion_.
> One slow move could keep the Mover idle for a long time.
> It will be a performance improvement to schedule the moves as soon as any (source ,
target) slot is available instead of waiting for all the scheduled moves to finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
|