hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (HADOOP-2094) DFS should not use round robin policy in determing on which volume (file system partition) to allocate for the next block
Date Tue, 18 Mar 2008 04:53:24 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

dhruba borthakur resolved HADOOP-2094.
--------------------------------------

       Resolution: Duplicate
    Fix Version/s: 0.17.0

Duplicate of HADOOP-2559.

> DFS should not use round robin policy in determing on which volume (file system partition)
 to allocate for the next block
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2094
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2094
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: Runping Qi
>            Assignee: dhruba borthakur
>             Fix For: 0.17.0
>
>         Attachments: randomDatanodePartition.patch
>
>
> When multiple file system partitions are configured for the data storage of a data node,
> it uses a strict round robin policy to decide which partition to use for writing the
next block.
> This may result in anormaly cases in which the blocks of a file are not evenly distributed
across 
> the partitions. For example, when we use distcp to copy files with each node have 4 mappers
running concurrently, 
> those 4 mappers are writing to DFS at about the same rate. Thus, it is possible that
the 4 mappers write out
> blocks interleavingly. If there are 4 file system partitions configured for the local
data node, it is possible that each mapper will
> continue to write its blocks on to the same file system partition.
> A simple random placement policy will avoid such anormaly cases, and does not have any
obvious drawbacks.
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message