hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8468) Umbrella of enhancements to support different failure and locality topologies
Date Fri, 23 Nov 2012 08:01:06 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503100#comment-13503100
] 

Junping Du commented on HADOOP-8468:
------------------------------------

Hi Konstantin, Thanks for your question and carefully reading on the result. Yes. TestDFSIO
has no locality awareness in task scheduling as you said. However, after tasks are scheduled,
the work in this umbrella (let's call it HVE for short) can enhance the possibility for client
to choose local physical host's data block for two reasons: 
1. HVE make sure all replica cross 3 physical hosts (total 6 hosts), so for any HDFS read,
there is 50% chance to have a replica living on the same physical host (previously, it is
between 1/3 - 1/2)
2. With HVE, HDFS client can correctly sort the replicas to have nodegroup-local replica have
priority to be chosen rather than rack-local replica.
The first reason is just special for this case, but second reason affects more general.
Is that make sense?
                
> Umbrella of enhancements to support different failure and locality topologies
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-8468
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8468
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ha, io
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Junping Du
>            Assignee: Junping Du
>         Attachments: HADOOP-8468-total.patch, HADOOP-8468-total-v3.patch, HVE_Hadoop
World Meetup 2012.pptx, HVE User Guide on branch-1(draft ).pdf, Proposal for enchanced failure
and locality topologies.pdf, Proposal for enchanced failure and locality topologies (revised-1.0).pdf
>
>
> The current hadoop network topology (described in some previous issues like: Hadoop-692)
works well in classic three-tiers network when it comes out. However, it does not take into
account other failure models or changes in the infrastructure that can affect network bandwidth
efficiency like: virtualization. 
> Virtualized platform has following genes that shouldn't been ignored by hadoop topology
in scheduling tasks, placing replica, do balancing or fetching block for reading: 
> 1. VMs on the same physical host are affected by the same hardware failure. In order
to match the reliability of a physical deployment, replication of data across two virtual
machines on the same host should be avoided.
> 2. The network between VMs on the same physical host has higher throughput and lower
latency and does not consume any physical switch bandwidth.
> Thus, we propose to make hadoop network topology extend-able and introduce a new level
in the hierarchical topology, a node group level, which maps well onto an infrastructure that
is based on a virtualized environment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message