hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2492) BlockManager cross-rack replication checks only work for ScriptBasedMapping
Date Wed, 26 Oct 2011 07:55:32 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135781#comment-13135781
] 

Steve Loughran commented on HDFS-2492:
--------------------------------------

I'm not sure it's quite so serious, as for the normal script based mapper it works; it looks
for the script file and assumes the presence of the file means cross-rack.

It's only for people who have written their own DNS mapper that the problem will arise, and
the problem isn't topology mapping, its that the replication logic that tries to ensure that
replicas are spread across multiple racks.

The risk then is: if you have a custom DNS mapper, all your blocks may end up on the same
rack, making you less resilient to rack failures. Your data is still spread across machines,
and the actual placement code itself will work to place the blocks in multiple racks. The
only issue is that at a switch or rack failure is slightly more likely to take all replicas
of a block offline until the switch comes back up. 

It's easy to fix though, I will try and do something today.

Before then: who is running a custom DNS Mapper? Do you expect it to work unchanged on 0.23+,
or is adding another method to an interface something you can handle? 

                
> BlockManager cross-rack replication checks only work for ScriptBasedMapping
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-2492
>                 URL: https://issues.apache.org/jira/browse/HDFS-2492
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.23.0, 0.24.0
>            Reporter: Steve Loughran
>            Priority: Blocker
>
> The BlockManager cross-rack replication checks only works if script files are used for
replication, not if alternate plugins provide the topology information.
> This is because the BlockManager sets its rack checking flag if there is a filename key
> {code}
> shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY)
!= null;
> {code}
> yet this filename key is only used if the topology mapper defined by 
> {code}
> DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY
> {code}
> is an instance of {{ScriptBasedMapping}}
> If any other mapper is used, the system may be multi rack, but the Block Manager will
not be aware of this fact unless the filename key is set to something non-null

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message