hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6469) Coordinated replication of the namespace using ConsensusNode
Date Tue, 15 Jul 2014 19:58:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062559#comment-14062559
] 

Sanjay Radia commented on HDFS-6469:
------------------------------------

Todd said:
bq. a fully usable solution would be available to the community at large, whereas the design
you're proposing seems like it will only be usably implemented by a proprietary extension
(I don't consider the ZK "reference implementation" likely to actually work in a usable fashion).

Konstanine I had mentioned exactly the above point to you at the Hadoop summit Europe.  ZK
is a coordination service and for this to be practical it needs to be an inline Paxos protocol.
We had also discussed 2 potential  paxos libraries  that could come into open source: I believe
Facebook has one that they may contribute and CMU has one called E-Paxos; if either of these
become available then it addresses this particular issue. I have no objections to a customer
going to Wandisco for the enterprise supported  version, but if the community is going to
maintain such an extension then there needs to a practical, in-production-usable  free solution;
sending offline messages to a coordinator service  for each transaction is not usable. Lets
discuss the performance part in a separate comment. Let me comment on your comparisons to
 the topology and windows examples that the community supported in the past:
* Topology - these changes allowed Hadoop to be used on containers such as VMs. 
** Both KVM and VirtualBox offer free VM solutions - the customer does not need to buy ESX.
 
** The topology solution would will also help with a Docker container deployment which is
freely available and offers even better performance than VMs. 
** Hadoop is commonly used in cloud environment (e.g. AWS, or Azure, or Altiscale) which all
use VMs or containers
** Further, it was recognized that while, in the past, we had considered racks to be a failure
zone, that there could be other failure zones: nodes (for the case of VMs or containers on
a host) and also groups of machines.
* Windows - this was done for platform support which is very different than what we are talking
about here; many open source solutions support multiple platforms to enable the widest adoption.
BTW Hadoop supported windows via cygwin but we made it first class since the initial support
via cygwin was messy. 

> Coordinated replication of the namespace using ConsensusNode
> ------------------------------------------------------------
>
>                 Key: HDFS-6469
>                 URL: https://issues.apache.org/jira/browse/HDFS-6469
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: CNodeDesign.pdf
>
>
> This is a proposal to introduce ConsensusNode - an evolution of the NameNode, which enables
replication of the namespace on multiple nodes of an HDFS cluster by means of a Coordination
Engine.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message