hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinayakumar B <vinayakum...@apache.org>
Subject Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk
Date Mon, 05 Oct 2015 07:48:27 GMT
Hi Andrew,
 I see CHANGES.txt entries not yet merged from CHANGES-HDFS-EC-7285.txt.

 Was this intentional?

Regards,
Vinay

On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <andrew.wang@cloudera.com>
wrote:

> Branch has been merged to trunk, thanks again to everyone who worked on the
> feature!
>
> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang <zhezhang@cloudera.com> wrote:
>
> > Thanks everyone who has participated in this discussion.
> >
> > With 7 +1's (5 binding and 2 non-binding), and no -1, this vote has
> passed.
> > I will do a final 'git merge' with trunk and work with Andrew to merge
> the
> > branch to trunk. I'll update on this thread when the merge is done.
> >
> > ---
> > Zhe Zhang
> >
> > On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A <yi.a.liu@intel.com> wrote:
> >
> > > (Change it to binding.)
> > >
> > > +1
> > > I have been involved in the development and code review on the feature
> > > branch. It's a great feature and I think it's ready to merge it into
> > trunk.
> > >
> > > Thanks all for the contribution.
> > >
> > > Regards,
> > > Yi Liu
> > >
> > >
> > > -----Original Message-----
> > > From: Liu, Yi A
> > > Sent: Friday, September 25, 2015 1:51 PM
> > > To: hdfs-dev@hadoop.apache.org
> > > Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk
> > >
> > > +1 (non-binding)
> > > I have been involved in the development and code review on the feature
> > > branch. It's a great feature and I think it's ready to merge it into
> > trunk.
> > >
> > > Thanks all for the contribution.
> > >
> > > Regards,
> > > Yi Liu
> > >
> > >
> > > -----Original Message-----
> > > From: Vinayakumar B [mailto:vinayakumarb@apache.org]
> > > Sent: Friday, September 25, 2015 12:21 PM
> > > To: hdfs-dev@hadoop.apache.org
> > > Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk
> > >
> > > +1,
> > >
> > > I've been involved starting from design and development of
> ErasureCoding.
> > > I think phase 1 of this development is ready to be merged to trunk.
> > > It had come a long way to the current state with significant effort of
> > > many Contributors and Reviewers for both design and code.
> > >
> > > Thanks Everyone for the efforts.
> > >
> > > Regards,
> > > Vinay
> > >
> > > On Wed, Sep 23, 2015 at 10:53 PM, Jing Zhao <jing9@apache.org> wrote:
> > >
> > > > +1
> > > >
> > > > I've been involved in both development and review on the branch, and
> I
> > > > believe it's now ready to get merged into trunk. Many thanks to all
> > > > the contributors and reviewers!
> > > >
> > > > Thanks,
> > > > -Jing
> > > >
> > > > On Tue, Sep 22, 2015 at 6:17 PM, Zheng, Kai <kai.zheng@intel.com>
> > wrote:
> > > >
> > > > > Non-binding +1
> > > > >
> > > > > According to our extensive performance tests, striping + ISA-L
> coder
> > > > based
> > > > > erasure coding not only can save storage, but also can increase the
> > > > > throughput of a client or a cluster. It will be a great addition
to
> > > > > HDFS and its users. Based on the latest branch codes, we also
> > > > > observed it's
> > > > very
> > > > > reliable in the concurrent tests. We'll provide the perf test
> report
> > > > after
> > > > > it's sorted out and hope it helps.
> > > > > Thanks!
> > > > >
> > > > > Regards,
> > > > > Kai
> > > > >
> > > > > -----Original Message-----
> > > > > From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
> > > > > Sent: Wednesday, September 23, 2015 8:50 AM
> > > > > To: hdfs-dev@hadoop.apache.org; common-dev@hadoop.apache.org
> > > > > Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to
> trunk
> > > > >
> > > > > +1
> > > > >
> > > > > Great addition to HDFS. Thanks all contributors for the nice work.
> > > > >
> > > > > Regards,
> > > > > Uma
> > > > >
> > > > > On 9/22/15, 3:40 PM, "Zhe Zhang" <zhezhang@cloudera.com> wrote:
> > > > >
> > > > > >Hi,
> > > > > >
> > > > > >I'd like to propose a vote to merge the HDFS-7285 feature branch
> > > > > >back to trunk. Since November 2014 we have been designing and
> > > > > >developing this feature under the umbrella JIRAs HDFS-7285 and
> > > > > >HADOOP-11264, and have committed approximately 210 patches.
> > > > > >
> > > > > >The HDFS-7285 feature branch was created to support the first
> phase
> > > > > >of HDFS erasure coding (HDFS-EC). The objective of HDFS-EC is
to
> > > > > >significantly reduce storage space usage in HDFS clusters. Instead
> > > > > >of always creating 3 replicas of each block with 200% storage
> space
> > > > > >overhead, HDFS-EC provides data durability through parity data
> > blocks.
> > > > > >With most EC configurations, the storage overhead is no more
than
> > 50%.
> > > > > >Based on profiling results of production clusters, we decided
to
> > > > > >support EC with the striped block layout in the first phase,
so
> > > > > >that small files can be better handled. This means dividing each
> > > > > >logical HDFS file block into smaller units (striping cells) and
> > > > > >spreading them on a set of DataNodes in round-robin fashion.
> Parity
> > > > > >cells are generated for each stripe of original data cells. We
> have
> > > > > >made changes to NameNode, client, and DataNode to generalize
the
> > > > > >block concept and handle the mapping between a logical file block
> > > > > >and its internal storage blocks. For further details please see
> the
> > > > > >design doc on HDFS-7285.
> > > > > >HADOOP-11264 focuses on providing flexible and high-performance
> > > > > >codec calculation support.
> > > > > >
> > > > > >The nightly Jenkins job of the branch has reported several
> > > > > >successful runs, and doesn't show new flaky tests compared with
> > > > > >trunk. We have posted several versions of the test plan including
> > > > > >both unit testing and cluster testing, and have executed most
> tests
> > > > > >in the plan. The most basic functionalities have been extensively
> > > > > >tested and verified in several real clusters with different
> > > > > >hardware configurations; results have been very stable. We have
> > > > > >created follow-on tasks for more advanced error handling and
> > > optimization under the umbrella HDFS-8031.
> > > > > >We also plan to implement or harden the integration of EC with
> > > > > >existing features such as WebHDFS, snapshot, append, truncate,
> > > > > >hflush, hsync, and so forth.
> > > > > >
> > > > > >Development of this feature has been a collaboration across many
> > > > > >companies and institutions. I'd like to thank J. Andreina,
> Takanobu
> > > > > >Asanuma, Vinayakumar B, Li Bo, Takuya Fukudome, Uma Maheswara
Rao
> > > > > >G, Rui Li, Yi Liu, Colin McCabe, Xinwei Qin, Rakesh R, Gao Rui,
> Kai
> > > > > >Sasaki, Walter Su, Tsz Wo Nicholas Sze, Andrew Wang, Yong Zhang,
> > > > > >Jing Zhao, Hui Zheng and Kai Zheng for their code contributions
> and
> > > reviews.
> > > > > >Andrew and Kai Zheng also made fundamental contributions to the
> > > > > >initial design. Rui Li, Gao Rui, Kai Sasaki, Kai Zheng and many
> > > > > >other contributors have made great efforts in system testing.
Many
> > > > > >thanks go to Weihua Jiang for proposing the JIRA, and ATM, Todd
> > > > > >Lipcon, Silvius Rus, Suresh, as well as many others for providing
> > > helpful feedbacks.
> > > > > >
> > > > > >Following the community convention, this vote will last for 7
days
> > > > > >(ending September 29th). Votes from Hadoop committers are binding
> > > > > >but non-binding votes are very welcome as well. And here's my
> > > > > >non-binding
> > > > +1.
> > > > > >
> > > > > >Thanks,
> > > > > >---
> > > > > >Zhe Zhang
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message