hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhe Zhang <zhezh...@cloudera.com>
Subject Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Date Mon, 02 Nov 2015 21:33:34 GMT
Thanks Vinod for the proposal and Andrew/Jing for the comments. 2.9 sounds
a good plan. As Andrew pointed out, 2.8 is already quite big. And even when
disabled, EC logic has been baked in NN pretty deeply.

Do we have a tentative date or estimate for 2.9?

---
Zhe Zhang

On Mon, Nov 2, 2015 at 1:22 PM, Jing Zhao <jing9@apache.org> wrote:

> Thanks for the discussion, Vinod and Andrew. Backporting EC to 2.9 sounds
> good to me.
>
> On Mon, Nov 2, 2015 at 12:06 PM, Andrew Wang <andrew.wang@cloudera.com>
> wrote:
>
> > Thanks for forking the thread Vinod,
> >
> > SGTM, though I really do recommend waiting for 2.9 given the current size
> > of 2.8. I'm not a fan of an "off by default" half-measure, since it
> doesn't
> > change our compatibility requirements, and there's some major NN surgery
> > that can't really be disabled.
> >
> > If we do find a major user who's backported this to their own branch-2
> > fork, I agree that's motivation to get it in an upstream release
> quicker. I
> > haven't heard anything along these lines though.
> >
> > On Mon, Nov 2, 2015 at 11:49 AM, Vinod Vavilapalli <
> > vinodkv@hortonworks.com>
> > wrote:
> >
> > > Forking the thread. Started looking at the 2.8 list, various features’
> > > status and arrived here.
> > >
> > > While I understand the pervasive nature of EC and a need for a
> > significant
> > > bake-in, moving this to a 3.x release is not a good idea. We will
> surely
> > > get a 2.8 out this year and, as needed, I can even spend time getting
> > > started on a 2.9. OTOH, 3.x is long ways off, and given all the
> > > incompatibilities there, it would be a while before users can get their
> > > hands on EC if it were to be only on 3.x. At best, this may force sites
> > > that want EC to backport the entire EC feature to older releases, at
> > worst
> > > this will be repeat the mess of 0.20 security release forks.
> > >
> > > If we think adding this to 2.8 (even if it switched off) is too much
> risk
> > > per our original plan, let’s move this to 2.9, there by leaving enough
> > time
> > > for stability, integration testing and bake-in, and a realistic chance
> of
> > > having it end up on users’ clusters soonish.
> > >
> > > +Vinod
> > >
> > > > On Oct 19, 2015, at 1:44 PM, Andrew Wang <andrew.wang@cloudera.com>
> > > wrote:
> > > >
> > > > I think our plan thus far has been to target this for 3.0. I'm okay
> > with
> > > > putting it in branch-2 if we've given a hard look at compatibility,
> but
> > > > I'll note though that 2.8 is already looking like quite a large
> > release,
> > > > and our release bandwidth has been focused on the 2.6 and 2.7
> > maintenance
> > > > releases. Adding another multi-hundred JIRAs to 2.8 might make it too
> > > > unwieldy to get out the door. If we bump EC past that, 3.0 might very
> > > well
> > > > be our next release vehicle. I do plan to revive the 3.0 schedule
> some
> > > time
> > > > next year. With EC and JDK8 in a good spot, the only big feature
> > > remaining
> > > > is classpath isolation.
> > > >
> > > > EC is also a pretty fundamental change to HDFS. Even if it's
> > compatible,
> > > in
> > > > terms of size and impact it might best belong in a new major release.
> > > >
> > > > Best,
> > > > Andrew
> > > >
> > > > On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
> > > > vinayakumarb.apache@gmail.com> wrote:
> > > >
> > > >> Is anyone else also thinks that feature is ready to goto branch-2
> as
> > > well?
> > > >>
> > > >> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since
> > then
> > > and
> > > >> ready to go in branch-2.
> > > >>
> > > >> -Vinay
> > > >> On Oct 6, 2015 12:51 AM, "Zhe Zhang" <zhezhang@cloudera.com>
wrote:
> > > >>
> > > >>> Thanks Vinay for capturing the issue and Uma for offering the
help.
> > > >>>
> > > >>> ---
> > > >>> Zhe Zhang
> > > >>>
> > > >>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
> > > >> uma.gangumalla@intel.com
> > > >>>>
> > > >>> wrote:
> > > >>>
> > > >>>> Vinay,
> > > >>>>
> > > >>>>
> > > >>>> I would merge them as part of HDFS-9182.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Uma
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On 10/5/15, 12:48 AM, "Vinayakumar B" <vinayakumarb@apache.org>
> > > wrote:
> > > >>>>
> > > >>>>> Hi Andrew,
> > > >>>>> I see CHANGES.txt entries not yet merged from
> > > >> CHANGES-HDFS-EC-7285.txt.
> > > >>>>>
> > > >>>>> Was this intentional?
> > > >>>>>
> > > >>>>> Regards,
> > > >>>>> Vinay
> > > >>>>>
> > > >>>>> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
> > > >> andrew.wang@cloudera.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Branch has been merged to trunk, thanks again to everyone
who
> > worked
> > > >>> on
> > > >>>>>> the
> > > >>>>>> feature!
> > > >>>>>>
> > > >>>>>> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang <
> > zhezhang@cloudera.com>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Thanks everyone who has participated in this discussion.
> > > >>>>>>>
> > > >>>>>>> With 7 +1's (5 binding and 2 non-binding), and
no -1, this vote
> > > >> has
> > > >>>>>> passed.
> > > >>>>>>> I will do a final 'git merge' with trunk and work
with Andrew
> to
> > > >>> merge
> > > >>>>>> the
> > > >>>>>>> branch to trunk. I'll update on this thread when
the merge is
> > > >> done.
> > > >>>>>>>
> > > >>>>>>> ---
> > > >>>>>>> Zhe Zhang
> > > >>>>>>>
> > > >>>>>>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A <
> yi.a.liu@intel.com>
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> (Change it to binding.)
> > > >>>>>>>>
> > > >>>>>>>> +1
> > > >>>>>>>> I have been involved in the development and
code review on the
> > > >>>>>> feature
> > > >>>>>>>> branch. It's a great feature and I think it's
ready to merge
> it
> > > >>> into
> > > >>>>>>> trunk.
> > > >>>>>>>>
> > > >>>>>>>> Thanks all for the contribution.
> > > >>>>>>>>
> > > >>>>>>>> Regards,
> > > >>>>>>>> Yi Liu
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> -----Original Message-----
> > > >>>>>>>> From: Liu, Yi A
> > > >>>>>>>> Sent: Friday, September 25, 2015 1:51 PM
> > > >>>>>>>> To: hdfs-dev@hadoop.apache.org
> > > >>>>>>>> Subject: RE: [VOTE] Merge HDFS-7285 (erasure
coding) branch to
> > > >>> trunk
> > > >>>>>>>>
> > > >>>>>>>> +1 (non-binding)
> > > >>>>>>>> I have been involved in the development and
code review on the
> > > >>>>>> feature
> > > >>>>>>>> branch. It's a great feature and I think it's
ready to merge
> it
> > > >>> into
> > > >>>>>>> trunk.
> > > >>>>>>>>
> > > >>>>>>>> Thanks all for the contribution.
> > > >>>>>>>>
> > > >>>>>>>> Regards,
> > > >>>>>>>> Yi Liu
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> -----Original Message-----
> > > >>>>>>>> From: Vinayakumar B [mailto:vinayakumarb@apache.org]
> > > >>>>>>>> Sent: Friday, September 25, 2015 12:21 PM
> > > >>>>>>>> To: hdfs-dev@hadoop.apache.org
> > > >>>>>>>> Subject: Re: [VOTE] Merge HDFS-7285 (erasure
coding) branch to
> > > >>> trunk
> > > >>>>>>>>
> > > >>>>>>>> +1,
> > > >>>>>>>>
> > > >>>>>>>> I've been involved starting from design and
development of
> > > >>>>>> ErasureCoding.
> > > >>>>>>>> I think phase 1 of this development is ready
to be merged to
> > > >>> trunk.
> > > >>>>>>>> It had come a long way to the current state
with significant
> > > >>> effort
> > > >>>>>> of
> > > >>>>>>>> many Contributors and Reviewers for both design
and code.
> > > >>>>>>>>
> > > >>>>>>>> Thanks Everyone for the efforts.
> > > >>>>>>>>
> > > >>>>>>>> Regards,
> > > >>>>>>>> Vinay
> > > >>>>>>>>
> > > >>>>>>>> On Wed, Sep 23, 2015 at 10:53 PM, Jing Zhao
<jing9@apache.org
> >
> > > >>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> +1
> > > >>>>>>>>>
> > > >>>>>>>>> I've been involved in both development
and review on the
> > > >> branch,
> > > >>>>>> and
> > > >>>>>> I
> > > >>>>>>>>> believe it's now ready to get merged into
trunk. Many thanks
> > > >> to
> > > >>>>>> all
> > > >>>>>>>>> the contributors and reviewers!
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> -Jing
> > > >>>>>>>>>
> > > >>>>>>>>> On Tue, Sep 22, 2015 at 6:17 PM, Zheng,
Kai <
> > > >>> kai.zheng@intel.com>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Non-binding +1
> > > >>>>>>>>>>
> > > >>>>>>>>>> According to our extensive performance
tests, striping +
> > > >> ISA-L
> > > >>>>>> coder
> > > >>>>>>>>> based
> > > >>>>>>>>>> erasure coding not only can save storage,
but also can
> > > >>> increase
> > > >>>>>> the
> > > >>>>>>>>>> throughput of a client or a cluster.
It will be a great
> > > >>>>>> addition to
> > > >>>>>>>>>> HDFS and its users. Based on the latest
branch codes, we
> > > >> also
> > > >>>>>>>>>> observed it's
> > > >>>>>>>>> very
> > > >>>>>>>>>> reliable in the concurrent tests.
We'll provide the perf
> > > >> test
> > > >>>>>> report
> > > >>>>>>>>> after
> > > >>>>>>>>>> it's sorted out and hope it helps.
> > > >>>>>>>>>> Thanks!
> > > >>>>>>>>>>
> > > >>>>>>>>>> Regards,
> > > >>>>>>>>>> Kai
> > > >>>>>>>>>>
> > > >>>>>>>>>> -----Original Message-----
> > > >>>>>>>>>> From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
> > > >>>>>>>>>> Sent: Wednesday, September 23, 2015
8:50 AM
> > > >>>>>>>>>> To: hdfs-dev@hadoop.apache.org;
> > > >> common-dev@hadoop.apache.org
> > > >>>>>>>>>> Subject: Re: [VOTE] Merge HDFS-7285
(erasure coding) branch
> > > >> to
> > > >>>>>> trunk
> > > >>>>>>>>>>
> > > >>>>>>>>>> +1
> > > >>>>>>>>>>
> > > >>>>>>>>>> Great addition to HDFS. Thanks all
contributors for the nice
> > > >>>>>> work.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Regards,
> > > >>>>>>>>>> Uma
> > > >>>>>>>>>>
> > > >>>>>>>>>> On 9/22/15, 3:40 PM, "Zhe Zhang" <zhezhang@cloudera.com>
> > > >>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'd like to propose a vote to
merge the HDFS-7285 feature
> > > >>>>>> branch
> > > >>>>>>>>>>> back to trunk. Since November
2014 we have been designing
> > > >> and
> > > >>>>>>>>>>> developing this feature under
the umbrella JIRAs HDFS-7285
> > > >>> and
> > > >>>>>>>>>>> HADOOP-11264, and have committed
approximately 210 patches.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> The HDFS-7285 feature branch was
created to support the
> > > >> first
> > > >>>>>> phase
> > > >>>>>>>>>>> of HDFS erasure coding (HDFS-EC).
The objective of HDFS-EC
> > > >> is
> > > >>>>>> to
> > > >>>>>>>>>>> significantly reduce storage space
usage in HDFS clusters.
> > > >>>>>> Instead
> > > >>>>>>>>>>> of always creating 3 replicas
of each block with 200%
> > > >> storage
> > > >>>>>> space
> > > >>>>>>>>>>> overhead, HDFS-EC provides data
durability through parity
> > > >>> data
> > > >>>>>>> blocks.
> > > >>>>>>>>>>> With most EC configurations, the
storage overhead is no
> > > >> more
> > > >>>>>> than
> > > >>>>>>> 50%.
> > > >>>>>>>>>>> Based on profiling results of
production clusters, we
> > > >> decided
> > > >>>>>> to
> > > >>>>>>>>>>> support EC with the striped block
layout in the first
> > > >> phase,
> > > >>> so
> > > >>>>>>>>>>> that small files can be better
handled. This means dividing
> > > >>>>>> each
> > > >>>>>>>>>>> logical HDFS file block into smaller
units (striping cells)
> > > >>> and
> > > >>>>>>>>>>> spreading them on a set of DataNodes
in round-robin
> > > >> fashion.
> > > >>>>>> Parity
> > > >>>>>>>>>>> cells are generated for each stripe
of original data cells.
> > > >>> We
> > > >>>>>> have
> > > >>>>>>>>>>> made changes to NameNode, client,
and DataNode to
> > > >> generalize
> > > >>>>>> the
> > > >>>>>>>>>>> block concept and handle the mapping
between a logical file
> > > >>>>>> block
> > > >>>>>>>>>>> and its internal storage blocks.
For further details please
> > > >>> see
> > > >>>>>> the
> > > >>>>>>>>>>> design doc on HDFS-7285.
> > > >>>>>>>>>>> HADOOP-11264 focuses on providing
flexible and
> > > >>> high-performance
> > > >>>>>>>>>>> codec calculation support.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> The nightly Jenkins job of the
branch has reported several
> > > >>>>>>>>>>> successful runs, and doesn't show
new flaky tests compared
> > > >>> with
> > > >>>>>>>>>>> trunk. We have posted several
versions of the test plan
> > > >>>>>> including
> > > >>>>>>>>>>> both unit testing and cluster
testing, and have executed
> > > >> most
> > > >>>>>> tests
> > > >>>>>>>>>>> in the plan. The most basic functionalities
have been
> > > >>>>>> extensively
> > > >>>>>>>>>>> tested and verified in several
real clusters with different
> > > >>>>>>>>>>> hardware configurations; results
have been very stable. We
> > > >>> have
> > > >>>>>>>>>>> created follow-on tasks for more
advanced error handling
> > > >> and
> > > >>>>>>>> optimization under the umbrella HDFS-8031.
> > > >>>>>>>>>>> We also plan to implement or harden
the integration of EC
> > > >>> with
> > > >>>>>>>>>>> existing features such as WebHDFS,
snapshot, append,
> > > >>> truncate,
> > > >>>>>>>>>>> hflush, hsync, and so forth.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Development of this feature has
been a collaboration across
> > > >>>>>> many
> > > >>>>>>>>>>> companies and institutions. I'd
like to thank J. Andreina,
> > > >>>>>> Takanobu
> > > >>>>>>>>>>> Asanuma, Vinayakumar B, Li Bo,
Takuya Fukudome, Uma
> > > >> Maheswara
> > > >>>>>> Rao
> > > >>>>>>>>>>> G, Rui Li, Yi Liu, Colin McCabe,
Xinwei Qin, Rakesh R, Gao
> > > >>> Rui,
> > > >>>>>> Kai
> > > >>>>>>>>>>> Sasaki, Walter Su, Tsz Wo Nicholas
Sze, Andrew Wang, Yong
> > > >>>>>> Zhang,
> > > >>>>>>>>>>> Jing Zhao, Hui Zheng and Kai Zheng
for their code
> > > >>> contributions
> > > >>>>>> and
> > > >>>>>>>> reviews.
> > > >>>>>>>>>>> Andrew and Kai Zheng also made
fundamental contributions to
> > > >>> the
> > > >>>>>>>>>>> initial design. Rui Li, Gao Rui,
Kai Sasaki, Kai Zheng and
> > > >>> many
> > > >>>>>>>>>>> other contributors have made great
efforts in system
> > > >> testing.
> > > >>>>>> Many
> > > >>>>>>>>>>> thanks go to Weihua Jiang for
proposing the JIRA, and ATM,
> > > >>> Todd
> > > >>>>>>>>>>> Lipcon, Silvius Rus, Suresh, as
well as many others for
> > > >>>>>> providing
> > > >>>>>>>> helpful feedbacks.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Following the community convention,
this vote will last
> > > >> for 7
> > > >>>>>> days
> > > >>>>>>>>>>> (ending September 29th). Votes
from Hadoop committers are
> > > >>>>>> binding
> > > >>>>>>>>>>> but non-binding votes are very
welcome as well. And here's
> > > >> my
> > > >>>>>>>>>>> non-binding
> > > >>>>>>>>> +1.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks,
> > > >>>>>>>>>>> ---
> > > >>>>>>>>>>> Zhe Zhang
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message