hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng, Kai" <kai.zh...@intel.com>
Subject RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Date Mon, 02 Nov 2015 23:21:49 GMT
Sounds good to me. When it's determined to include EC in 2.9 release, it may be good to have
a rough release date as Zhe asked, so accordingly the scope of EC can be discussed out. We
still have quite a few of things as Phase I follow-on tasks to do before EC can be deployed
in a production system. Phase II to develop non-striping EC for cold data would possibly be
started after that. We might consider to include only Phase I and leave Phase II for next
release according to the rough release date.

Regards,
Kai

-----Original Message-----
From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com] 
Sent: Tuesday, November 03, 2015 5:41 AM
To: hdfs-dev@hadoop.apache.org
Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch
to trunk]

+1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan 
+to
have 2.8 and 2.9 releases.

Regards,
Uma

On 11/2/15, 11:49 AM, "Vinod Vavilapalli" <vinodkv@hortonworks.com> wrote:

>Forking the thread. Started looking at the 2.8 list, various features¹ 
>status and arrived here.
>
>While I understand the pervasive nature of EC and a need for a 
>significant bake-in, moving this to a 3.x release is not a good idea. 
>We will surely get a 2.8 out this year and, as needed, I can even spend 
>time getting started on a 2.9. OTOH, 3.x is long ways off, and given 
>all the incompatibilities there, it would be a while before users can 
>get their hands on EC if it were to be only on 3.x. At best, this may 
>force sites that want EC to backport the entire EC feature to older 
>releases, at worst this will be repeat the mess of 0.20 security release forks.
>
>If we think adding this to 2.8 (even if it switched off) is too much 
>risk per our original plan, let¹s move this to 2.9, there by leaving 
>enough time for stability, integration testing and bake-in, and a 
>realistic chance of having it end up on users¹ clusters soonish.
>
>+Vinod
>
>> On Oct 19, 2015, at 1:44 PM, Andrew Wang <andrew.wang@cloudera.com>
>>wrote:
>> 
>> I think our plan thus far has been to target this for 3.0. I'm okay 
>>with  putting it in branch-2 if we've given a hard look at 
>>compatibility, but  I'll note though that 2.8 is already looking like 
>>quite a large release,  and our release bandwidth has been focused on 
>>the 2.6 and 2.7 maintenance  releases. Adding another multi-hundred 
>>JIRAs to 2.8 might make it too  unwieldy to get out the door. If we 
>>bump EC past that, 3.0 might very well  be our next release vehicle. I 
>>do plan to revive the 3.0 schedule some time  next year. With EC and 
>>JDK8 in a good spot, the only big feature remaining  is classpath 
>>isolation.
>> 
>> EC is also a pretty fundamental change to HDFS. Even if it's 
>>compatible, in  terms of size and impact it might best belong in a new 
>>major release.
>> 
>> Best,
>> Andrew
>> 
>> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < 
>> vinayakumarb.apache@gmail.com> wrote:
>> 
>>> Is anyone else also thinks that feature is ready to goto branch-2  
>>>as well?
>>> 
>>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since 
>>>then and  ready to go in branch-2.
>>> 
>>> -Vinay
>>> On Oct 6, 2015 12:51 AM, "Zhe Zhang" <zhezhang@cloudera.com> wrote:
>>> 
>>>> Thanks Vinay for capturing the issue and Uma for offering the help.
>>>> 
>>>> ---
>>>> Zhe Zhang
>>>> 
>>>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
>>> uma.gangumalla@intel.com
>>>>> 
>>>> wrote:
>>>> 
>>>>> Vinay,
>>>>> 
>>>>> 
>>>>> I would merge them as part of HDFS-9182.
>>>>> 
>>>>> Thanks,
>>>>> Uma
>>>>> 
>>>>> 
>>>>> 
>>>>> On 10/5/15, 12:48 AM, "Vinayakumar B" <vinayakumarb@apache.org>
>>>>>wrote:
>>>>> 
>>>>>> Hi Andrew,
>>>>>> I see CHANGES.txt entries not yet merged from
>>> CHANGES-HDFS-EC-7285.txt.
>>>>>> 
>>>>>> Was this intentional?
>>>>>> 
>>>>>> Regards,
>>>>>> Vinay
>>>>>> 
>>>>>> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
>>> andrew.wang@cloudera.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Branch has been merged to trunk, thanks again to everyone who

>>>>>>>worked
>>>> on
>>>>>>> the
>>>>>>> feature!
>>>>>>> 
>>>>>>> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang 
>>>>>>> <zhezhang@cloudera.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Thanks everyone who has participated in this discussion.
>>>>>>>> 
>>>>>>>> With 7 +1's (5 binding and 2 non-binding), and no -1, this
vote
>>> has
>>>>>>> passed.
>>>>>>>> I will do a final 'git merge' with trunk and work with Andrew

>>>>>>>> to
>>>> merge
>>>>>>> the
>>>>>>>> branch to trunk. I'll update on this thread when the merge
is
>>> done.
>>>>>>>> 
>>>>>>>> ---
>>>>>>>> Zhe Zhang
>>>>>>>> 
>>>>>>>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A 
>>>>>>>> <yi.a.liu@intel.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> (Change it to binding.)
>>>>>>>>> 
>>>>>>>>> +1
>>>>>>>>> I have been involved in the development and code review
on the
>>>>>>> feature
>>>>>>>>> branch. It's a great feature and I think it's ready to
merge 
>>>>>>>>> it
>>>> into
>>>>>>>> trunk.
>>>>>>>>> 
>>>>>>>>> Thanks all for the contribution.
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Yi Liu
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Liu, Yi A
>>>>>>>>> Sent: Friday, September 25, 2015 1:51 PM
>>>>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>>>>> Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding)
branch to
>>>> trunk
>>>>>>>>> 
>>>>>>>>> +1 (non-binding)
>>>>>>>>> I have been involved in the development and code review
on the
>>>>>>> feature
>>>>>>>>> branch. It's a great feature and I think it's ready to
merge 
>>>>>>>>> it
>>>> into
>>>>>>>> trunk.
>>>>>>>>> 
>>>>>>>>> Thanks all for the contribution.
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Yi Liu
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Vinayakumar B [mailto:vinayakumarb@apache.org]
>>>>>>>>> Sent: Friday, September 25, 2015 12:21 PM
>>>>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>>>>> Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding)
branch to
>>>> trunk
>>>>>>>>> 
>>>>>>>>> +1,
>>>>>>>>> 
>>>>>>>>> I've been involved starting from design and development
of
>>>>>>> ErasureCoding.
>>>>>>>>> I think phase 1 of this development is ready to be merged
to
>>>> trunk.
>>>>>>>>> It had come a long way to the current state with significant
>>>> effort
>>>>>>> of
>>>>>>>>> many Contributors and Reviewers for both design and code.
>>>>>>>>> 
>>>>>>>>> Thanks Everyone for the efforts.
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Vinay
>>>>>>>>> 
>>>>>>>>> On Wed, Sep 23, 2015 at 10:53 PM, Jing Zhao <jing9@apache.org>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> +1
>>>>>>>>>> 
>>>>>>>>>> I've been involved in both development and review
on the
>>> branch,
>>>>>>> and
>>>>>>> I
>>>>>>>>>> believe it's now ready to get merged into trunk.
Many thanks
>>> to
>>>>>>> all
>>>>>>>>>> the contributors and reviewers!
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> -Jing
>>>>>>>>>> 
>>>>>>>>>> On Tue, Sep 22, 2015 at 6:17 PM, Zheng, Kai <
>>>> kai.zheng@intel.com>
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Non-binding +1
>>>>>>>>>>> 
>>>>>>>>>>> According to our extensive performance tests,
striping +
>>> ISA-L
>>>>>>> coder
>>>>>>>>>> based
>>>>>>>>>>> erasure coding not only can save storage, but
also can
>>>> increase
>>>>>>> the
>>>>>>>>>>> throughput of a client or a cluster. It will
be a great
>>>>>>> addition to
>>>>>>>>>>> HDFS and its users. Based on the latest branch
codes, we
>>> also
>>>>>>>>>>> observed it's
>>>>>>>>>> very
>>>>>>>>>>> reliable in the concurrent tests. We'll provide
the perf
>>> test
>>>>>>> report
>>>>>>>>>> after
>>>>>>>>>>> it's sorted out and hope it helps.
>>>>>>>>>>> Thanks!
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Kai
>>>>>>>>>>> 
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
>>>>>>>>>>> Sent: Wednesday, September 23, 2015 8:50 AM
>>>>>>>>>>> To: hdfs-dev@hadoop.apache.org;
>>> common-dev@hadoop.apache.org
>>>>>>>>>>> Subject: Re: [VOTE] Merge HDFS-7285 (erasure
coding) branch
>>> to
>>>>>>> trunk
>>>>>>>>>>> 
>>>>>>>>>>> +1
>>>>>>>>>>> 
>>>>>>>>>>> Great addition to HDFS. Thanks all contributors
for the nice
>>>>>>> work.
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Uma
>>>>>>>>>>> 
>>>>>>>>>>> On 9/22/15, 3:40 PM, "Zhe Zhang" <zhezhang@cloudera.com>
>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> I'd like to propose a vote to merge the HDFS-7285
feature
>>>>>>> branch
>>>>>>>>>>>> back to trunk. Since November 2014 we have
been designing
>>> and
>>>>>>>>>>>> developing this feature under the umbrella
JIRAs HDFS-7285
>>>> and
>>>>>>>>>>>> HADOOP-11264, and have committed approximately
210 patches.
>>>>>>>>>>>> 
>>>>>>>>>>>> The HDFS-7285 feature branch was created
to support the
>>> first
>>>>>>> phase
>>>>>>>>>>>> of HDFS erasure coding (HDFS-EC). The objective
of HDFS-EC
>>> is
>>>>>>> to
>>>>>>>>>>>> significantly reduce storage space usage
in HDFS clusters.
>>>>>>> Instead
>>>>>>>>>>>> of always creating 3 replicas of each block
with 200%
>>> storage
>>>>>>> space
>>>>>>>>>>>> overhead, HDFS-EC provides data durability
through parity
>>>> data
>>>>>>>> blocks.
>>>>>>>>>>>> With most EC configurations, the storage
overhead is no
>>> more
>>>>>>> than
>>>>>>>> 50%.
>>>>>>>>>>>> Based on profiling results of production
clusters, we
>>> decided
>>>>>>> to
>>>>>>>>>>>> support EC with the striped block layout
in the first
>>> phase,
>>>> so
>>>>>>>>>>>> that small files can be better handled. This
means dividing
>>>>>>> each
>>>>>>>>>>>> logical HDFS file block into smaller units
(striping cells)
>>>> and
>>>>>>>>>>>> spreading them on a set of DataNodes in round-robin
>>> fashion.
>>>>>>> Parity
>>>>>>>>>>>> cells are generated for each stripe of original
data cells.
>>>> We
>>>>>>> have
>>>>>>>>>>>> made changes to NameNode, client, and DataNode
to
>>> generalize
>>>>>>> the
>>>>>>>>>>>> block concept and handle the mapping between
a logical file
>>>>>>> block
>>>>>>>>>>>> and its internal storage blocks. For further
details please
>>>> see
>>>>>>> the
>>>>>>>>>>>> design doc on HDFS-7285.
>>>>>>>>>>>> HADOOP-11264 focuses on providing flexible
and
>>>> high-performance
>>>>>>>>>>>> codec calculation support.
>>>>>>>>>>>> 
>>>>>>>>>>>> The nightly Jenkins job of the branch has
reported several 
>>>>>>>>>>>> successful runs, and doesn't show new flaky
tests compared
>>>> with
>>>>>>>>>>>> trunk. We have posted several versions of
the test plan
>>>>>>> including
>>>>>>>>>>>> both unit testing and cluster testing, and
have executed
>>> most
>>>>>>> tests
>>>>>>>>>>>> in the plan. The most basic functionalities
have been
>>>>>>> extensively
>>>>>>>>>>>> tested and verified in several real clusters
with different 
>>>>>>>>>>>> hardware configurations; results have been
very stable. We
>>>> have
>>>>>>>>>>>> created follow-on tasks for more advanced
error handling
>>> and
>>>>>>>>> optimization under the umbrella HDFS-8031.
>>>>>>>>>>>> We also plan to implement or harden the integration
of EC
>>>> with
>>>>>>>>>>>> existing features such as WebHDFS, snapshot,
append,
>>>> truncate,
>>>>>>>>>>>> hflush, hsync, and so forth.
>>>>>>>>>>>> 
>>>>>>>>>>>> Development of this feature has been a collaboration
across
>>>>>>> many
>>>>>>>>>>>> companies and institutions. I'd like to thank
J. Andreina,
>>>>>>> Takanobu
>>>>>>>>>>>> Asanuma, Vinayakumar B, Li Bo, Takuya Fukudome,
Uma
>>> Maheswara
>>>>>>> Rao
>>>>>>>>>>>> G, Rui Li, Yi Liu, Colin McCabe, Xinwei Qin,
Rakesh R, Gao
>>>> Rui,
>>>>>>> Kai
>>>>>>>>>>>> Sasaki, Walter Su, Tsz Wo Nicholas Sze, Andrew
Wang, Yong
>>>>>>> Zhang,
>>>>>>>>>>>> Jing Zhao, Hui Zheng and Kai Zheng for their
code
>>>> contributions
>>>>>>> and
>>>>>>>>> reviews.
>>>>>>>>>>>> Andrew and Kai Zheng also made fundamental
contributions to
>>>> the
>>>>>>>>>>>> initial design. Rui Li, Gao Rui, Kai Sasaki,
Kai Zheng and
>>>> many
>>>>>>>>>>>> other contributors have made great efforts
in system
>>> testing.
>>>>>>> Many
>>>>>>>>>>>> thanks go to Weihua Jiang for proposing the
JIRA, and ATM,
>>>> Todd
>>>>>>>>>>>> Lipcon, Silvius Rus, Suresh, as well as many
others for
>>>>>>> providing
>>>>>>>>> helpful feedbacks.
>>>>>>>>>>>> 
>>>>>>>>>>>> Following the community convention, this
vote will last
>>> for 7
>>>>>>> days
>>>>>>>>>>>> (ending September 29th). Votes from Hadoop
committers are
>>>>>>> binding
>>>>>>>>>>>> but non-binding votes are very welcome as
well. And here's
>>> my
>>>>>>>>>>>> non-binding
>>>>>>>>>> +1.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> ---
>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>


Mime
View raw message