Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
From: "Zheng, Kai" <kai.zheng@intel.com>
To: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>
Subject: RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285
 (erasure coding) branch to trunk]
Thread-Topic: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285
 (erasure coding) branch to trunk]
Thread-Index: AQHRFaeyn6o+LaIGnEaydpYZeEbJtJ6IvNaAgACefVA=
Date: Mon, 2 Nov 2015 23:21:49 +0000
Message-ID: 
 <8D5F7E3237B3ED47B84CF187BB17B66611C99BA9@SHSMSX103.ccr.corp.intel.com>
References: 
 <CAAjoTXnx+79aGB=wV+NxQA9jy7KS9aw_b=19WMf9hBKu0tEVAA@mail.gmail.com>
 <D22743D3.671D%uma.gangumalla@intel.com>
 <8D5F7E3237B3ED47B84CF187BB17B66611C57EAB@shsmsx102.ccr.corp.intel.com>
 <CA+13OA1DGNN=6B4hskO=_9RnM-i5s7RhVnB_jwxka8KejpEhNw@mail.gmail.com>
 <CADoWKTCGVj1bDRym7m0sT6icbW0MC53ZDi-s=bQGh_ooc9fQBQ@mail.gmail.com>
 <0ACA11997C562042A7FDB41B0D58461001F09F86@SHSMSX104.ccr.corp.intel.com>
 <CAAjoTXkHh+u86v4FfAQj0TBv2OF07WrSqfnbmU63p40Gn+=59A@mail.gmail.com>
 <CAGB5D2Zk6scMrzKQANaeDjfM0875oNZ0pbrLD2-3fV1u66BxYw@mail.gmail.com>
 <CADoWKTA-iFaGF7H+iA10LWfJohzc=Q=E64+ww29Whf9-BbhX7w@mail.gmail.com>
 <D2381801.6B42%uma.gangumalla@intel.com>
 <CAAjoTXnj=5_hG7sfiVsk7sA6msGWmhz6UL1H1cZp2Lvutjk93Q@mail.gmail.com>
 <CADoWKTDriK8_z+0jb3-FH47ca46+JKt-Z+vxN4=a9QuHzhZBJA@mail.gmail.com>
 <CAGB5D2ZqjnOU1aGN0PWQc2z63nSmArFxh=ov-k2+5_gZnbw0QA@mail.gmail.com>
 <9B794423-846D-4BA9-B1F1-1631126C593D@hortonworks.com>
 <D25D177B.747E%uma.gangumalla@intel.com>
In-Reply-To: <D25D177B.747E%uma.gangumalla@intel.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Sounds good to me. When it's determined to include EC in 2.9 release, it ma=
y be good to have a rough release date as Zhe asked, so accordingly the sco=
pe of EC can be discussed out. We still have quite a few of things as Phase=
 I follow-on tasks to do before EC can be deployed in a production system. =
Phase II to develop non-striping EC for cold data would possibly be started=
 after that. We might consider to include only Phase I and leave Phase II f=
or next release according to the rough release date.

Regards,
Kai

-----Original Message-----
From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]=20
Sent: Tuesday, November 03, 2015 5:41 AM
To: hdfs-dev@hadoop.apache.org
Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (er=
asure coding) branch to trunk]

+1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan=20
+to
have 2.8 and 2.9 releases.

Regards,
Uma

On 11/2/15, 11:49 AM, "Vinod Vavilapalli" <vinodkv@hortonworks.com> wrote:

>Forking the thread. Started looking at the 2.8 list, various features=B9=20
>status and arrived here.
>
>While I understand the pervasive nature of EC and a need for a=20
>significant bake-in, moving this to a 3.x release is not a good idea.=20
>We will surely get a 2.8 out this year and, as needed, I can even spend=20
>time getting started on a 2.9. OTOH, 3.x is long ways off, and given=20
>all the incompatibilities there, it would be a while before users can=20
>get their hands on EC if it were to be only on 3.x. At best, this may=20
>force sites that want EC to backport the entire EC feature to older=20
>releases, at worst this will be repeat the mess of 0.20 security release f=
orks.
>
>If we think adding this to 2.8 (even if it switched off) is too much=20
>risk per our original plan, let=B9s move this to 2.9, there by leaving=20
>enough time for stability, integration testing and bake-in, and a=20
>realistic chance of having it end up on users=B9 clusters soonish.
>
>+Vinod
>
>> On Oct 19, 2015, at 1:44 PM, Andrew Wang <andrew.wang@cloudera.com>
>>wrote:
>>=20
>> I think our plan thus far has been to target this for 3.0. I'm okay=20
>>with  putting it in branch-2 if we've given a hard look at=20
>>compatibility, but  I'll note though that 2.8 is already looking like=20
>>quite a large release,  and our release bandwidth has been focused on=20
>>the 2.6 and 2.7 maintenance  releases. Adding another multi-hundred=20
>>JIRAs to 2.8 might make it too  unwieldy to get out the door. If we=20
>>bump EC past that, 3.0 might very well  be our next release vehicle. I=20
>>do plan to revive the 3.0 schedule some time  next year. With EC and=20
>>JDK8 in a good spot, the only big feature remaining  is classpath=20
>>isolation.
>>=20
>> EC is also a pretty fundamental change to HDFS. Even if it's=20
>>compatible, in  terms of size and impact it might best belong in a new=20
>>major release.
>>=20
>> Best,
>> Andrew
>>=20
>> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <=20
>> vinayakumarb.apache@gmail.com> wrote:
>>=20
>>> Is anyone else also thinks that feature is ready to goto branch-2 =20
>>>as well?
>>>=20
>>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since=20
>>>then and  ready to go in branch-2.
>>>=20
>>> -Vinay
>>> On Oct 6, 2015 12:51 AM, "Zhe Zhang" <zhezhang@cloudera.com> wrote:
>>>=20
>>>> Thanks Vinay for capturing the issue and Uma for offering the help.
>>>>=20
>>>> ---
>>>> Zhe Zhang
>>>>=20
>>>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
>>> uma.gangumalla@intel.com
>>>>>=20
>>>> wrote:
>>>>=20
>>>>> Vinay,
>>>>>=20
>>>>>=20
>>>>> I would merge them as part of HDFS-9182.
>>>>>=20
>>>>> Thanks,
>>>>> Uma
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> On 10/5/15, 12:48 AM, "Vinayakumar B" <vinayakumarb@apache.org>
>>>>>wrote:
>>>>>=20
>>>>>> Hi Andrew,
>>>>>> I see CHANGES.txt entries not yet merged from
>>> CHANGES-HDFS-EC-7285.txt.
>>>>>>=20
>>>>>> Was this intentional?
>>>>>>=20
>>>>>> Regards,
>>>>>> Vinay
>>>>>>=20
>>>>>> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
>>> andrew.wang@cloudera.com>
>>>>>> wrote:
>>>>>>=20
>>>>>>> Branch has been merged to trunk, thanks again to everyone who=20
>>>>>>>worked
>>>> on
>>>>>>> the
>>>>>>> feature!
>>>>>>>=20
>>>>>>> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang=20
>>>>>>> <zhezhang@cloudera.com>
>>>>>>> wrote:
>>>>>>>=20
>>>>>>>> Thanks everyone who has participated in this discussion.
>>>>>>>>=20
>>>>>>>> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote
>>> has
>>>>>>> passed.
>>>>>>>> I will do a final 'git merge' with trunk and work with Andrew=20
>>>>>>>> to
>>>> merge
>>>>>>> the
>>>>>>>> branch to trunk. I'll update on this thread when the merge is
>>> done.
>>>>>>>>=20
>>>>>>>> ---
>>>>>>>> Zhe Zhang
>>>>>>>>=20
>>>>>>>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A=20
>>>>>>>> <yi.a.liu@intel.com>
>>>>>>> wrote:
>>>>>>>>=20
>>>>>>>>> (Change it to binding.)
>>>>>>>>>=20
>>>>>>>>> +1
>>>>>>>>> I have been involved in the development and code review on the
>>>>>>> feature
>>>>>>>>> branch. It's a great feature and I think it's ready to merge=20
>>>>>>>>> it
>>>> into
>>>>>>>> trunk.
>>>>>>>>>=20
>>>>>>>>> Thanks all for the contribution.
>>>>>>>>>=20
>>>>>>>>> Regards,
>>>>>>>>> Yi Liu
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Liu, Yi A
>>>>>>>>> Sent: Friday, September 25, 2015 1:51 PM
>>>>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>>>>> Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to
>>>> trunk
>>>>>>>>>=20
>>>>>>>>> +1 (non-binding)
>>>>>>>>> I have been involved in the development and code review on the
>>>>>>> feature
>>>>>>>>> branch. It's a great feature and I think it's ready to merge=20
>>>>>>>>> it
>>>> into
>>>>>>>> trunk.
>>>>>>>>>=20
>>>>>>>>> Thanks all for the contribution.
>>>>>>>>>=20
>>>>>>>>> Regards,
>>>>>>>>> Yi Liu
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Vinayakumar B [mailto:vinayakumarb@apache.org]
>>>>>>>>> Sent: Friday, September 25, 2015 12:21 PM
>>>>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>>>>> Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to
>>>> trunk
>>>>>>>>>=20
>>>>>>>>> +1,
>>>>>>>>>=20
>>>>>>>>> I've been involved starting from design and development of
>>>>>>> ErasureCoding.
>>>>>>>>> I think phase 1 of this development is ready to be merged to
>>>> trunk.
>>>>>>>>> It had come a long way to the current state with significant
>>>> effort
>>>>>>> of
>>>>>>>>> many Contributors and Reviewers for both design and code.
>>>>>>>>>=20
>>>>>>>>> Thanks Everyone for the efforts.
>>>>>>>>>=20
>>>>>>>>> Regards,
>>>>>>>>> Vinay
>>>>>>>>>=20
>>>>>>>>> On Wed, Sep 23, 2015 at 10:53 PM, Jing Zhao <jing9@apache.org>
>>>>>>> wrote:
>>>>>>>>>=20
>>>>>>>>>> +1
>>>>>>>>>>=20
>>>>>>>>>> I've been involved in both development and review on the
>>> branch,
>>>>>>> and
>>>>>>> I
>>>>>>>>>> believe it's now ready to get merged into trunk. Many thanks
>>> to
>>>>>>> all
>>>>>>>>>> the contributors and reviewers!
>>>>>>>>>>=20
>>>>>>>>>> Thanks,
>>>>>>>>>> -Jing
>>>>>>>>>>=20
>>>>>>>>>> On Tue, Sep 22, 2015 at 6:17 PM, Zheng, Kai <
>>>> kai.zheng@intel.com>
>>>>>>>> wrote:
>>>>>>>>>>=20
>>>>>>>>>>> Non-binding +1
>>>>>>>>>>>=20
>>>>>>>>>>> According to our extensive performance tests, striping +
>>> ISA-L
>>>>>>> coder
>>>>>>>>>> based
>>>>>>>>>>> erasure coding not only can save storage, but also can
>>>> increase
>>>>>>> the
>>>>>>>>>>> throughput of a client or a cluster. It will be a great
>>>>>>> addition to
>>>>>>>>>>> HDFS and its users. Based on the latest branch codes, we
>>> also
>>>>>>>>>>> observed it's
>>>>>>>>>> very
>>>>>>>>>>> reliable in the concurrent tests. We'll provide the perf
>>> test
>>>>>>> report
>>>>>>>>>> after
>>>>>>>>>>> it's sorted out and hope it helps.
>>>>>>>>>>> Thanks!
>>>>>>>>>>>=20
>>>>>>>>>>> Regards,
>>>>>>>>>>> Kai
>>>>>>>>>>>=20
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
>>>>>>>>>>> Sent: Wednesday, September 23, 2015 8:50 AM
>>>>>>>>>>> To: hdfs-dev@hadoop.apache.org;
>>> common-dev@hadoop.apache.org
>>>>>>>>>>> Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch
>>> to
>>>>>>> trunk
>>>>>>>>>>>=20
>>>>>>>>>>> +1
>>>>>>>>>>>=20
>>>>>>>>>>> Great addition to HDFS. Thanks all contributors for the nice
>>>>>>> work.
>>>>>>>>>>>=20
>>>>>>>>>>> Regards,
>>>>>>>>>>> Uma
>>>>>>>>>>>=20
>>>>>>>>>>> On 9/22/15, 3:40 PM, "Zhe Zhang" <zhezhang@cloudera.com>
>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>=20
>>>>>>>>>>>> I'd like to propose a vote to merge the HDFS-7285 feature
>>>>>>> branch
>>>>>>>>>>>> back to trunk. Since November 2014 we have been designing
>>> and
>>>>>>>>>>>> developing this feature under the umbrella JIRAs HDFS-7285
>>>> and
>>>>>>>>>>>> HADOOP-11264, and have committed approximately 210 patches.
>>>>>>>>>>>>=20
>>>>>>>>>>>> The HDFS-7285 feature branch was created to support the
>>> first
>>>>>>> phase
>>>>>>>>>>>> of HDFS erasure coding (HDFS-EC). The objective of HDFS-EC
>>> is
>>>>>>> to
>>>>>>>>>>>> significantly reduce storage space usage in HDFS clusters.
>>>>>>> Instead
>>>>>>>>>>>> of always creating 3 replicas of each block with 200%
>>> storage
>>>>>>> space
>>>>>>>>>>>> overhead, HDFS-EC provides data durability through parity
>>>> data
>>>>>>>> blocks.
>>>>>>>>>>>> With most EC configurations, the storage overhead is no
>>> more
>>>>>>> than
>>>>>>>> 50%.
>>>>>>>>>>>> Based on profiling results of production clusters, we
>>> decided
>>>>>>> to
>>>>>>>>>>>> support EC with the striped block layout in the first
>>> phase,
>>>> so
>>>>>>>>>>>> that small files can be better handled. This means dividing
>>>>>>> each
>>>>>>>>>>>> logical HDFS file block into smaller units (striping cells)
>>>> and
>>>>>>>>>>>> spreading them on a set of DataNodes in round-robin
>>> fashion.
>>>>>>> Parity
>>>>>>>>>>>> cells are generated for each stripe of original data cells.
>>>> We
>>>>>>> have
>>>>>>>>>>>> made changes to NameNode, client, and DataNode to
>>> generalize
>>>>>>> the
>>>>>>>>>>>> block concept and handle the mapping between a logical file
>>>>>>> block
>>>>>>>>>>>> and its internal storage blocks. For further details please
>>>> see
>>>>>>> the
>>>>>>>>>>>> design doc on HDFS-7285.
>>>>>>>>>>>> HADOOP-11264 focuses on providing flexible and
>>>> high-performance
>>>>>>>>>>>> codec calculation support.
>>>>>>>>>>>>=20
>>>>>>>>>>>> The nightly Jenkins job of the branch has reported several=20
>>>>>>>>>>>> successful runs, and doesn't show new flaky tests compared
>>>> with
>>>>>>>>>>>> trunk. We have posted several versions of the test plan
>>>>>>> including
>>>>>>>>>>>> both unit testing and cluster testing, and have executed
>>> most
>>>>>>> tests
>>>>>>>>>>>> in the plan. The most basic functionalities have been
>>>>>>> extensively
>>>>>>>>>>>> tested and verified in several real clusters with different=20
>>>>>>>>>>>> hardware configurations; results have been very stable. We
>>>> have
>>>>>>>>>>>> created follow-on tasks for more advanced error handling
>>> and
>>>>>>>>> optimization under the umbrella HDFS-8031.
>>>>>>>>>>>> We also plan to implement or harden the integration of EC
>>>> with
>>>>>>>>>>>> existing features such as WebHDFS, snapshot, append,
>>>> truncate,
>>>>>>>>>>>> hflush, hsync, and so forth.
>>>>>>>>>>>>=20
>>>>>>>>>>>> Development of this feature has been a collaboration across
>>>>>>> many
>>>>>>>>>>>> companies and institutions. I'd like to thank J. Andreina,
>>>>>>> Takanobu
>>>>>>>>>>>> Asanuma, Vinayakumar B, Li Bo, Takuya Fukudome, Uma
>>> Maheswara
>>>>>>> Rao
>>>>>>>>>>>> G, Rui Li, Yi Liu, Colin McCabe, Xinwei Qin, Rakesh R, Gao
>>>> Rui,
>>>>>>> Kai
>>>>>>>>>>>> Sasaki, Walter Su, Tsz Wo Nicholas Sze, Andrew Wang, Yong
>>>>>>> Zhang,
>>>>>>>>>>>> Jing Zhao, Hui Zheng and Kai Zheng for their code
>>>> contributions
>>>>>>> and
>>>>>>>>> reviews.
>>>>>>>>>>>> Andrew and Kai Zheng also made fundamental contributions to
>>>> the
>>>>>>>>>>>> initial design. Rui Li, Gao Rui, Kai Sasaki, Kai Zheng and
>>>> many
>>>>>>>>>>>> other contributors have made great efforts in system
>>> testing.
>>>>>>> Many
>>>>>>>>>>>> thanks go to Weihua Jiang for proposing the JIRA, and ATM,
>>>> Todd
>>>>>>>>>>>> Lipcon, Silvius Rus, Suresh, as well as many others for
>>>>>>> providing
>>>>>>>>> helpful feedbacks.
>>>>>>>>>>>>=20
>>>>>>>>>>>> Following the community convention, this vote will last
>>> for 7
>>>>>>> days
>>>>>>>>>>>> (ending September 29th). Votes from Hadoop committers are
>>>>>>> binding
>>>>>>>>>>>> but non-binding votes are very welcome as well. And here's
>>> my
>>>>>>>>>>>> non-binding
>>>>>>>>>> +1.
>>>>>>>>>>>>=20
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> ---
>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>=20
>>>>>=20
>>>>>=20
>>>>=20
>>>=20
>