hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <andrew.purt...@gmail.com>
Subject Re: HBASE 2.0 release progress
Date Tue, 22 Nov 2016 15:12:55 GMT
Favored nodes version 1 did not see much adoption and was hard to use or unusable, depending
on who you ask, so please expect a focus on usability and feature completeness when/if v2
is reviewed for possible inclusion in 2.0. While I am not suggesting this is the case, if
the v2 candidate shares the feature incompleteness or usability problems of its predecessor
it should not go in. 


> On Nov 21, 2016, at 6:01 PM, Thiruvel Thirumoolan <thiruvel@yahoo-inc.com.INVALID>
wrote:
> 
> We would like to add the favored node enhancements as part of https://issues.apache.org/jira/browse/HBASE-15531
to 2.0. Most of the code should be new, there will be very less code changes to Master/AM
etc.
> 
> -Thiruvel
> 
> On Monday, November 14, 2016, 10:06:49 PM PST, Anoop John <anoop.hbase@gmail.com>
wrote:>-  Anyone (Ram, Anoop?) wants to post a high-level writeup on the current
> up-to-date state of offheaping?
> 
> Off heaping the read path (HBASE-11425) this is completed some time
> back.  As per my knowledge at least 2 users back ported this work to
> 1.x based version and even running in production. Am leaving it them
> for giving details..
> 
> Write path off heaping HBASE-15179 is the umbrella issue.  As u can
> see most of the sub tasks are done by now..  Mainly 2 more sub tasks
> are yet to get committed.  Both are some what bigger sized also..
> Patch is already available for them.  We hope it can be completed by
> Nov end.
> 
> Some more off heaping work, (in compaction path etc) might be taken up
> after write path work. Alibaba guys might be joining with that.  Had
> some discuss with them.. We will come up with jira and doc for that
> before actual work begin.  But as Enis said ya all that can go in 2.x
> (x>0) :-)
> 
> Just giving some high level status update. Thanks ..
> 
> -Anoop-
> 
> 
>> On Tue, Nov 15, 2016 at 7:31 AM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
>> Enis, by your criteria it seems the log4j2 upgrade ticket should be
>> considered for 2.0 as well: HBASE-12341.
>> 
>>> On Monday, November 14, 2016, Enis Söztutar <enis.soz@gmail.com> wrote:
>>> 
>>> Another way to look at whether a feature is a "blocker" or not for 2.0 is a
>>> simple test:
>>>   - Can this feature be committed for 2.1, 2.2, etc or not assuming it does
>>> not make into 2.0.
>>> 
>>> It is a simple test, but affectively validates whether the feature is
>>> client-visible and have backwards compatibility implications. If it is ok
>>> to bring the feature in for 2.1 or later, it means that the release should
>>> never block on it. Otherwise, we should at least have the API / Client
>>> visible parts of it in 2.0 if not the full thing.
>>> 
>>> Looking at the features under discussion through this lens reveals easier
>>> decision points I think. A couple of examples:
>>>   - Java async client / C++ Client: It is independent work, can come in 2.1
>>> or 2.2, etc. Whenever it is completed.
>>>   - Offheaping : transparent ti clients, so can come in incrementally in
>>> 2.0, 2.1, 2.2, etc.
>>> 
>>> On the other hand, 1.x releases have been going on for >1.5 years, and will
>>> likely go on for another year at least. So the choices (dependency, APIs,
>>> etc) that we make now for 2.0 will likely live for >2 years. In that
>>> respect I would rather be a bit more aggressive in terms of dropping
>>> support for older stuff and updating wherever we can. For example require
>>> Hadoop-2.6+, Java-8, etc.
>>> 
>>> Enis
>>> 
>>> On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov <olorinbant@gmail.com
>>> <javascript:;>>
>>> wrote:
>>> 
>>>> Great to see progress here.
>>>> 
>>>> As I'm reading through the list, here're some high-level questions I
>>> have:
>>>> 
>>>>   -  Regarding the work on new AssignmentManager - any notes on
>>>> perf/stability testing? Are you guys running tips of master branch
>>> through
>>>> ITBLL setup?
>>>>   -  Anyone (Ram, Anoop?) wants to post a high-level writeup on the
>>> current
>>>> up-to-date state of offheaping?
>>>>   -  Logical clock - at this point is it more like a nice to have feature
>>>> than "need to be done before 2.0"? What are the features blocked or
>>>> affected by lack thereof?
>>>> 
>>>> -Mikhail
>>>> 
>>>> On Sun, Nov 13, 2016 at 3:57 PM, Stack <stack@duboce.net <javascript:;>>
>>> wrote:
>>>> 
>>>>> Thanks for the writeup Stephen.
>>>>> 
>>>>> See below.
>>>>> 
>>>>> On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <
>>> syuanjiangdev@gmail.com <javascript:;>>
>>>>> wrote:
>>>>> 
>>>>>> Hello, fellow HBASE developers,
>>>>>> 
>>>>>> We are making progress towards HBASE 2.0 releases.  I am using the
>>>>>> following queries to search for on-going HBASE 2.0 feature work items
>>>>>> (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0)
>>>> AND
>>>>>> resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
>>>>>> issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we
>>> have
>>>>>> 247!  That is a lot.  At some time in near future, we definitely
need
>>>> to
>>>>>> trim down the list; otherwise, 2.0 will never be released.
>>>>>> 
>>>>>> 
>>>>> Agree. Suggest we all do our own review but at a certain stage, it is
>>> up
>>>> to
>>>>> the RMs to make a call on what shape of 2.0 will be.
>>>>> 
>>>>> 
>>>>> 
>>>>>> For now, Matteo and I are tracking some big projects that are
>>> on-going:
>>>>>> 
>>>>>> (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
>>>>>> - This is a blocker to have branch-2 cut.  In the past few weeks,
we
>>>> made
>>>>>> good progress and majority of implementation is done.  The patches
>>> are
>>>>>> under review and testing.  Matto is drafting a document for review.
>>>>>> 
>>>>>> (2). HBASE-14414 Backup/Restore Phase 3
>>>>>> - Currently it is blocked by HBASE-14123.  The giant HBASE-14123
>>>> patches
>>>>>> was discussed and reviewed from the community (see
>>>>>> http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for
>>> the
>>>>>> long
>>>>>> discussion); and all feedback were taken care of in the latest patch.
>>>>>> Currently I marked HBASE-14123 as 2.0 blocker, as without it, the
>>>> further
>>>>>> develpment of backup/restore is blocked - the backup/restore is a
key
>>>>>> enterprise feature for 2.0 release.  I think HBASE-14123 is ready
for
>>>>>> another round of vote.
>>>>>> 
>>>>>> 
>>>>> IMO, not a blocker since it ancillary function but something we should
>>>> get
>>>>> in.
>>>>> 
>>>>> 
>>>>> 
>>>>>> (3). HBASE-15179 Offheap
>>>>>> - This is another important feature that I think it is MUST for 2.0.
>>>>> Since
>>>>>> stack works closely with Intel developers, he has some insight on
>>> this
>>>>>> project: "Intel are betting on this. Alibaba are using the offheap
>>> read
>>>>>> path and interested in write path too. This work is still very much
>>> up
>>>> in
>>>>>> the air and being worked out as we go (especially the Y! Israel
>>>> inmemory
>>>>>> compaction component).  It is a little shakey dependent on mslab
>>>> pooling,
>>>>>> blockcache being on by default, async wal being default, and then
>>>>> dependent
>>>>>> on lots of perf and ITBLL testing."
>>>>>> 
>>>>>> 
>>>>> The above is from a private, hurried email that does not do the current
>>>>> state justice.
>>>>> 
>>>>> Ram and Anoop are the authorities on offheaping and have been keeping
>>> up
>>>>> their state in an associated doc. The lads might be persuaded do a
>>>> summary
>>>>> of current state here.
>>>>> 
>>>>> Ditto for inmemory compaction. Our Anastasia and Eshcar are working
>>> hard
>>>> at
>>>>> the moment doing laste pieces and perf testing. Maybe we can a status
>>>>> dumped here in dev.
>>>>> 
>>>>> 
>>>>> 
>>>>>> (4). HBASE-16952 Protobuf3
>>>>>> - Good news, stack got the majority of work done already.  This is
a
>>>> MUST
>>>>>> for 2.0 release.  Now we only have a small sub-task HBASE-16967
>>>>> (findbugs)
>>>>>> left.
>>>>>> 
>>>>>> 
>>>>> To be clear, pb3 is under the offheaping umbrella. Let's add async WAL
>>> to
>>>>> this list, also needed by offheaping.
>>>>> 
>>>>> Will be back after taking a look at current 2.0 list.
>>>>> 
>>>>> Agree that hbase2 needs to support hadoop3.
>>>>> 
>>>>> St.Ack
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> (5). HBASE-14070 Logical Clock
>>>>>> - This needs to be done before 2.0 release.  At this time, seems
not
>>>>> making
>>>>>> much progress.
>>>>>> 
>>>>>> (6). HBASE-16833 JAVA Async Client
>>>>>> - A lot of progress was made by Duo and his associates.  We should
be
>>>> in
>>>>> a
>>>>>> good shape in JAVA client when 2.0 release.
>>>>>> 
>>>>>> (7). HBASE-14850 C++ Async Client
>>>>>> - Another project that is making good progress.  I know some customer
>>>>> wants
>>>>>> this.  This is long overdue project.  Hopefully we will make those
>>>>> customer
>>>>>> happy in 2.0 release with a good performed C++ client.
>>>>>> 
>>>>>> Please let me know other on-going projects that needs to be in HBASE
>>>> 2.0
>>>>>> release (stack mentioned "logical file system tier", but I am not
>>> sure
>>>>>> whether it would make enough progress to have a realistic chance
>>> making
>>>>>> 2.0).  As I said at the beginning of this email, at some point soon,
>>> we
>>>>>> will be more picky and trim down the unfinished features in 2.0.
>>>>>> 
>>>>>> Happy Weekend!
>>>>>> Stephen
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Thanks,
>>>> Michael Antonov
>>>> 

Mime
View raw message