hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <andrew.purt...@gmail.com>
Subject Re: HBASE 2.0 release progress
Date Tue, 22 Nov 2016 18:04:41 GMT
Thanks. Are your experiences with FNv2 in production documented? Perhaps on the JIRA? That
would go a long way.  


> On Nov 22, 2016, at 12:44 PM, Thiruvel Thirumoolan <thiruvel@yahoo-inc.com> wrote:
> 
> Hi Andrew,
> 
> Thanks for your comments.
> 
> FN v2 happened due to the same reasons you mention. We believe v2 is complete based on
our experience using and operating it in production since early 2016. We have a design doc
@ umbrella jira HBASE-15531 and have responded to all questions we had so far. More eyes will
always help.
> 
> -Thiruvel
> 
> 
> On Tuesday, November 22, 2016, 7:13:08 AM PST, Andrew Purtell <andrew.purtell@gmail.com>
wrote:
> Favored nodes version 1 did not see much adoption and was hard to use or unusable, depending
on who you ask, so please expect a focus on usability and feature completeness when/if v2
is reviewed for possible inclusion in 2.0. While I am not suggesting this is the case, if
the v2 candidate shares the feature incompleteness or usability problems of its predecessor
it should not go in. 
> 
> 
> > On Nov 21, 2016, at 6:01 PM, Thiruvel Thirumoolan <thiruvel@yahoo-inc.com.INVALID>
wrote:
> > 
> > We would like to add the favored node enhancements as part of https://issues.apache.org/jira/browse/HBASE-15531
to 2.0. Most of the code should be new, there will be very less code changes to Master/AM
etc.
> > 
> > -Thiruvel
> > 
> > On Monday, November 14, 2016, 10:06:49 PM PST, Anoop John <anoop.hbase@gmail.com>
wrote:>-  Anyone (Ram, Anoop?) wants to post a high-level writeup on the current
> > up-to-date state of offheaping?
> > 
> > Off heaping the read path (HBASE-11425) this is completed some time
> > back.  As per my knowledge at least 2 users back ported this work to
> > 1.x based version and even running in production. Am leaving it them
> > for giving details..
> > 
> > Write path off heaping HBASE-15179 is the umbrella issue.  As u can
> > see most of the sub tasks are done by now..  Mainly 2 more sub tasks
> > are yet to get committed.  Both are some what bigger sized also..
> > Patch is already available for them.  We hope it can be completed by
> > Nov end.
> > 
> > Some more off heaping work, (in compaction path etc) might be taken up
> > after write path work. Alibaba guys might be joining with that.  Had
> > some discuss with them.. We will come up with jira and doc for that
> > before actual work begin.  But as Enis said ya all that can go in 2.x
> > (x>0) :-)
> > 
> > Just giving some high level status update. Thanks ..
> > 
> > -Anoop-
> > 
> > 
> >> On Tue, Nov 15, 2016 at 7:31 AM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
> >> Enis, by your criteria it seems the log4j2 upgrade ticket should be
> >> considered for 2.0 as well: HBASE-12341.
> >> 
> >>> On Monday, November 14, 2016, Enis Söztutar <enis.soz@gmail.com>
wrote:
> >>> 
> >>> Another way to look at whether a feature is a "blocker" or not for 2.0 is
a
> >>> simple test:
> >>>  - Can this feature be committed for 2.1, 2.2, etc or not assuming it does
> >>> not make into 2.0.
> >>> 
> >>> It is a simple test, but affectively validates whether the feature is
> >>> client-visible and have backwards compatibility implications. If it is ok
> >>> to bring the feature in for 2.1 or later, it means that the release should
> >>> never block on it. Otherwise, we should at least have the API / Client
> >>> visible parts of it in 2.0 if not the full thing.
> >>> 
> >>> Looking at the features under discussion through this lens reveals easier
> >>> decision points I think. A couple of examples:
> >>>  - Java async client / C++ Client: It is independent work, can come in 2.1
> >>> or 2.2, etc. Whenever it is completed.
> >>>  - Offheaping : transparent ti clients, so can come in incrementally in
> >>> 2.0, 2.1, 2.2, etc.
> >>> 
> >>> On the other hand, 1.x releases have been going on for >1.5 years, and
will
> >>> likely go on for another year at least. So the choices (dependency, APIs,
> >>> etc) that we make now for 2.0 will likely live for >2 years. In that
> >>> respect I would rather be a bit more aggressive in terms of dropping
> >>> support for older stuff and updating wherever we can. For example require
> >>> Hadoop-2.6+, Java-8, etc.
> >>> 
> >>> Enis
> >>> 
> >>> On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov <olorinbant@gmail.com
> >>> <javascript:;>>
> >>> wrote:
> >>> 
> >>>> Great to see progress here.
> >>>> 
> >>>> As I'm reading through the list, here're some high-level questions I
> >>> have:
> >>>> 
> >>>>  -  Regarding the work on new AssignmentManager - any notes on
> >>>> perf/stability testing? Are you guys running tips of master branch
> >>> through
> >>>> ITBLL setup?
> >>>>  -  Anyone (Ram, Anoop?) wants to post a high-level writeup on the
> >>> current
> >>>> up-to-date state of offheaping?
> >>>>  -  Logical clock - at this point is it more like a nice to have feature
> >>>> than "need to be done before 2.0"? What are the features blocked or
> >>>> affected by lack thereof?
> >>>> 
> >>>> -Mikhail
> >>>> 
> >>>> On Sun, Nov 13, 2016 at 3:57 PM, Stack <stack@duboce.net <javascript:;>>
> >>> wrote:
> >>>> 
> >>>>> Thanks for the writeup Stephen.
> >>>>> 
> >>>>> See below.
> >>>>> 
> >>>>> On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <
> >>> syuanjiangdev@gmail.com <javascript:;>>
> >>>>> wrote:
> >>>>> 
> >>>>>> Hello, fellow HBASE developers,
> >>>>>> 
> >>>>>> We are making progress towards HBASE 2.0 releases.  I am using
the
> >>>>>> following queries to search for on-going HBASE 2.0 feature work
items
> >>>>>> (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion
= 2.0.0)
> >>>> AND
> >>>>>> resolution is EMPTY AND (issuetype != Bug AND issuetype != Test
AND
> >>>>>> issuetype != Sub-task) ORDER BY issuetype DESC), at this time,
we
> >>> have
> >>>>>> 247!  That is a lot.  At some time in near future, we definitely
need
> >>>> to
> >>>>>> trim down the list; otherwise, 2.0 will never be released.
> >>>>>> 
> >>>>>> 
> >>>>> Agree. Suggest we all do our own review but at a certain stage,
it is
> >>> up
> >>>> to
> >>>>> the RMs to make a call on what shape of 2.0 will be.
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> For now, Matteo and I are tracking some big projects that are
> >>> on-going:
> >>>>>> 
> >>>>>> (1). HBASE-14350 the stable Assignment Manager (using Procedure
V2)
> >>>>>> - This is a blocker to have branch-2 cut.  In the past few weeks,
we
> >>>> made
> >>>>>> good progress and majority of implementation is done.  The patches
> >>> are
> >>>>>> under review and testing.  Matto is drafting a document for
review.
> >>>>>> 
> >>>>>> (2). HBASE-14414 Backup/Restore Phase 3
> >>>>>> - Currently it is blocked by HBASE-14123.  The giant HBASE-14123
> >>>> patches
> >>>>>> was discussed and reviewed from the community (see
> >>>>>> http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html
for
> >>> the
> >>>>>> long
> >>>>>> discussion); and all feedback were taken care of in the latest
patch.
> >>>>>> Currently I marked HBASE-14123 as 2.0 blocker, as without it,
the
> >>>> further
> >>>>>> develpment of backup/restore is blocked - the backup/restore
is a key
> >>>>>> enterprise feature for 2.0 release.  I think HBASE-14123 is
ready for
> >>>>>> another round of vote.
> >>>>>> 
> >>>>>> 
> >>>>> IMO, not a blocker since it ancillary function but something we
should
> >>>> get
> >>>>> in.
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> (3). HBASE-15179 Offheap
> >>>>>> - This is another important feature that I think it is MUST
for 2.0.
> >>>>> Since
> >>>>>> stack works closely with Intel developers, he has some insight
on
> >>> this
> >>>>>> project: "Intel are betting on this. Alibaba are using the offheap
> >>> read
> >>>>>> path and interested in write path too. This work is still very
much
> >>> up
> >>>> in
> >>>>>> the air and being worked out as we go (especially the Y! Israel
> >>>> inmemory
> >>>>>> compaction component).  It is a little shakey dependent on mslab
> >>>> pooling,
> >>>>>> blockcache being on by default, async wal being default, and
then
> >>>>> dependent
> >>>>>> on lots of perf and ITBLL testing."
> >>>>>> 
> >>>>>> 
> >>>>> The above is from a private, hurried email that does not do the
current
> >>>>> state justice.
> >>>>> 
> >>>>> Ram and Anoop are the authorities on offheaping and have been keeping
> >>> up
> >>>>> their state in an associated doc. The lads might be persuaded do
a
> >>>> summary
> >>>>> of current state here.
> >>>>> 
> >>>>> Ditto for inmemory compaction. Our Anastasia and Eshcar are working
> >>> hard
> >>>> at
> >>>>> the moment doing laste pieces and perf testing. Maybe we can a status
> >>>>> dumped here in dev.
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> (4). HBASE-16952 Protobuf3
> >>>>>> - Good news, stack got the majority of work done already.  This
is a
> >>>> MUST
> >>>>>> for 2.0 release.  Now we only have a small sub-task HBASE-16967
> >>>>> (findbugs)
> >>>>>> left.
> >>>>>> 
> >>>>>> 
> >>>>> To be clear, pb3 is under the offheaping umbrella. Let's add async
WAL
> >>> to
> >>>>> this list, also needed by offheaping.
> >>>>> 
> >>>>> Will be back after taking a look at current 2.0 list.
> >>>>> 
> >>>>> Agree that hbase2 needs to support hadoop3.
> >>>>> 
> >>>>> St.Ack
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> (5). HBASE-14070 Logical Clock
> >>>>>> - This needs to be done before 2.0 release.  At this time, seems
not
> >>>>> making
> >>>>>> much progress.
> >>>>>> 
> >>>>>> (6). HBASE-16833 JAVA Async Client
> >>>>>> - A lot of progress was made by Duo and his associates.  We
should be
> >>>> in
> >>>>> a
> >>>>>> good shape in JAVA client when 2.0 release.
> >>>>>> 
> >>>>>> (7). HBASE-14850 C++ Async Client
> >>>>>> - Another project that is making good progress.  I know some
customer
> >>>>> wants
> >>>>>> this.  This is long overdue project.  Hopefully we will make
those
> >>>>> customer
> >>>>>> happy in 2.0 release with a good performed C++ client.
> >>>>>> 
> >>>>>> Please let me know other on-going projects that needs to be
in HBASE
> >>>> 2.0
> >>>>>> release (stack mentioned "logical file system tier", but I am
not
> >>> sure
> >>>>>> whether it would make enough progress to have a realistic chance
> >>> making
> >>>>>> 2.0).  As I said at the beginning of this email, at some point
soon,
> >>> we
> >>>>>> will be more picky and trim down the unfinished features in
2.0.
> >>>>>> 
> >>>>>> Happy Weekend!
> >>>>>> Stephen
> >>>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> --
> >>>> Thanks,
> >>>> Michael Antonov
> >>>>

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message