hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: [DISCUSSION] Update on HBASE-10070 / Merge into trunk
Date Sat, 07 Jun 2014 05:21:20 GMT
Thanks for the responses lads.  Sounds good.

I took the tip of HBASE-10070 for a spin just now.  It started up fine over
existing dataset.  At first it threw me off because it is missing trunk
features but after I got over that, it looked fine.  No strange errors in
logs using my config from pre-HBASE-10070 startup. Thats good.

Running a pure randomread test on a small cluster I seem to see a slight
slowdown (14.2k/second vs 13.7k/second out of bucketcache).  You seen that?
 Just wondering.

St.Ack






On Tue, May 27, 2014 at 5:27 PM, Enis Söztutar <enis.soz@gmail.com> wrote:

> Agreed with Devaraj. I do not think that these changes will destabilize the
> code base much. All of the code committed to branch has been unit tested
> and integration tests, both specific to region replicas and others have
> been running for some time.
>
> Specifically, various large scale data ingestion + verification tests, bulk
> load tests, hbck tests, shell tests, replication, mapreduce based tests,
> snapshot tests, etc has been running on this code base.
>
> For testing the feature itself, HBASE-10572, HBASE-10616, and HBASE-10818
> adds tests for get/multi-get/scan and bulk load which has been running with
> CM. HBASE-10817 adds a test for region replicas + replication. HBASE-10791
> changes PerformanceEvaluation to be able to work with region replicas.
> Nicolas is also working on further perf testing and improvements.
>
> For 1.0, I think it should be fine to include this as an experimental
> feature. Otherwise, it will be very hard to add this to the 1.x line.
>
> Are there any more concerns? If not, I would like to raise a VOTE soon.
>
> Enis
>
>
> On Sat, May 24, 2014 at 10:40 AM, Devaraj Das <ddas@hortonworks.com>
> wrote:
>
> > Thanks Stack. On the testing, we have been adding unit tests for all
> > patches. We have also tested the patches quite extensively on real
> clusters
> > via IT tests (existing and newly added) and issues discovered have been
> > fixed. AFAICT, and from the tests run, this feature shouldn't destabilize
> > the mainline. There are some limitations when region-replica is used -
> > better integration with hbck (HBASE-10674) and support for split/merge
> (and
> > these are being worked on as part of phase 2).
> >  On May 23, 2014 9:56 PM, "Stack" <stack@duboce.net> wrote:
> >
> > > I think merge sooner rather than later.
> > >
> > > What sort of testing has been done on 10070 and what can you say about
> > how
> > > well the feature works?
> > >
> > > Where do you see risk, if any, of it destabilizing the mainline
> > especially
> > > what with us coming up to a 1.0.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > >
> > >
> > > On Wed, May 21, 2014 at 5:08 PM, Enis Söztutar <enis@apache.org>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > We would like to give an update on the status of HBASE-10070 work,
> and
> > > open
> > > > up discussion for how we can do further development.
> > > >
> > > > We seem to be at a point where we have the core functionality of the
> > > > region replica, as described in HBASE-10070 working. As pointed out
> > > > under the section "Development Phases" in the design doc posted on
> the
> > > > jira HBASE-10070, this work was divided into two broad phases. The
> > first
> > > > phase introduces region replicas concept, the new consistency model,
> > and
> > > > corresponding RPC implementations. All of the issues for Phase 1 can
> be
> > > > found under [3]. Phase 2 is still in the works, and contains the
> > proposed
> > > > changes listed under [4].
> > > >
> > > > With all the issues committed in HBASE-10070 branch in svn, we think
> > that
> > > > the "phase-1" is complete. The user documentation on HBASE-10513
> gives
> > an
> > > > accurate picture of what has been done in phase-1 and what the impact
> > of
> > > > using this feature is, APIs etc. We have added
> > > > a couple of IT tests as part of this work and we have tested the work
> > > > we did in "phase-1" of the project quite extensively in Hortonworks'
> > > > infrastructure.
> > > >
> > > > In summary, with the code in branch, you can create tables with
> region
> > > > replicas, do gets / multi gets and scans using TIMELINE consistency
> > with
> > > > high availability. Region replicas periodically scan the files of the
> > > > primary and pick up flushed / committed files. The RPC paths /
> > > assignment,
> > > > balancing etc are pretty stable. However some more performance
> analysis
> > > and
> > > > tuning is needed. More information can be found in [1] and [2].
> > > >
> > > >
> > > > As a reminder, the development has been happening in the branch -
> > > > hbase-10070 (https://github.com/apache/hbase/tree/hbase-10070). We
> > have
> > > > been pretty diligent about more than one committer's +1 on the branch
> > > > commits and for almost all the subtasks in HBASE-10070 there is more
> > than
> > > > one +1 except for test fix issues or more trivial issues, where there
> > > maybe
> > > >  only one +1.  Enis/Nicolas/Sergey/Devaraj/Nick are the main
> > contributors
> > > > of code in the phase-1 and many patches have been reviewed already by
> > > > people outside
> > > > this group (mainly Stack, Jimmy)
> > > >
> > > > For Phase 2, we think that we can deliver on the proposed design
> > > > incrementally over the next couple of months. However, we think that
> it
> > > > might be ok to merge the changes from phase 1 first, then do a
> > > > commit-as-you-go approach for remaining items. Therefore, we would
> like
> > > to
> > > > propose  to merge the branch to trunk, and continue the work over
> > there.
> > > > This might also result in more reviews.
> > > >
> > > > Alternatively, we can continue the work in the branch, and do the
> merge
> > > at
> > > > the end of Phase 2, but that will make the review process a bit more
> > > > tricky, which is why we would like the merge sooner.
> > > >
> > > > What do you think? Comments, concerns?
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/attachment/12644237/hbase-10513_v1.patch
> > > > [2]
> > > >
> > > >
> > >
> >
> http://www.slideshare.net/enissoz/hbase-high-availability-for-reads-with-time
> > > > [3] https://issues.apache.org/jira/browse/HBASE-10070
> > > > [4] https://issues.apache.org/jira/browse/HBASE-11183
> > > >
> > > > Thanks,
> > > >
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message