hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enis Söztutar <enis....@gmail.com>
Subject Re: [DISCUSSION] Update on HBASE-10070 / Merge into trunk
Date Wed, 28 May 2014 00:27:11 GMT
Agreed with Devaraj. I do not think that these changes will destabilize the
code base much. All of the code committed to branch has been unit tested
and integration tests, both specific to region replicas and others have
been running for some time.

Specifically, various large scale data ingestion + verification tests, bulk
load tests, hbck tests, shell tests, replication, mapreduce based tests,
snapshot tests, etc has been running on this code base.

For testing the feature itself, HBASE-10572, HBASE-10616, and HBASE-10818
adds tests for get/multi-get/scan and bulk load which has been running with
CM. HBASE-10817 adds a test for region replicas + replication. HBASE-10791
changes PerformanceEvaluation to be able to work with region replicas.
Nicolas is also working on further perf testing and improvements.

For 1.0, I think it should be fine to include this as an experimental
feature. Otherwise, it will be very hard to add this to the 1.x line.

Are there any more concerns? If not, I would like to raise a VOTE soon.

Enis


On Sat, May 24, 2014 at 10:40 AM, Devaraj Das <ddas@hortonworks.com> wrote:

> Thanks Stack. On the testing, we have been adding unit tests for all
> patches. We have also tested the patches quite extensively on real clusters
> via IT tests (existing and newly added) and issues discovered have been
> fixed. AFAICT, and from the tests run, this feature shouldn't destabilize
> the mainline. There are some limitations when region-replica is used -
> better integration with hbck (HBASE-10674) and support for split/merge (and
> these are being worked on as part of phase 2).
>  On May 23, 2014 9:56 PM, "Stack" <stack@duboce.net> wrote:
>
> > I think merge sooner rather than later.
> >
> > What sort of testing has been done on 10070 and what can you say about
> how
> > well the feature works?
> >
> > Where do you see risk, if any, of it destabilizing the mainline
> especially
> > what with us coming up to a 1.0.
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> > On Wed, May 21, 2014 at 5:08 PM, Enis Söztutar <enis@apache.org> wrote:
> >
> > > Hi,
> > >
> > > We would like to give an update on the status of HBASE-10070 work, and
> > open
> > > up discussion for how we can do further development.
> > >
> > > We seem to be at a point where we have the core functionality of the
> > > region replica, as described in HBASE-10070 working. As pointed out
> > > under the section "Development Phases" in the design doc posted on the
> > > jira HBASE-10070, this work was divided into two broad phases. The
> first
> > > phase introduces region replicas concept, the new consistency model,
> and
> > > corresponding RPC implementations. All of the issues for Phase 1 can be
> > > found under [3]. Phase 2 is still in the works, and contains the
> proposed
> > > changes listed under [4].
> > >
> > > With all the issues committed in HBASE-10070 branch in svn, we think
> that
> > > the "phase-1" is complete. The user documentation on HBASE-10513 gives
> an
> > > accurate picture of what has been done in phase-1 and what the impact
> of
> > > using this feature is, APIs etc. We have added
> > > a couple of IT tests as part of this work and we have tested the work
> > > we did in "phase-1" of the project quite extensively in Hortonworks'
> > > infrastructure.
> > >
> > > In summary, with the code in branch, you can create tables with region
> > > replicas, do gets / multi gets and scans using TIMELINE consistency
> with
> > > high availability. Region replicas periodically scan the files of the
> > > primary and pick up flushed / committed files. The RPC paths /
> > assignment,
> > > balancing etc are pretty stable. However some more performance analysis
> > and
> > > tuning is needed. More information can be found in [1] and [2].
> > >
> > >
> > > As a reminder, the development has been happening in the branch -
> > > hbase-10070 (https://github.com/apache/hbase/tree/hbase-10070). We
> have
> > > been pretty diligent about more than one committer's +1 on the branch
> > > commits and for almost all the subtasks in HBASE-10070 there is more
> than
> > > one +1 except for test fix issues or more trivial issues, where there
> > maybe
> > >  only one +1.  Enis/Nicolas/Sergey/Devaraj/Nick are the main
> contributors
> > > of code in the phase-1 and many patches have been reviewed already by
> > > people outside
> > > this group (mainly Stack, Jimmy)
> > >
> > > For Phase 2, we think that we can deliver on the proposed design
> > > incrementally over the next couple of months. However, we think that it
> > > might be ok to merge the changes from phase 1 first, then do a
> > > commit-as-you-go approach for remaining items. Therefore, we would like
> > to
> > > propose  to merge the branch to trunk, and continue the work over
> there.
> > > This might also result in more reviews.
> > >
> > > Alternatively, we can continue the work in the branch, and do the merge
> > at
> > > the end of Phase 2, but that will make the review process a bit more
> > > tricky, which is why we would like the merge sooner.
> > >
> > > What do you think? Comments, concerns?
> > >
> > > [1]
> > >
> > >
> >
> https://issues.apache.org/jira/secure/attachment/12644237/hbase-10513_v1.patch
> > > [2]
> > >
> > >
> >
> http://www.slideshare.net/enissoz/hbase-high-availability-for-reads-with-time
> > > [3] https://issues.apache.org/jira/browse/HBASE-10070
> > > [4] https://issues.apache.org/jira/browse/HBASE-11183
> > >
> > > Thanks,
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message