hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Lee <roth2...@gmail.com>
Subject Re: Haeinsa: linear scalable multi-row transaction library for HBase
Date Fri, 08 Nov 2013 12:53:45 GMT
Thanks for letting us know the good feature.

Maybe I can find the way to optimize Haeinsa transaction which mutations
are in single HBase region with MultiRowMutationEndpoint.


On Thu, Nov 7, 2013 at 11:11 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Thanks for the update.
>
> Have you taken a look at the following endpoint in 0.94 ?
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java
>
>
> On Wed, Nov 6, 2013 at 5:40 PM, James Lee <roth2520@gmail.com> wrote:
>
> > That's for optimization of performance in Haeinsa.
> > We can use fewer HBase operation in single row transaction.
> > It is related to algorithm design of Haeinsa.
> >
> > For example, consider about following raw-HBase code. (2get/2put to 2
> rows)
> >
> > get(row1, col1)
> > get(row2, col2)
> > put(row1, col1, value1)
> > put(row2, col2, value2)
> >
> > If you use Haeinsa, this will be executed by following order.
> > (2get/5checkAndPut)
> >
> > get(row1, col1)
> > get(row2, col2)
> > // commit() here
> > checkAndPut(row1, prevLock1, {col1:value1})
> > checkAndPut(row2, prevLock2, {col2:value2})
> > checkAndPut(row1, prevLock1, COMMITED)
> > checkAndPut(row2, prevLock2, STABLE)
> > checkAndPut(row1, prevLock1, STABLE)
> >
> > Now, consider single row transaction. (2get/2put to single row)
> >
> > get(row1, col1)
> > get(row1, col2)
> > put(row1, col1, value1)
> > put(row1, col2, value2)
> >
> > If you use Haeinsa, this will be executed by following order.
> > (2get/1checkAndPut)
> >
> > get(row1, col1)
> > get(row1, col2)
> > // commit() here
> > checkAndPut(row1, prevLock1, {col1:value1, col2:value2, lock:STABLE})
> >
> > Single row transaction needs much fewer operation to achieve ACID
> > properties in Haeinsa
> >
> > Thanks.
> >
> >
> > On Thu, Nov 7, 2013 at 7:40 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Jung-Haeng:
> > > In HaeinsaTransaction.java :
> > >
> > >         /**
> > >          * If there is only one rowTx and type of its mutation is
> > > HaeinsaPut.
> > >          */
> > >         SINGLE_ROW_PUT_ONLY,
> > >
> > > Can you elaborate a bit on why the single row put gets special
> treatment
> > ?
> > >
> > > Cheers
> > >
> > >
> > > On Fri, Oct 11, 2013 at 6:09 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > I am now able to build on Linux.
> > > >
> > > > Running test suite.
> > > >
> > > > Will look deeper into the code over the weekend.
> > > >
> > > >
> > > > On Thu, Oct 10, 2013 at 10:16 PM, Jung-Haeng Lee <roth2520@gmail.com
> > > >wrote:
> > > >
> > > >> Thanks for your advice.
> > > >> I fixed to Hbase 0.94.3 on pom.xml.
> > > >> (To ensure consistency, HBASE-7051 should be solved.)
> > > >>
> > > >> I also fixed pom.xml of Haeinsa:
> > > >> Now maven-thrift-plugin is activate on linux using default profile.
> > > >> Please ensure the "thrift" executable is in your PATH.
> > > >>
> > > >> And you should build Haeinsa with jdk 1.7.
> > > >> I think System.lineSeparator() is new method of Java7.
> > > >>
> > > >> - James Lee
> > > >>
> > > >>
> > > >> On Fri, Oct 11, 2013 at 3:18 AM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> > > >>
> > > >> > Using the 'mac' profile, I got the following:
> > > >> >
> > > >> > Number of foreign imports: 1
> > > >> > import: Entry[import  from realm ClassRealm[maven.api, parent:
> > null]]
> > > >> >
> > > >> > -----------------------------------------------------
> > > >> >
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
> > > >> > ... 20 more
> > > >> > Caused by: java.lang.NoSuchMethodError:
> > > >> > java.lang.System.lineSeparator()Ljava/lang/String;
> > > >> > at
> > > io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
> > > >> > at java.lang.Class.forName0(Native Method)
> > > >> > at java.lang.Class.forName(Class.java:249)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
> > > >> > at
> > > com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> > > >> > ... 20 more
> > > >> >
> > > >> >
> > > >> > On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yuzhihong@gmail.com>
> > wrote:
> > > >> >
> > > >> > > Looks like HBase 0.92.1 is used. How about upgrading to
0.94.12
> ?
> > > >> > >
> > > >> > > I cloned the git repo. But I got some compilation errors:
> > > >> > >
> > > >> > > [ERROR]
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
> > > >> > > cannot find symbol
> > > >> > >  [ERROR] symbol:   class TRowKey
> > > >> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> > > >> > > [ERROR]
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
> > > >> > > cannot find symbol
> > > >> > > [ERROR] symbol:   class TRowKey
> > > >> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> > > >> > >
> > > >> > > How do I activate maven-thrift-plugin on Linux ?
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> > > On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <
> > > >> > msegel_hadoop@hotmail.com>wrote:
> > > >> > >
> > > >> > >> Which level of isolation do you support?
> > > >> > >>
> > > >> > >>
> > > >> > >> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mbkim@vcnc.co.kr>
> > > wrote:
> > > >> > >>
> > > >> > >> > Hello everyone in the mailing list.
> > > >> > >> >
> > > >> > >> > I want to introduce an open-source library Haeinsa
which
> > supports
> > > >> > >> multi-row, multi-table transaction on HBase.
> > > >> > >> > Haeinsa is client-only library that support serializability
> and
> > > >> linear
> > > >> > >> scalability.
> > > >> > >> > Here is the github repository and presentation
that describes
> > > >> > mechanism
> > > >> > >> of it.
> > > >> > >> >
> > > >> > >> > github : https://github.com/VCNC/haeinsa
> > > >> > >> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
> > > >> > >> >
> > > >> > >> > It was inspired by Google's percolator, but implementation
> > detail
> > > >> is
> > > >> > >> different.
> > > >> > >> > It use two-phase commit protocol and optimistic
concurrency
> > > >> control to
> > > >> > >> implement, and Haeinsa now processes more than 300M+
> transactions
> > > per
> > > >> > day
> > > >> > >> in single cluster without any consistency problem for
more
> than 2
> > > >> month.
> > > >> > >> > I know that there has been lots of libraries and
papers for
> > HBase
> > > >> > >> transaction, but within my knowledge, this is the only
> > open-source
> > > >> > library
> > > >> > >> which support serializability and linear scalability.
> > > >> > >> > There is no theoretical limit of transaction throughput.
> > > >> > >> > It was tested against cluster on AWS until 40,000
> > > transaction/sec.
> > > >> > >> (Still testing on bigger cluster)
> > > >> > >> > If you find it interesting, please leave me comment.
> > > >> > >> >
> > > >> > >> > Thanks,
> > > >> > >> > Andrew Kim
> > > >> > >>
> > > >> > >>
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> *이 정 행 / Jung-Haeng Lee*
> > > >> *
> > > >> *
> > > >> Blog: http://eincs.net
> > > >> Facebook: http://www.facebook.com/eincs
> > > >> Twitter: http://twitter.com/eincs
> > > >> LinkedIn: http://linkedin.com/in/eincs
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > *이 정 행 / Jung-Haeng Lee*
> >
> > Blog: http://eincs.net
> > Facebook: http://www.facebook.com/eincs
> > Twitter: http://twitter.com/eincs
> > LinkedIn: http://linkedin.com/in/eincs
> >
>



-- 
*이 정 행 / Jung-Haeng Lee*

Blog: http://eincs.net
Facebook: http://www.facebook.com/eincs
Twitter: http://twitter.com/eincs
LinkedIn: http://linkedin.com/in/eincs

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message