hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Lee <roth2...@gmail.com>
Subject Re: Haeinsa: linear scalable multi-row transaction library for HBase
Date Thu, 07 Nov 2013 01:40:51 GMT
That's for optimization of performance in Haeinsa.
We can use fewer HBase operation in single row transaction.
It is related to algorithm design of Haeinsa.

For example, consider about following raw-HBase code. (2get/2put to 2 rows)

get(row1, col1)
get(row2, col2)
put(row1, col1, value1)
put(row2, col2, value2)

If you use Haeinsa, this will be executed by following order.
(2get/5checkAndPut)

get(row1, col1)
get(row2, col2)
// commit() here
checkAndPut(row1, prevLock1, {col1:value1})
checkAndPut(row2, prevLock2, {col2:value2})
checkAndPut(row1, prevLock1, COMMITED)
checkAndPut(row2, prevLock2, STABLE)
checkAndPut(row1, prevLock1, STABLE)

Now, consider single row transaction. (2get/2put to single row)

get(row1, col1)
get(row1, col2)
put(row1, col1, value1)
put(row1, col2, value2)

If you use Haeinsa, this will be executed by following order.
(2get/1checkAndPut)

get(row1, col1)
get(row1, col2)
// commit() here
checkAndPut(row1, prevLock1, {col1:value1, col2:value2, lock:STABLE})

Single row transaction needs much fewer operation to achieve ACID
properties in Haeinsa

Thanks.


On Thu, Nov 7, 2013 at 7:40 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Jung-Haeng:
> In HaeinsaTransaction.java :
>
>         /**
>          * If there is only one rowTx and type of its mutation is
> HaeinsaPut.
>          */
>         SINGLE_ROW_PUT_ONLY,
>
> Can you elaborate a bit on why the single row put gets special treatment ?
>
> Cheers
>
>
> On Fri, Oct 11, 2013 at 6:09 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > I am now able to build on Linux.
> >
> > Running test suite.
> >
> > Will look deeper into the code over the weekend.
> >
> >
> > On Thu, Oct 10, 2013 at 10:16 PM, Jung-Haeng Lee <roth2520@gmail.com
> >wrote:
> >
> >> Thanks for your advice.
> >> I fixed to Hbase 0.94.3 on pom.xml.
> >> (To ensure consistency, HBASE-7051 should be solved.)
> >>
> >> I also fixed pom.xml of Haeinsa:
> >> Now maven-thrift-plugin is activate on linux using default profile.
> >> Please ensure the "thrift" executable is in your PATH.
> >>
> >> And you should build Haeinsa with jdk 1.7.
> >> I think System.lineSeparator() is new method of Java7.
> >>
> >> - James Lee
> >>
> >>
> >> On Fri, Oct 11, 2013 at 3:18 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>
> >> > Using the 'mac' profile, I got the following:
> >> >
> >> > Number of foreign imports: 1
> >> > import: Entry[import  from realm ClassRealm[maven.api, parent: null]]
> >> >
> >> > -----------------------------------------------------
> >> >
> >> > at
> >> >
> >> >
> >>
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
> >> > ... 20 more
> >> > Caused by: java.lang.NoSuchMethodError:
> >> > java.lang.System.lineSeparator()Ljava/lang/String;
> >> > at
> io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
> >> > at java.lang.Class.forName0(Native Method)
> >> > at java.lang.Class.forName(Class.java:249)
> >> > at
> >> >
> >> >
> >>
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
> >> > at
> >> >
> >> >
> >>
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
> >> > at
> >> >
> >> >
> >>
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
> >> > at
> com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
> >> > at
> >> >
> >> >
> >>
> com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
> >> > at
> >> >
> >> >
> >>
> org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
> >> > at
> >> >
> >> >
> >>
> org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
> >> > at
> >> >
> >> >
> >>
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> >> > ... 20 more
> >> >
> >> >
> >> > On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >> >
> >> > > Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12 ?
> >> > >
> >> > > I cloned the git repo. But I got some compilation errors:
> >> > >
> >> > > [ERROR]
> >> > >
> >> >
> >>
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
> >> > > cannot find symbol
> >> > >  [ERROR] symbol:   class TRowKey
> >> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> >> > > [ERROR]
> >> > >
> >> >
> >>
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
> >> > > cannot find symbol
> >> > > [ERROR] symbol:   class TRowKey
> >> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> >> > >
> >> > > How do I activate maven-thrift-plugin on Linux ?
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <
> >> > msegel_hadoop@hotmail.com>wrote:
> >> > >
> >> > >> Which level of isolation do you support?
> >> > >>
> >> > >>
> >> > >> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mbkim@vcnc.co.kr>
> wrote:
> >> > >>
> >> > >> > Hello everyone in the mailing list.
> >> > >> >
> >> > >> > I want to introduce an open-source library Haeinsa which
supports
> >> > >> multi-row, multi-table transaction on HBase.
> >> > >> > Haeinsa is client-only library that support serializability
and
> >> linear
> >> > >> scalability.
> >> > >> > Here is the github repository and presentation that describes
> >> > mechanism
> >> > >> of it.
> >> > >> >
> >> > >> > github : https://github.com/VCNC/haeinsa
> >> > >> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
> >> > >> >
> >> > >> > It was inspired by Google's percolator, but implementation
detail
> >> is
> >> > >> different.
> >> > >> > It use two-phase commit protocol and optimistic concurrency
> >> control to
> >> > >> implement, and Haeinsa now processes more than 300M+ transactions
> per
> >> > day
> >> > >> in single cluster without any consistency problem for more than
2
> >> month.
> >> > >> > I know that there has been lots of libraries and papers for
HBase
> >> > >> transaction, but within my knowledge, this is the only open-source
> >> > library
> >> > >> which support serializability and linear scalability.
> >> > >> > There is no theoretical limit of transaction throughput.
> >> > >> > It was tested against cluster on AWS until 40,000
> transaction/sec.
> >> > >> (Still testing on bigger cluster)
> >> > >> > If you find it interesting, please leave me comment.
> >> > >> >
> >> > >> > Thanks,
> >> > >> > Andrew Kim
> >> > >>
> >> > >>
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> *이 정 행 / Jung-Haeng Lee*
> >> *
> >> *
> >> Blog: http://eincs.net
> >> Facebook: http://www.facebook.com/eincs
> >> Twitter: http://twitter.com/eincs
> >> LinkedIn: http://linkedin.com/in/eincs
> >>
> >
> >
>



-- 
*이 정 행 / Jung-Haeng Lee*

Blog: http://eincs.net
Facebook: http://www.facebook.com/eincs
Twitter: http://twitter.com/eincs
LinkedIn: http://linkedin.com/in/eincs

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message