hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Haeinsa: linear scalable multi-row transaction library for HBase
Date Fri, 08 Nov 2013 13:06:58 GMT
This optimization is feasible. 

Cheers

On Nov 8, 2013, at 4:53 AM, James Lee <roth2520@gmail.com> wrote:

> Thanks for letting us know the good feature.
> 
> Maybe I can find the way to optimize Haeinsa transaction which mutations
> are in single HBase region with MultiRowMutationEndpoint.
> 
> 
> On Thu, Nov 7, 2013 at 11:11 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> 
>> Thanks for the update.
>> 
>> Have you taken a look at the following endpoint in 0.94 ?
>> 
>> src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java
>> 
>> 
>> On Wed, Nov 6, 2013 at 5:40 PM, James Lee <roth2520@gmail.com> wrote:
>> 
>>> That's for optimization of performance in Haeinsa.
>>> We can use fewer HBase operation in single row transaction.
>>> It is related to algorithm design of Haeinsa.
>>> 
>>> For example, consider about following raw-HBase code. (2get/2put to 2
>> rows)
>>> 
>>> get(row1, col1)
>>> get(row2, col2)
>>> put(row1, col1, value1)
>>> put(row2, col2, value2)
>>> 
>>> If you use Haeinsa, this will be executed by following order.
>>> (2get/5checkAndPut)
>>> 
>>> get(row1, col1)
>>> get(row2, col2)
>>> // commit() here
>>> checkAndPut(row1, prevLock1, {col1:value1})
>>> checkAndPut(row2, prevLock2, {col2:value2})
>>> checkAndPut(row1, prevLock1, COMMITED)
>>> checkAndPut(row2, prevLock2, STABLE)
>>> checkAndPut(row1, prevLock1, STABLE)
>>> 
>>> Now, consider single row transaction. (2get/2put to single row)
>>> 
>>> get(row1, col1)
>>> get(row1, col2)
>>> put(row1, col1, value1)
>>> put(row1, col2, value2)
>>> 
>>> If you use Haeinsa, this will be executed by following order.
>>> (2get/1checkAndPut)
>>> 
>>> get(row1, col1)
>>> get(row1, col2)
>>> // commit() here
>>> checkAndPut(row1, prevLock1, {col1:value1, col2:value2, lock:STABLE})
>>> 
>>> Single row transaction needs much fewer operation to achieve ACID
>>> properties in Haeinsa
>>> 
>>> Thanks.
>>> 
>>> 
>>> On Thu, Nov 7, 2013 at 7:40 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>> 
>>>> Jung-Haeng:
>>>> In HaeinsaTransaction.java :
>>>> 
>>>>        /**
>>>>         * If there is only one rowTx and type of its mutation is
>>>> HaeinsaPut.
>>>>         */
>>>>        SINGLE_ROW_PUT_ONLY,
>>>> 
>>>> Can you elaborate a bit on why the single row put gets special
>> treatment
>>> ?
>>>> 
>>>> Cheers
>>>> 
>>>> 
>>>> On Fri, Oct 11, 2013 at 6:09 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>> 
>>>>> I am now able to build on Linux.
>>>>> 
>>>>> Running test suite.
>>>>> 
>>>>> Will look deeper into the code over the weekend.
>>>>> 
>>>>> 
>>>>> On Thu, Oct 10, 2013 at 10:16 PM, Jung-Haeng Lee <roth2520@gmail.com
>>>>> wrote:
>>>>> 
>>>>>> Thanks for your advice.
>>>>>> I fixed to Hbase 0.94.3 on pom.xml.
>>>>>> (To ensure consistency, HBASE-7051 should be solved.)
>>>>>> 
>>>>>> I also fixed pom.xml of Haeinsa:
>>>>>> Now maven-thrift-plugin is activate on linux using default profile.
>>>>>> Please ensure the "thrift" executable is in your PATH.
>>>>>> 
>>>>>> And you should build Haeinsa with jdk 1.7.
>>>>>> I think System.lineSeparator() is new method of Java7.
>>>>>> 
>>>>>> - James Lee
>>>>>> 
>>>>>> 
>>>>>> On Fri, Oct 11, 2013 at 3:18 AM, Ted Yu <yuzhihong@gmail.com>
>> wrote:
>>>>>> 
>>>>>>> Using the 'mac' profile, I got the following:
>>>>>>> 
>>>>>>> Number of foreign imports: 1
>>>>>>> import: Entry[import  from realm ClassRealm[maven.api, parent:
>>> null]]
>>>>>>> 
>>>>>>> -----------------------------------------------------
>>>>>>> 
>>>>>>> at
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
>>>>>>> ... 20 more
>>>>>>> Caused by: java.lang.NoSuchMethodError:
>>>>>>> java.lang.System.lineSeparator()Ljava/lang/String;
>>>>>>> at
>>>> io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>> at java.lang.Class.forName(Class.java:249)
>>>>>>> at
>> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
>>>>>>> at
>> com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
>>>>>>> at
>> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
>>>>>>> at
>>>> com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
>>>>>>> at
>> com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
>>>>>>> at
>> org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
>>>>>>> at
>> org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
>>>>>>> at
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
>>>>>>> ... 20 more
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yuzhihong@gmail.com>
>>> wrote:
>>>>>>> 
>>>>>>>> Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12
>> ?
>>>>>>>> 
>>>>>>>> I cloned the git repo. But I got some compilation errors:
>>>>>>>> 
>>>>>>>> [ERROR]
>> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
>>>>>>>> cannot find symbol
>>>>>>>> [ERROR] symbol:   class TRowKey
>>>>>>>> [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
>>>>>>>> [ERROR]
>> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
>>>>>>>> cannot find symbol
>>>>>>>> [ERROR] symbol:   class TRowKey
>>>>>>>> [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
>>>>>>>> 
>>>>>>>> How do I activate maven-thrift-plugin on Linux ?
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> 
>>>>>>>> On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <
>>>>>>> msegel_hadoop@hotmail.com>wrote:
>>>>>>>> 
>>>>>>>>> Which level of isolation do you support?
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mbkim@vcnc.co.kr>
>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hello everyone in the mailing list.
>>>>>>>>>> 
>>>>>>>>>> I want to introduce an open-source library Haeinsa
which
>>> supports
>>>>>>>>> multi-row, multi-table transaction on HBase.
>>>>>>>>>> Haeinsa is client-only library that support serializability
>> and
>>>>>> linear
>>>>>>>>> scalability.
>>>>>>>>>> Here is the github repository and presentation that
describes
>>>>>>> mechanism
>>>>>>>>> of it.
>>>>>>>>>> 
>>>>>>>>>> github : https://github.com/VCNC/haeinsa
>>>>>>>>>> presentation : https://speakerdeck.com/vcnc/haeinsa-overview
>>>>>>>>>> 
>>>>>>>>>> It was inspired by Google's percolator, but implementation
>>> detail
>>>>>> is
>>>>>>>>> different.
>>>>>>>>>> It use two-phase commit protocol and optimistic concurrency
>>>>>> control to
>>>>>>>>> implement, and Haeinsa now processes more than 300M+
>> transactions
>>>> per
>>>>>>> day
>>>>>>>>> in single cluster without any consistency problem for
more
>> than 2
>>>>>> month.
>>>>>>>>>> I know that there has been lots of libraries and
papers for
>>> HBase
>>>>>>>>> transaction, but within my knowledge, this is the only
>>> open-source
>>>>>>> library
>>>>>>>>> which support serializability and linear scalability.
>>>>>>>>>> There is no theoretical limit of transaction throughput.
>>>>>>>>>> It was tested against cluster on AWS until 40,000
>>>> transaction/sec.
>>>>>>>>> (Still testing on bigger cluster)
>>>>>>>>>> If you find it interesting, please leave me comment.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Andrew Kim
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> *이 정 행 / Jung-Haeng Lee*
>>>>>> *
>>>>>> *
>>>>>> Blog: http://eincs.net
>>>>>> Facebook: http://www.facebook.com/eincs
>>>>>> Twitter: http://twitter.com/eincs
>>>>>> LinkedIn: http://linkedin.com/in/eincs
>>> 
>>> 
>>> 
>>> --
>>> *이 정 행 / Jung-Haeng Lee*
>>> 
>>> Blog: http://eincs.net
>>> Facebook: http://www.facebook.com/eincs
>>> Twitter: http://twitter.com/eincs
>>> LinkedIn: http://linkedin.com/in/eincs
> 
> 
> 
> -- 
> *이 정 행 / Jung-Haeng Lee*
> 
> Blog: http://eincs.net
> Facebook: http://www.facebook.com/eincs
> Twitter: http://twitter.com/eincs
> LinkedIn: http://linkedin.com/in/eincs

Mime
View raw message