hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Themis : implements cross-row/corss-table transaction on HBase.
Date Tue, 11 Nov 2014 01:43:55 GMT
Thanks Jianwei for the suggestion.

I checked out for_hbase_0.98 but got the following error when building:

[ERROR] Failed to execute goal on project themis-protocol: Could not
resolve dependencies for project
com.xiaomi.infra:themis-protocol:jar:1.0-SNAPSHOT: Could not find artifact
org.apache.hbase:hbase-protocol:jar:0.98.5 in central (
http://repo.maven.apache.org/maven2) -> [Help 1]

I think this is due to the fact that 0.98 is released with two profiles:
one for hadoop-1 and one for hadoop-2

Currently each module of themis defines hbase version as:

$ find . -name 'pom.xml' -exec grep '0.98' {} \; -print
    <hbase.version>0.98.5</hbase.version>
./themis-client/pom.xml
    <hbase.version>0.98.5</hbase.version>
./themis-coprocessor/pom.xml
    <hbase.version>0.98.5</hbase.version>
./themis-index/pom.xml
    <hbase.version>0.98.5</hbase.version>
./themis-protocol/pom.xml

I tried adding -Dhbase.version=0.98.7-hadoop2 to maven command line but
still got same error.

You can add a property to parent pom.xml which defines hbase version so
that child pom.xml can reference.
That way, it is easy to override on command line.

Cheers

On Mon, Nov 10, 2014 at 5:13 PM, 崔建伟 <cuijianwei@xiaomi.com> wrote:

> Hi Ted:
>  thanks for your feedback. I tried to clone for_hbase_0.98 branch by: "git
> clone https://github.com/XiaoMi/themis/tree/for_hbase_0.98", it returns:
>  "error: The requested URL returned error: 403 while accessing
> https://github.com/XiaoMi/themis/tree/for_hbase_0.98/info/refs".
>
>  I think we can clone themis by: "git clone
> https://github.com/XiaoMi/themis" firstly. Then, view the remote
> for_hbase_0.98 branch by:
>  "git branch -r" and get:
>   origin/HEAD -> origin/master
>   origin/for_hbase_0.98
>   origin/master
>
>   And then checkout remote origin/for_hbase_0.98 to local as:
>   "git checkout -b for_hbase_0.98 origin/for_hbase_0.98"
>
> Thanks.
>
> Best
> cuijianwei
> ________________________________________
> From: Ted Yu <yuzhihong@gmail.com>
> Sent: Monday, November 10, 2014 11:39 PM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: Themis : implements cross-row/corss-table transaction on
> HBase.
>
> Jianwei:
> I used this command to clone your repo:
> git clone https://github.com/XiaoMi/themis/tree/for_hbase_0.98 themis
>
> But I only found 0.94 being referenced in the pom.xml files:
>
> $ find . -name pom.xml -exec grep '0.94.' {} \; -print
>     <hbase.version>0.94.21</hbase.version>
> ./themis-client/pom.xml
>     <hbase.version>0.94.21</hbase.version>
> ./themis-coprocessor/pom.xml
>     <hbase.version>0.94.21</hbase.version>
> ./themis-index/pom.xml
>
> Did I miss something ?
>
> Cheers
>
> On Mon, Nov 10, 2014 at 12:50 AM, 崔建伟 <cuijianwei@xiaomi.com> wrote:
>
> > Hi everyone:
> > In last few months, we have updated Themis to achieve better performance
> > and include more features:
> >
> > 1. Improve the single-row write performance from 23%(relative drop
> > compared with HBase's put) to 60%(for most test cases). For single-row
> > write transaction, we only write lock to MemStore in prewrite-phase,
> then,
> > we erase corresponding lock, write data and commit information to HLog in
> > commit-phase. This won't break the correctness of percolator algorithm
> and
> > will help improve the performance a lot for single-row write.
> >
> > 2. Support HBase 0.98. We create a branch:
> > https://github.com/XiaoMi/themis/tree/for_hbase_0.98 to make themis
> > support HBase 0.98(Currently, support HBase 0.98.5). All the functions of
> > master branch will also be implemented in this branch.
> >
> > 3. Transaction TTL support and Old Data Clean. Users could set TTL for
> > read/write transaction respectively. Then, old data which could not be
> read
> > will be cleaned periodically.
> >
> > 4. MapReduce Support. We implement a group of classes to support read
> data
> > by themis transaction in Mapper job and write data by themis transaction
> in
> > Reduce job.
> >
> > For more details, please see the github:
> > https://github.com/XiaoMi/themis(or
> > https://github.com/XiaoMi/themis/tree/for_hbase_0.98) or jira:
> > https://issues.apache.org/jira/browse/HBASE-10999 . If you find Themis
> > interesting, please leave us comment in the mail, jira or github.
> >
> > Best
> > cuijianwei
> >
> > ________________________________________
> > From: saint.ack@gmail.com <saint.ack@gmail.com> on behalf of Stack <
> > stack@duboce.net>
> > Sent: Sunday, July 13, 2014 1:12 PM
> > To: HBase Dev List
> > Cc: user@hbase.apache.org
> > Subject: Re: Themis : implements cross-row/corss-table transaction on
> > HBase.
> >
> > On Tue, Jul 8, 2014 at 12:34 AM, 崔建伟 <cuijianwei@xiaomi.com> wrote:
> >
> > > Hi everyone, I want to introduce our open-source project Themis which
> > > implements cross-row/corss-table transaction on HBase.
> > >
> > > Themis follows google's percolator algorithm(
> > > http://research.google.com/pubs/pub36726.html), which provides
> > > ACID-compliant transaction and snapshot isolation. The cross-row
> > > transaction is based on HBase's single-row atomic semantics and doesn't
> > use
> > > a central transaction server, so that supports linear-scalability.
> > >
> > > Themis depends on a timestamp server to provides global strictly
> > > incremental timestamp to define the order of transactions, which will
> be
> > > used to resolve the write-write and read-write conflicts. The timestamp
> > > server is lightweight and could achieve hight throughput(500, 000 +
> qps),
> > > and Themis will batch timestamp requests across transactions in one
> Rpc,
> > so
> > > that it won't become the bottleneck of the system even when processing
> > > billions of transactions every day.
> > >
> > > Although Themis could be implemented totally in client-side, we adopt
> > > coprocessor framework of HBase to achieve higher performance. Themis
> > > includes a client-side library to provides transaction APIs, such as
> > > themisPut/themisGet/themisScan/themisDelete, and a coprocessor library
> > > loaded on regionserver. Therefore, Themis could be used without
> changing
> > > the code and logic of HBase.
> > >
> > > We have been validating the correctness of Themis for a few months by a
> > > AccountTransfer simulation program, which concurrently does cross-row
> > > transactions by transferring money among different accounts(each
> account
> > is
> > > a row in HBase) and verifies total money of all accounts doesn't change
> > in
> > > the simulation. We have also run Themis on our production environment.
> > >
> > > We test the performance of Themis and get comparable result as
> > percolator.
> > > The single-column transaction represents the worst performance case for
> > > Themis compared with HBase, the result is:
> > > 1) For read, the performance of percolator is 90% of HBase;
> > > 2) For write, the performance of percolator is 23% of HBase.
> > > The write performance drops a lot because Themis uses two-phase commit
> > > protocol to achieve ACID of transaction. For multi-row write, we
> improve
> > > the performance by paralleling all writes of pre-write phase. For
> > > single-row write, we are optimizing two-phase commit protocol to
> achieve
> > > better performance and will update the result when it is ready. The
> > details
> > > of performance result could be found in github.
> > >
> > > The repository and introduction of Themis include:
> > > 1. Themis github: https://github.com/XiaoMi/themis/. The source code,
> > > performance test result and user guide could be found here.
> > > 2. Themis jira : https://issues.apache.org/jira/browse/HBASE-10999
> > > 3. Chronos github: https://github.com/XiaoMi/chronos. Chronos is our
> > > open-source high-availability, high-performance timestamp server to
> > provide
> > > global strictly incremental timestamp for Themis.
> > >
> > > If you find Themis interesting, please leave us comment in the mail,
> jira
> > > or github.
> > >
> > > Best
> > > cuijianwei
> > >
> >
> > Excellent.  Nice work lads!
> > St.Ack
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message