hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Themis : implements cross-row/corss-table transaction on HBase.
Date Tue, 11 Nov 2014 03:32:49 GMT
bq. do we have hadoop2 version for hbase 0.94 in central maven repository?

Not that I know of.

w.r.t. dependency on hbase-annotations module, maybe you can introduce a
new profile that includes this dependency.

Cheers

On Mon, Nov 10, 2014 at 7:18 PM, 崔建伟 <cuijianwei@xiaomi.com> wrote:

> Hi Ted:
>  Thanks for your advice. Yes, it is more reasonable to add common
> properties in parent pom:). I update themis to set hbase version/hadoop
> version in parent pom, and set hbase version to 0.98.5-hadoop2 for
> for_hbase_0.98 branch. The unit tests are passed under these modifications.
>   When using 0.98.7-hadoop2 in for_hbase_0.98 branch, there are some
> compile errors(missing MediumTests class), we cam add the following
> dependencies to the pom of themis-client to include MediumTests class.
>    <dependency>
>       <groupId>org.apache.hbase</groupId>
>       <artifactId>hbase-annotations</artifactId>
>       <version>${hbase.version}</version>
>       <classifier>tests</classifier>
>       <scope>test</scope>
>    </dependency>
>
>  There are some common dependencies among sub-projects, I will polish pom
> to add common dependencies to parent pom.
>   BTW, do we have hadoop2 version for hbase 0.94 in central maven
> repository?
>
> Best
> cuijianwei
> ________________________________________
> From: Ted Yu <yuzhihong@gmail.com>
> Sent: Tuesday, November 11, 2014 9:43 AM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: Themis : implements cross-row/corss-table transaction on
> HBase.
>
> Thanks Jianwei for the suggestion.
>
> I checked out for_hbase_0.98 but got the following error when building:
>
> [ERROR] Failed to execute goal on project themis-protocol: Could not
> resolve dependencies for project
> com.xiaomi.infra:themis-protocol:jar:1.0-SNAPSHOT: Could not find artifact
> org.apache.hbase:hbase-protocol:jar:0.98.5 in central (
> http://repo.maven.apache.org/maven2) -> [Help 1]
>
> I think this is due to the fact that 0.98 is released with two profiles:
> one for hadoop-1 and one for hadoop-2
>
> Currently each module of themis defines hbase version as:
>
> $ find . -name 'pom.xml' -exec grep '0.98' {} \; -print
>     <hbase.version>0.98.5</hbase.version>
> ./themis-client/pom.xml
>     <hbase.version>0.98.5</hbase.version>
> ./themis-coprocessor/pom.xml
>     <hbase.version>0.98.5</hbase.version>
> ./themis-index/pom.xml
>     <hbase.version>0.98.5</hbase.version>
> ./themis-protocol/pom.xml
>
> I tried adding -Dhbase.version=0.98.7-hadoop2 to maven command line but
> still got same error.
>
> You can add a property to parent pom.xml which defines hbase version so
> that child pom.xml can reference.
> That way, it is easy to override on command line.
>
> Cheers
>
> On Mon, Nov 10, 2014 at 5:13 PM, 崔建伟 <cuijianwei@xiaomi.com> wrote:
>
> > Hi Ted:
> >  thanks for your feedback. I tried to clone for_hbase_0.98 branch by:
> "git
> > clone https://github.com/XiaoMi/themis/tree/for_hbase_0.98", it returns:
> >  "error: The requested URL returned error: 403 while accessing
> > https://github.com/XiaoMi/themis/tree/for_hbase_0.98/info/refs".
> >
> >  I think we can clone themis by: "git clone
> > https://github.com/XiaoMi/themis" firstly. Then, view the remote
> > for_hbase_0.98 branch by:
> >  "git branch -r" and get:
> >   origin/HEAD -> origin/master
> >   origin/for_hbase_0.98
> >   origin/master
> >
> >   And then checkout remote origin/for_hbase_0.98 to local as:
> >   "git checkout -b for_hbase_0.98 origin/for_hbase_0.98"
> >
> > Thanks.
> >
> > Best
> > cuijianwei
> > ________________________________________
> > From: Ted Yu <yuzhihong@gmail.com>
> > Sent: Monday, November 10, 2014 11:39 PM
> > To: dev@hbase.apache.org
> > Cc: user@hbase.apache.org
> > Subject: Re: Themis : implements cross-row/corss-table transaction on
> > HBase.
> >
> > Jianwei:
> > I used this command to clone your repo:
> > git clone https://github.com/XiaoMi/themis/tree/for_hbase_0.98 themis
> >
> > But I only found 0.94 being referenced in the pom.xml files:
> >
> > $ find . -name pom.xml -exec grep '0.94.' {} \; -print
> >     <hbase.version>0.94.21</hbase.version>
> > ./themis-client/pom.xml
> >     <hbase.version>0.94.21</hbase.version>
> > ./themis-coprocessor/pom.xml
> >     <hbase.version>0.94.21</hbase.version>
> > ./themis-index/pom.xml
> >
> > Did I miss something ?
> >
> > Cheers
> >
> > On Mon, Nov 10, 2014 at 12:50 AM, 崔建伟 <cuijianwei@xiaomi.com> wrote:
> >
> > > Hi everyone:
> > > In last few months, we have updated Themis to achieve better
> performance
> > > and include more features:
> > >
> > > 1. Improve the single-row write performance from 23%(relative drop
> > > compared with HBase's put) to 60%(for most test cases). For single-row
> > > write transaction, we only write lock to MemStore in prewrite-phase,
> > then,
> > > we erase corresponding lock, write data and commit information to HLog
> in
> > > commit-phase. This won't break the correctness of percolator algorithm
> > and
> > > will help improve the performance a lot for single-row write.
> > >
> > > 2. Support HBase 0.98. We create a branch:
> > > https://github.com/XiaoMi/themis/tree/for_hbase_0.98 to make themis
> > > support HBase 0.98(Currently, support HBase 0.98.5). All the functions
> of
> > > master branch will also be implemented in this branch.
> > >
> > > 3. Transaction TTL support and Old Data Clean. Users could set TTL for
> > > read/write transaction respectively. Then, old data which could not be
> > read
> > > will be cleaned periodically.
> > >
> > > 4. MapReduce Support. We implement a group of classes to support read
> > data
> > > by themis transaction in Mapper job and write data by themis
> transaction
> > in
> > > Reduce job.
> > >
> > > For more details, please see the github:
> > > https://github.com/XiaoMi/themis(or
> > > https://github.com/XiaoMi/themis/tree/for_hbase_0.98) or jira:
> > > https://issues.apache.org/jira/browse/HBASE-10999 . If you find Themis
> > > interesting, please leave us comment in the mail, jira or github.
> > >
> > > Best
> > > cuijianwei
> > >
> > > ________________________________________
> > > From: saint.ack@gmail.com <saint.ack@gmail.com> on behalf of Stack <
> > > stack@duboce.net>
> > > Sent: Sunday, July 13, 2014 1:12 PM
> > > To: HBase Dev List
> > > Cc: user@hbase.apache.org
> > > Subject: Re: Themis : implements cross-row/corss-table transaction on
> > > HBase.
> > >
> > > On Tue, Jul 8, 2014 at 12:34 AM, 崔建伟 <cuijianwei@xiaomi.com> wrote:
> > >
> > > > Hi everyone, I want to introduce our open-source project Themis which
> > > > implements cross-row/corss-table transaction on HBase.
> > > >
> > > > Themis follows google's percolator algorithm(
> > > > http://research.google.com/pubs/pub36726.html), which provides
> > > > ACID-compliant transaction and snapshot isolation. The cross-row
> > > > transaction is based on HBase's single-row atomic semantics and
> doesn't
> > > use
> > > > a central transaction server, so that supports linear-scalability.
> > > >
> > > > Themis depends on a timestamp server to provides global strictly
> > > > incremental timestamp to define the order of transactions, which will
> > be
> > > > used to resolve the write-write and read-write conflicts. The
> timestamp
> > > > server is lightweight and could achieve hight throughput(500, 000 +
> > qps),
> > > > and Themis will batch timestamp requests across transactions in one
> > Rpc,
> > > so
> > > > that it won't become the bottleneck of the system even when
> processing
> > > > billions of transactions every day.
> > > >
> > > > Although Themis could be implemented totally in client-side, we adopt
> > > > coprocessor framework of HBase to achieve higher performance. Themis
> > > > includes a client-side library to provides transaction APIs, such as
> > > > themisPut/themisGet/themisScan/themisDelete, and a coprocessor
> library
> > > > loaded on regionserver. Therefore, Themis could be used without
> > changing
> > > > the code and logic of HBase.
> > > >
> > > > We have been validating the correctness of Themis for a few months
> by a
> > > > AccountTransfer simulation program, which concurrently does cross-row
> > > > transactions by transferring money among different accounts(each
> > account
> > > is
> > > > a row in HBase) and verifies total money of all accounts doesn't
> change
> > > in
> > > > the simulation. We have also run Themis on our production
> environment.
> > > >
> > > > We test the performance of Themis and get comparable result as
> > > percolator.
> > > > The single-column transaction represents the worst performance case
> for
> > > > Themis compared with HBase, the result is:
> > > > 1) For read, the performance of percolator is 90% of HBase;
> > > > 2) For write, the performance of percolator is 23% of HBase.
> > > > The write performance drops a lot because Themis uses two-phase
> commit
> > > > protocol to achieve ACID of transaction. For multi-row write, we
> > improve
> > > > the performance by paralleling all writes of pre-write phase. For
> > > > single-row write, we are optimizing two-phase commit protocol to
> > achieve
> > > > better performance and will update the result when it is ready. The
> > > details
> > > > of performance result could be found in github.
> > > >
> > > > The repository and introduction of Themis include:
> > > > 1. Themis github: https://github.com/XiaoMi/themis/. The source
> code,
> > > > performance test result and user guide could be found here.
> > > > 2. Themis jira : https://issues.apache.org/jira/browse/HBASE-10999
> > > > 3. Chronos github: https://github.com/XiaoMi/chronos. Chronos is our
> > > > open-source high-availability, high-performance timestamp server to
> > > provide
> > > > global strictly incremental timestamp for Themis.
> > > >
> > > > If you find Themis interesting, please leave us comment in the mail,
> > jira
> > > > or github.
> > > >
> > > > Best
> > > > cuijianwei
> > > >
> > >
> > > Excellent.  Nice work lads!
> > > St.Ack
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message