hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1252) Make atomic increment perform a binary increment
Date Tue, 10 Mar 2009 20:36:50 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680604#action_12680604
] 

Jonathan Gray commented on HBASE-1252:
--------------------------------------

Okay.  Got a far more optimized binary increment written now.  Beats long method across the
board.  Two methods of benchmarking, one which just runs a bunch of tests in succession, the
other includes my attempts at trying to prevent optimizations (especially starting a new jvm
for each test).  It made no difference in timings which order I tested them in.

4-10X performance improvement across the board now.

Doing my best to reduce optimizations:

{code}
[hbase@mb0 src]$ java IncrementTest 1000000 1
incrementAsLong : Incremented 1000000 times by 1 in 209 ms (current value is 1000000)
incrementAsBytes : Incremened 1000000 times by 1 in 20 ms (current value is 1000000)
[hbase@mb0 src]$ java IncrementTest 1000000 10
incrementAsLong : Incremented 1000000 times by 10 in 210 ms (current value is 10000000)
incrementAsBytes : Incremened 1000000 times by 10 in 20 ms (current value is 10000000)
[hbase@mb0 src]$ java IncrementTest 1000000 100
incrementAsLong : Incremented 1000000 times by 100 in 210 ms (current value is 100000000)
incrementAsBytes : Incremened 1000000 times by 100 in 26 ms (current value is 100000000)
[hbase@mb0 src]$ java IncrementTest 1000000 1000
incrementAsLong : Incremented 1000000 times by 1000 in 209 ms (current value is 1000000000)
incrementAsBytes : Incremened 1000000 times by 1000 in 30 ms (current value is 1000000000)
[hbase@mb0 src]$ java IncrementTest 1000000 10000
incrementAsLong : Incremented 1000000 times by 10000 in 211 ms (current value is 10000000000)
incrementAsBytes : Incremened 1000000 times by 10000 in 31 ms (current value is 10000000000)
[hbase@mb0 src]$ java IncrementTest 1000000 100000
incrementAsLong : Incremented 1000000 times by 100000 in 211 ms (current value is 100000000000)
incrementAsBytes : Incremened 1000000 times by 100000 in 41 ms (current value is 100000000000)
[hbase@mb0 src]$ java IncrementTest 1000000 1000000
incrementAsLong : Incremented 1000000 times by 1000000 in 209 ms (current value is 1000000000000)
incrementAsBytes : Incremened 1000000 times by 1000000 in 37 ms (current value is 1000000000000)
[hbase@mb0 src]$ java IncrementTest 1000000 10000000
incrementAsLong : Incremented 1000000 times by 10000000 in 213 ms (current value is 10000000000000)
incrementAsBytes : Incremened 1000000 times by 10000000 in 46 ms (current value is 10000000000000)
[hbase@mb0 src]$ java IncrementTest 1000000 100000000
incrementAsLong : Incremented 1000000 times by 100000000 in 213 ms (current value is 100000000000000)
incrementAsBytes : Incremened 1000000 times by 100000000 in 43 ms (current value is 100000000000000)
[hbase@mb0 src]$ java IncrementTest 1000000 1000000000
incrementAsLong : Incremented 1000000 times by 1000000000 in 211 ms (current value is 1000000000000000)
incrementAsBytes : Incremened 1000000 times by 1000000000 in 55 ms (current value is 1000000000000000)
{code}

Just a big sequence of tests in a single method:

{code}
incrementAsLong : Incremented 100000 times by 1 in 91 ms (current value is 100000)
incrementAsBytes : Incremened 100000 times by 1 in 11 ms (current value is 100000)
incrementAsLong : Incremented 100000 times by 10 in 26 ms (current value is 1000000)
incrementAsBytes : Incremened 100000 times by 10 in 1 ms (current value is 1000000)
incrementAsLong : Incremented 100000 times by 100 in 19 ms (current value is 10000000)
incrementAsBytes : Incremened 100000 times by 100 in 1 ms (current value is 10000000)
incrementAsLong : Incremented 100000 times by 1000 in 19 ms (current value is 100000000)
incrementAsBytes : Incremened 100000 times by 1000 in 1 ms (current value is 100000000)
incrementAsLong : Incremented 100000 times by 10000 in 20 ms (current value is 1000000000)
incrementAsBytes : Incremened 100000 times by 10000 in 1 ms (current value is 1000000000)
incrementAsLong : Incremented 100000 times by 100000 in 9 ms (current value is 10000000000)
incrementAsBytes : Incremened 100000 times by 100000 in 2 ms (current value is 10000000000)
incrementAsLong : Incremented 100000 times by 1000000 in 9 ms (current value is 100000000000)
incrementAsBytes : Incremened 100000 times by 1000000 in 2 ms (current value is 100000000000)
incrementAsLong : Incremented 100000 times by 10000000 in 9 ms (current value is 1000000000000)
incrementAsBytes : Incremened 100000 times by 10000000 in 4 ms (current value is 1000000000000)
incrementAsLong : Incremented 100000 times by 100000000 in 9 ms (current value is 10000000000000)
incrementAsBytes : Incremened 100000 times by 100000000 in 3 ms (current value is 10000000000000)
incrementAsLong : Incremented 100000 times by 1000000000 in 10 ms (current value is 100000000000000)
incrementAsBytes : Incremened 100000 times by 1000000000 in 4 ms (current value is 100000000000000)
{code}

> Make atomic increment perform a binary increment
> ------------------------------------------------
>
>                 Key: HBASE-1252
>                 URL: https://issues.apache.org/jira/browse/HBASE-1252
>             Project: Hadoop HBase
>          Issue Type: Improvement
>    Affects Versions: 0.19.0
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Minor
>             Fix For: 0.19.1, 0.20.0
>
>
> A few issues related to recently committed HBASE-803
> - The HTable api still takes an integer amount rather than long, mismatching HRI.
> - Binary increments are 10 times faster for small amounts than going Bytes.toLong, +=
amount, Bytes.toBytes.  Twice as fast for large amounts (binary incrementor just loops a bunch
of single increments, though there is plenty of room for optimizations in my current implementation)
> - Using a binary increment means we don't have to worry about the size of the value.
 If someone wants a 16 byte value they can have it, just have to initialize as such.  If no
existing value exists, will default to long/8 bytes.  Only odd behavior will be what happens
when you are at the max value, currently will just stay at all 11111 binary.  Could actually
grow the byte[] but then we can't do things in place. I'm okay with leaving it like that,
not exactly sure what the current implementation would do, throw an exception or wrap?
> - Using binary incrementing, we can directly manipulate values in the memcache rather
than sending updates with the same timestamp.  I think we should hold off on doing this until
HBASE-1234 goes in.  We'll then have to deal directly with hlog.  (this issue is not going
to address this)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message