hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject minor compaction bug (was: upsert case performance problem)
Date Sat, 19 Mar 2011 18:15:21 GMT
See below.

Doing some testing on that I let the mapreduce program and an hbase shell flushing every 60
seconds run overnight. The result on two tables was:

562 store files!

   ip-10-170-34-18.us-west-1.compute.internal:60020 1300494200158
       requests=51, regions=1, usedHeap=1980, maxHeap=3960
             stores=1, storefiles=562, storefileSizeMB=310, memstoreSizeMB=1, storefileIndexSizeMB=2

528 store files!

    ip-10-170-49-35.us-west-1.compute.internal:60020 1300494214101
        requests=79, regions=1, usedHeap=1830, maxHeap=3960
             stores=1, storefiles=528, storefileSizeMB=460, memstoreSizeMB=3, storefileIndexSizeMB=3

... so that killed performance after a while ...

Here's something else.

   - Andy

--- On Sat, 3/19/11, Andrew Purtell <apurtell@apache.org> wrote:

From: Andrew Purtell <apurtell@apache.org>
Subject: upsert case performance problem (doubts about ConcurrentSkipListMap)
To: dev@hbase.apache.org
Date: Saturday, March 19, 2011, 11:10 AM

I have a mapreduce task put together for experimentation which does a lot of Increments over
three tables and Puts to another. I set writeToWAL to false. My HBase includes the patch that
fixes serialization of writeToWAL for Increments. MemstoreLAB is enabled but is probably not
a factor, but still need to test to exclude it.

After starting a job up on a test cluster on EC2 with 20 mappers over 10 slaves I see initially
10-15K/ops/sec/server. This performance drops over a short time to stabilize around 1K/ops/sec/server.
So I flush the tables with the shell. Immediately after flushing the tables, performance is
back up to 10-15K/ops/sec/server. If I don't flush, performance remains low indefinitely.
If I flush only the table receiving the Gets, performance remains low. 

If I set the shell to flush in a loop every 60 seconds, performance repeatedly drops during
that interval, then recovers after flushing.

When Gary and I went to NCHC in Taiwan, we saw a guy from PhiCloud present something similar
to this regarding 0.89DR. He measured the performance of the memstore for a get-and-put use
case over time and graphed it, looked like time increased on a staircase with a trend to O(n).
This was a surprising result. ConcurrentSkipListMap#put is supposed to run in O(log n). His
workaround was to flush after some fixed number of gets+puts, 1000 I think. At the time we
weren't sure what was going on given the language barrier.

Sound familiar?

I don't claim to really understand what is going on, but need to get to the bottom of this.
Going to look at it in depth starting Monday.

   - Andy


View raw message