hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject upsert case performance problem (doubts about ConcurrentSkipListMap)
Date Sat, 19 Mar 2011 18:10:38 GMT
I have a mapreduce task put together for experimentation which does a lot of Increments over
three tables and Puts to another. I set writeToWAL to false. My HBase includes the patch that
fixes serialization of writeToWAL for Increments. MemstoreLAB is enabled but is probably not
a factor, but still need to test to exclude it.

After starting a job up on a test cluster on EC2 with 20 mappers over 10 slaves I see initially
10-15K/ops/sec/server. This performance drops over a short time to stabilize around 1K/ops/sec/server.
So I flush the tables with the shell. Immediately after flushing the tables, performance is
back up to 10-15K/ops/sec/server. If I don't flush, performance remains low indefinitely.
If I flush only the table receiving the Gets, performance remains low. 

If I set the shell to flush in a loop every 60 seconds, performance repeatedly drops during
that interval, then recovers after flushing.

When Gary and I went to NCHC in Taiwan, we saw a guy from PhiCloud present something similar
to this regarding 0.89DR. He measured the performance of the memstore for a get-and-put use
case over time and graphed it, looked like time increased on a staircase with a trend to O(n).
This was a surprising result. ConcurrentSkipListMap#put is supposed to run in O(log n). His
workaround was to flush after some fixed number of gets+puts, 1000 I think. At the time we
weren't sure what was going on given the language barrier.

Sound familiar?

I don't claim to really understand what is going on, but need to get to the bottom of this.
Going to look at it in depth starting Monday.

   - Andy



      

Mime
View raw message