hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: CSLM performance Was: SILT - nice keyvalue store paper
Date Tue, 25 Oct 2011 14:36:55 GMT
Also please make note of what JVM version and report it with the results.


Best regards,


   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)


----- Original Message -----
> From: Li Pi <li@idle.li>
> To: dev@hbase.apache.org
> Cc: 
> Sent: Monday, October 24, 2011 11:27 PM
> Subject: Re: CSLM performance Was: SILT - nice keyvalue store paper
> 
> You might also want to run the code a few times, and only take results
> from the latter half. Let the JVM warm up and JIT things for a fair
> comparison.
> 
> On Mon, Oct 24, 2011 at 11:14 PM, Akash Ashok <thehellmaker@gmail.com> 
> wrote:
>>  On Mon, Oct 24, 2011 at 10:49 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>> 
>>>  Akash:
>>>  Take a look at the following two possible replacements for CSLM:
>>>  https://github.com/mspiegel/lockfreeskiptree
>>>  https://github.com/nbronson/snaptree
>> 
>>  Thanks Ted :) :) :). Was pretty much on the lookout for other structures. I
>>  tried
>> 
> http://g.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/SyncSortedMap.html
>>  but didn't perform that great. Will look into these replacements.
>> 
>> 
>>> 
>>>  About your test, I got advice from other expert:
>>> 
>>>  - the JVM warm-up penalty is being accrued by the CSLM run
>>>  - Use thread.yield() to end a request, as otherwise the active thread 
> may
>>>  be
>>>  able to run through its time slice without incurring much lock 
> contention.
>>> 
>> 
>>  Hmm. I was just wondering. Since each thread 10000+ inserts and deletes are
>>  being run
>>  they have to context switch quite a few times during its lifecycle right ?
>>  Wouldn't it be enough if I increase the number of iterations by a 
> factor if
>>  say 100 per thread ?
>> 
>> 
>> 
>>>  You can publish your code and results under HBASE-3992.
>>> 
>>>  Thanks
>>> 
>>>  On Sun, Oct 23, 2011 at 5:05 PM, Jonathan Gray <jgray@fb.com> 
> wrote:
>>> 
>>>  > Oh, and when running these experiments, you should look at the 
> impact at
>>>  > which order they are run in, whether you run them multiple times 
> per JVM
>>>  > instance, etc.  Basically, you need to be cognizant of the HotSpot
>>>  > optimizations the JVM is doing at runtime.
>>>  >
>>>  > > -----Original Message-----
>>>  > > From: Jonathan Gray [mailto:jgray@fb.com]
>>>  > > Sent: Sunday, October 23, 2011 4:20 PM
>>>  > > To: dev@hbase.apache.org
>>>  > > Subject: RE: SILT - nice keyvalue store paper
>>>  > >
>>>  > > Very nice experiment, Akash.  Keep getting your hands dirty 
> and
>>>  digging!
>>>  >  :)
>>>  > >
>>>  > > I think your results might change if you bump the test up to 
> 1000
>>>  threads
>>>  > or
>>>  > > so.  100 threads can still perform okay when there's a 
> global lock but
>>>  > the
>>>  > > contention at 1000 threads will kill you and that's when 
> CSLM should do
>>>  > much
>>>  > > better.  (1000 handler threads is approx. what I run with on 
> RS in
>>>  prod).
>>>  > > Though I am a bit surprised that at 100 threads the TreeMap 
> was
>>>  > significantly
>>>  > > faster.  Your inconsistent results are a bit odd, you might 
> try an
>>>  order
>>>  > of
>>>  > > magnitude more operations per thread.  You might also gather 
> some
>>>  > > statistics about tree size and per operation latency.
>>>  > >
>>>  > > I've done some isolated CSLM benchmarks in the past and 
> have never been
>>>  > > able to reproduce any of the slowness people suggest.  I 
> recall trying
>>>  > some
>>>  > > impractically large MemStores and everything still being 
> quite fast.
>>>  > >
>>>  > > Over in Cassandra, I believe they have a two-level CSLM with 
> the first
>>>  > map
>>>  > > key being the row and then the columns for each row in their 
> own CSLM.
>>>  > > I've been told this is somewhat of a pain point for them. 
>  And keep in
>>>  > mind
>>>  > > they have one shard/region per node and we generally have 
> several
>>>  smaller
>>>  > > MemStores on each node (tens to thousands).  Not sure we 
> would want to
>>>  > > try that.  There could be some interesting optimizations if 
> you had
>>>  very
>>>  > > specific issues, like if you had a ton of reads to MemStore 
> and not
>>>  many
>>>  > > writes you could keep some kind of mirrored hashmap.
>>>  > >
>>>  > > And for writes, the WAL is definitely the latency bottleneck. 
>  But if
>>>  you
>>>  > are
>>>  > > doing lots of small operations, so your WALEdits are not 
> large, and
>>>  with
>>>  > some
>>>  > > of the HLog batching features going in to trunk, you end up 
> with
>>>  hundreds
>>>  > of
>>>  > > requests per HLog sync.  And although the syncs are higher 
> latency,
>>>  with
>>>  > > batching you end up getting high throughput.  And the 
> bottleneck
>>>  shifts.
>>>  > >
>>>  > > Each sync will take approx. 1-5ms, so let's say 250 
> requests per HLog
>>>  > sync
>>>  > > batch, 4ms per sync, so 62.5k req/sec.  (62.5k * 100 
> bytes/req =
>>>  > 600K/sec,
>>>  > > very reasonable).  If you're mixing in reads as well (or 
> if you're
>>>  doing
>>>  > > increments which do a read and write), then this adds to the 
> CPU usage
>>>  > and
>>>  > > contention without adding to HLog throughput.
>>>  > >
>>>  > > All of a sudden the bottleneck becomes CPU/contention and not 
> HLog
>>>  > > latency or throughput.  Highly concurrent increments/counters 
> with a
>>>  > largely
>>>  > > in-memory dataset can easily be CPU bottlenecked.
>>>  > >
>>>  > > For one specific application Dhruba and I worked on, we made 
> some good
>>>  > > improvements in CPU efficiency by reducing the number of 
> operations and
>>>  > > increasing efficiency on the CSLM.  Doing things like always 
> taking a
>>>  > tailMap
>>>  > > and working from that instead of starting at the root node, 
> using an
>>>  > iterator()
>>>  > > and taking advantage of the available remove() semantics, or 
> simply
>>>  just
>>>  > > mutating things that are normally immutable :)  Unfortunately 
> many of
>>>  > these
>>>  > > optimizations were semi-horrid hacks and introduced things 
> like
>>>  > > ModifiableKeyValues, so they all haven't made their way 
> to apache.
>>>  > >
>>>  > > In the end, after our optimizations, the real world workload 
> Dhruba and
>>>  I
>>>  > > were working with was not all in-memory so the bottleneck in 
> production
>>>  > > became the random reads (so increasing the block cache hit 
> ratio is the
>>>  > > focus) rather than CPU contention or HLog throughput.
>>>  > >
>>>  > > JG
>>>  > >
>>>  > > From: Akash Ashok [mailto:thehellmaker@gmail.com]
>>>  > > Sent: Sunday, October 23, 2011 2:57 AM
>>>  > > To: dev@hbase.apache.org
>>>  > > Subject: Re: SILT - nice keyvalue store paper
>>>  > >
>>>  > > I was running some similar tests and came across a surprising 
> finding.
>>>  I
>>>  > > compared reads and write on ConcurrentSkipListMap ( which the 
> memstore
>>>  > > uses) and synchronized TreeMap ( Which was literally treemap
>>>  > > synchronized). Executed concurrent reads, writes and deletes 
> on both of
>>>  > > them.
>>>  > > Surprisingly synchronized treeMap performed better, though 
> just
>>>  slightly
>>>  > > better, than ConcurrentSkipListMap which KeyValueSkipListSet 
> uses.
>>>  > >
>>>  > > Here are the output of a few runs
>>>  > >
>>>  > > Sometimes the difference was considerable Using HBaseMap it 
> took
>>>  > > 20438ms Using TreeMap it took 11613ms Time Difference:8825ms
>>>  > >
>>>  > > And sometimes the difference was negligible Using HBaseMap it 
> took
>>>  > > 13370ms Using TreeMap it took 9482ms Time Difference:3888ms
>>>  > >
>>>  > > I've attaching the test  java file which I wrote to test 
> it.
>>>  > > This might be a very minor differece but still surprising 
> considering
>>>  the
>>>  > fact
>>>  > > that ConcurrentSkipListMap uses fancy 2 level indexes which 
> they say
>>>  > > improves the deletion performance.
>>>  > >
>>>  > > And here are the details about the test run.
>>>  > > 100 Threads each fetching 1,000,000 records
>>>  > > 100 threads each adding 1,000,000 records.
>>>  > > 100 threads each deletin 1,000,000 records ( Reads, Writes 
> and deletes
>>>  > > simultaneously )
>>>  > >
>>>  > > Cheers,
>>>  > > Akash A
>>>  > > On Sun, Oct 23, 2011 at 3:25 AM, Stack
>>>  > > <stack@duboce.net<mailto:stack@duboce.net>> 
> wrote:
>>>  > > On Sat, Oct 22, 2011 at 2:41 PM, N Keywal
>>>  > > <nkeywal@gmail.com<mailto:nkeywal@gmail.com>> 
> wrote:
>>>  > > > I would think that the bottleneck for insert is the wal 
> part?
>>>  > > > It would be possible to do a kind of memory list 
> preparation during
>>>  > > > the wal insertion, and if the wal insertion is 
> confirmed, do the
>>>  > > > insertion in the memory list. But it's strange to 
> have the insertion
>>>  in
>>>  > > memory important vs.
>>>  > > > the insertion on disk...
>>>  > > >
>>>  > > Yes, WAL is the long pole writing.  But MemStore has issues 
> too; Dhruba
>>>  > says
>>>  > > cpu above.  Reading and writing it is also 'slow'.
>>>  > > St.Ack
>>>  >
>>>  >
>>> 
>> 
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message