hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: CSLM performance Was: SILT - nice keyvalue store paper
Date Tue, 25 Oct 2011 16:07:38 GMT
http://code.google.com/p/caliper/ might be useful.

On Tue, Oct 25, 2011 at 7:36 AM, Andrew Purtell <apurtell@apache.org> wrote:

> Also please make note of what JVM version and report it with the results.
>
>
> Best regards,
>
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>
> ----- Original Message -----
> > From: Li Pi <li@idle.li>
> > To: dev@hbase.apache.org
> > Cc:
> > Sent: Monday, October 24, 2011 11:27 PM
> > Subject: Re: CSLM performance Was: SILT - nice keyvalue store paper
> >
> > You might also want to run the code a few times, and only take results
> > from the latter half. Let the JVM warm up and JIT things for a fair
> > comparison.
> >
> > On Mon, Oct 24, 2011 at 11:14 PM, Akash Ashok <thehellmaker@gmail.com>
> > wrote:
> >>  On Mon, Oct 24, 2011 at 10:49 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>
> >>>  Akash:
> >>>  Take a look at the following two possible replacements for CSLM:
> >>>  https://github.com/mspiegel/lockfreeskiptree
> >>>  https://github.com/nbronson/snaptree
> >>
> >>  Thanks Ted :) :) :). Was pretty much on the lookout for other
> structures. I
> >>  tried
> >>
> >
> http://g.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/SyncSortedMap.html
> >>  but didn't perform that great. Will look into these replacements.
> >>
> >>
> >>>
> >>>  About your test, I got advice from other expert:
> >>>
> >>>  - the JVM warm-up penalty is being accrued by the CSLM run
> >>>  - Use thread.yield() to end a request, as otherwise the active thread
> > may
> >>>  be
> >>>  able to run through its time slice without incurring much lock
> > contention.
> >>>
> >>
> >>  Hmm. I was just wondering. Since each thread 10000+ inserts and deletes
> are
> >>  being run
> >>  they have to context switch quite a few times during its lifecycle
> right ?
> >>  Wouldn't it be enough if I increase the number of iterations by a
> > factor if
> >>  say 100 per thread ?
> >>
> >>
> >>
> >>>  You can publish your code and results under HBASE-3992.
> >>>
> >>>  Thanks
> >>>
> >>>  On Sun, Oct 23, 2011 at 5:05 PM, Jonathan Gray <jgray@fb.com>
> > wrote:
> >>>
> >>>  > Oh, and when running these experiments, you should look at the
> > impact at
> >>>  > which order they are run in, whether you run them multiple times
> > per JVM
> >>>  > instance, etc.  Basically, you need to be cognizant of the HotSpot
> >>>  > optimizations the JVM is doing at runtime.
> >>>  >
> >>>  > > -----Original Message-----
> >>>  > > From: Jonathan Gray [mailto:jgray@fb.com]
> >>>  > > Sent: Sunday, October 23, 2011 4:20 PM
> >>>  > > To: dev@hbase.apache.org
> >>>  > > Subject: RE: SILT - nice keyvalue store paper
> >>>  > >
> >>>  > > Very nice experiment, Akash.  Keep getting your hands dirty
> > and
> >>>  digging!
> >>>  >  :)
> >>>  > >
> >>>  > > I think your results might change if you bump the test up to
> > 1000
> >>>  threads
> >>>  > or
> >>>  > > so.  100 threads can still perform okay when there's a
> > global lock but
> >>>  > the
> >>>  > > contention at 1000 threads will kill you and that's when
> > CSLM should do
> >>>  > much
> >>>  > > better.  (1000 handler threads is approx. what I run with on
> > RS in
> >>>  prod).
> >>>  > > Though I am a bit surprised that at 100 threads the TreeMap
> > was
> >>>  > significantly
> >>>  > > faster.  Your inconsistent results are a bit odd, you might
> > try an
> >>>  order
> >>>  > of
> >>>  > > magnitude more operations per thread.  You might also gather
> > some
> >>>  > > statistics about tree size and per operation latency.
> >>>  > >
> >>>  > > I've done some isolated CSLM benchmarks in the past and
> > have never been
> >>>  > > able to reproduce any of the slowness people suggest.  I
> > recall trying
> >>>  > some
> >>>  > > impractically large MemStores and everything still being
> > quite fast.
> >>>  > >
> >>>  > > Over in Cassandra, I believe they have a two-level CSLM with
> > the first
> >>>  > map
> >>>  > > key being the row and then the columns for each row in their
> > own CSLM.
> >>>  > > I've been told this is somewhat of a pain point for them.
> >  And keep in
> >>>  > mind
> >>>  > > they have one shard/region per node and we generally have
> > several
> >>>  smaller
> >>>  > > MemStores on each node (tens to thousands).  Not sure we
> > would want to
> >>>  > > try that.  There could be some interesting optimizations if
> > you had
> >>>  very
> >>>  > > specific issues, like if you had a ton of reads to MemStore
> > and not
> >>>  many
> >>>  > > writes you could keep some kind of mirrored hashmap.
> >>>  > >
> >>>  > > And for writes, the WAL is definitely the latency bottleneck.
> >  But if
> >>>  you
> >>>  > are
> >>>  > > doing lots of small operations, so your WALEdits are not
> > large, and
> >>>  with
> >>>  > some
> >>>  > > of the HLog batching features going in to trunk, you end up
> > with
> >>>  hundreds
> >>>  > of
> >>>  > > requests per HLog sync.  And although the syncs are higher
> > latency,
> >>>  with
> >>>  > > batching you end up getting high throughput.  And the
> > bottleneck
> >>>  shifts.
> >>>  > >
> >>>  > > Each sync will take approx. 1-5ms, so let's say 250
> > requests per HLog
> >>>  > sync
> >>>  > > batch, 4ms per sync, so 62.5k req/sec.  (62.5k * 100
> > bytes/req =
> >>>  > 600K/sec,
> >>>  > > very reasonable).  If you're mixing in reads as well (or
> > if you're
> >>>  doing
> >>>  > > increments which do a read and write), then this adds to the
> > CPU usage
> >>>  > and
> >>>  > > contention without adding to HLog throughput.
> >>>  > >
> >>>  > > All of a sudden the bottleneck becomes CPU/contention and not
> > HLog
> >>>  > > latency or throughput.  Highly concurrent increments/counters
> > with a
> >>>  > largely
> >>>  > > in-memory dataset can easily be CPU bottlenecked.
> >>>  > >
> >>>  > > For one specific application Dhruba and I worked on, we made
> > some good
> >>>  > > improvements in CPU efficiency by reducing the number of
> > operations and
> >>>  > > increasing efficiency on the CSLM.  Doing things like always
> > taking a
> >>>  > tailMap
> >>>  > > and working from that instead of starting at the root node,
> > using an
> >>>  > iterator()
> >>>  > > and taking advantage of the available remove() semantics, or
> > simply
> >>>  just
> >>>  > > mutating things that are normally immutable :)  Unfortunately
> > many of
> >>>  > these
> >>>  > > optimizations were semi-horrid hacks and introduced things
> > like
> >>>  > > ModifiableKeyValues, so they all haven't made their way
> > to apache.
> >>>  > >
> >>>  > > In the end, after our optimizations, the real world workload
> > Dhruba and
> >>>  I
> >>>  > > were working with was not all in-memory so the bottleneck in
> > production
> >>>  > > became the random reads (so increasing the block cache hit
> > ratio is the
> >>>  > > focus) rather than CPU contention or HLog throughput.
> >>>  > >
> >>>  > > JG
> >>>  > >
> >>>  > > From: Akash Ashok [mailto:thehellmaker@gmail.com]
> >>>  > > Sent: Sunday, October 23, 2011 2:57 AM
> >>>  > > To: dev@hbase.apache.org
> >>>  > > Subject: Re: SILT - nice keyvalue store paper
> >>>  > >
> >>>  > > I was running some similar tests and came across a surprising
> > finding.
> >>>  I
> >>>  > > compared reads and write on ConcurrentSkipListMap ( which the
> > memstore
> >>>  > > uses) and synchronized TreeMap ( Which was literally treemap
> >>>  > > synchronized). Executed concurrent reads, writes and deletes
> > on both of
> >>>  > > them.
> >>>  > > Surprisingly synchronized treeMap performed better, though
> > just
> >>>  slightly
> >>>  > > better, than ConcurrentSkipListMap which KeyValueSkipListSet
> > uses.
> >>>  > >
> >>>  > > Here are the output of a few runs
> >>>  > >
> >>>  > > Sometimes the difference was considerable Using HBaseMap it
> > took
> >>>  > > 20438ms Using TreeMap it took 11613ms Time Difference:8825ms
> >>>  > >
> >>>  > > And sometimes the difference was negligible Using HBaseMap it
> > took
> >>>  > > 13370ms Using TreeMap it took 9482ms Time Difference:3888ms
> >>>  > >
> >>>  > > I've attaching the test  java file which I wrote to test
> > it.
> >>>  > > This might be a very minor differece but still surprising
> > considering
> >>>  the
> >>>  > fact
> >>>  > > that ConcurrentSkipListMap uses fancy 2 level indexes which
> > they say
> >>>  > > improves the deletion performance.
> >>>  > >
> >>>  > > And here are the details about the test run.
> >>>  > > 100 Threads each fetching 1,000,000 records
> >>>  > > 100 threads each adding 1,000,000 records.
> >>>  > > 100 threads each deletin 1,000,000 records ( Reads, Writes
> > and deletes
> >>>  > > simultaneously )
> >>>  > >
> >>>  > > Cheers,
> >>>  > > Akash A
> >>>  > > On Sun, Oct 23, 2011 at 3:25 AM, Stack
> >>>  > > <stack@duboce.net<mailto:stack@duboce.net>>
> > wrote:
> >>>  > > On Sat, Oct 22, 2011 at 2:41 PM, N Keywal
> >>>  > > <nkeywal@gmail.com<mailto:nkeywal@gmail.com>>
> > wrote:
> >>>  > > > I would think that the bottleneck for insert is the wal
> > part?
> >>>  > > > It would be possible to do a kind of memory list
> > preparation during
> >>>  > > > the wal insertion, and if the wal insertion is
> > confirmed, do the
> >>>  > > > insertion in the memory list. But it's strange to
> > have the insertion
> >>>  in
> >>>  > > memory important vs.
> >>>  > > > the insertion on disk...
> >>>  > > >
> >>>  > > Yes, WAL is the long pole writing.  But MemStore has issues
> > too; Dhruba
> >>>  > says
> >>>  > > cpu above.  Reading and writing it is also 'slow'.
> >>>  > > St.Ack
> >>>  >
> >>>  >
> >>>
> >>
> >

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message