hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Serega Sheypak <serega.shey...@gmail.com>
Subject Re: PerformanceEvaluation randomWrite and custom "put" test
Date Thu, 13 Aug 2015 19:52:20 GMT
Haha. It explains results. Performance does batch instead of single put
with full round trip. I can't do it right now. I create htable per request
using single hconnection for the whole app.

четверг, 13 августа 2015 г. пользователь Vladimir Rodionov написал:

> --autoFlush=true is ignored in PE. Make sure you run your own tests with
> autoFlush= false in HTable.
>
> -Vlad
>
> On Thu, Aug 13, 2015 at 1:22 AM, Serega Sheypak <serega.sheypak@gmail.com
> <javascript:;>>
> wrote:
>
> > Hi. I used PerformanceEvaluation randomWrite  for perf measurement.
> >
> > Here are my metrics:
> >
> > -- Timers
> > ----------------------------------------------------------------------
> >
> > .putTimer
> >
> >              count = 3944591
> >
> >          mean rate = 12389.71 calls/second
> >
> >      1-minute rate = 8853.79 calls/second
> >
> >      5-minute rate = 9800.49 calls/second
> >
> >     15-minute rate = 7816.68 calls/second
> >
> >                min = 1.85 milliseconds
> >
> >                max = 2524.64 milliseconds
> >
> >               mean = 17.36 milliseconds
> >
> >             stddev = 135.73 milliseconds
> >
> >             median = 6.18 milliseconds
> >
> >               75% <= 10.41 milliseconds
> >
> >               95% <= 24.38 milliseconds
> >
> >               98% <= 32.82 milliseconds
> >
> >               99% <= 40.77 milliseconds
> >
> >             99.9% <= 2524.64 milliseconds
> >
> >
> > And here are metrics from randomWrite:
> >
> >
> > ## sudo -u hdfs hbase org.apache.hadoop.hbase.PerformanceEvaluation
> > --nomapred --flushCommits=true  --autoFlush=true  --oneCon=true --latency
> > randomWrite 10
> >
> > 15/08/13 00:30:49 INFO hbase.PerformanceEvaluation: Finished TestClient-4
> > in 461794ms over 1048576 rows
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation:
> > 9437184/10485700/10485760, latency mean=440.17, min=1.00, max=5525520.00,
> > stdDev=20817.84, 95th=2.00, 99th=3.00
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> latency
> > log (microseconds), on 1048576 measures
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Min
> >  = 1.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Avg
> >  = 440.1488666534424
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> StdDev
> >   = 20817.257168459604
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 50th
> >   = 2.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 75th
> >   = 2.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 95th
> >   = 2.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99th
> >   = 3.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> 99.9th
> >   = 3884.876000001328
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> 99.99th
> >  = 922522.8667998798
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> > 99.999th = 2311970.1423621834
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Max
> >  = 5525520.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> > valueSize after 0 measures
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Min
> >  = 0.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Avg
> >  = 0.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> StdDev
> >   = 0.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 50th
> >   = 0.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 75th
> >   = 0.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 95th
> >   = 0.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99th
> >   = 0.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> 99.9th
> >   = 0.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> 99.99th
> >  = 0.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> > 99.999th = 0.0
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Max
> >  = 0.0
> >
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: [RandomWriteTest]
> > Summary of timings (ms): [458553, 459646, 461004, 458014, 461794, 455945,
> > 458489, 457333, 461017, 463250]
> >
> > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: [RandomWriteTest]
> Min:
> > 455945ms Max: 463250ms Avg: 459504ms
> >
> > So my 99th percentile is 40MS, randowWrite shows 4ms
> >
> > What could be the reason for 10 times diff?
> > My code does random put (Random string as a key), the size of put is
> > hundreds of bytes.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message