hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: PerformanceEvaluation randomWrite and custom "put" test
Date Thu, 13 Aug 2015 19:40:59 GMT
--autoFlush=true is ignored in PE. Make sure you run your own tests with
autoFlush= false in HTable.

-Vlad

On Thu, Aug 13, 2015 at 1:22 AM, Serega Sheypak <serega.sheypak@gmail.com>
wrote:

> Hi. I used PerformanceEvaluation randomWrite  for perf measurement.
>
> Here are my metrics:
>
> -- Timers
> ----------------------------------------------------------------------
>
> .putTimer
>
>              count = 3944591
>
>          mean rate = 12389.71 calls/second
>
>      1-minute rate = 8853.79 calls/second
>
>      5-minute rate = 9800.49 calls/second
>
>     15-minute rate = 7816.68 calls/second
>
>                min = 1.85 milliseconds
>
>                max = 2524.64 milliseconds
>
>               mean = 17.36 milliseconds
>
>             stddev = 135.73 milliseconds
>
>             median = 6.18 milliseconds
>
>               75% <= 10.41 milliseconds
>
>               95% <= 24.38 milliseconds
>
>               98% <= 32.82 milliseconds
>
>               99% <= 40.77 milliseconds
>
>             99.9% <= 2524.64 milliseconds
>
>
> And here are metrics from randomWrite:
>
>
> ## sudo -u hdfs hbase org.apache.hadoop.hbase.PerformanceEvaluation
> --nomapred --flushCommits=true  --autoFlush=true  --oneCon=true --latency
> randomWrite 10
>
> 15/08/13 00:30:49 INFO hbase.PerformanceEvaluation: Finished TestClient-4
> in 461794ms over 1048576 rows
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation:
> 9437184/10485700/10485760, latency mean=440.17, min=1.00, max=5525520.00,
> stdDev=20817.84, 95th=2.00, 99th=3.00
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest latency
> log (microseconds), on 1048576 measures
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Min
>  = 1.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Avg
>  = 440.1488666534424
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest StdDev
>   = 20817.257168459604
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 50th
>   = 2.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 75th
>   = 2.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 95th
>   = 2.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99th
>   = 3.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99.9th
>   = 3884.876000001328
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99.99th
>  = 922522.8667998798
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> 99.999th = 2311970.1423621834
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Max
>  = 5525520.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> valueSize after 0 measures
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Min
>  = 0.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Avg
>  = 0.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest StdDev
>   = 0.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 50th
>   = 0.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 75th
>   = 0.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 95th
>   = 0.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99th
>   = 0.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99.9th
>   = 0.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99.99th
>  = 0.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest
> 99.999th = 0.0
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Max
>  = 0.0
>
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: [RandomWriteTest]
> Summary of timings (ms): [458553, 459646, 461004, 458014, 461794, 455945,
> 458489, 457333, 461017, 463250]
>
> 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: [RandomWriteTest] Min:
> 455945ms Max: 463250ms Avg: 459504ms
>
> So my 99th percentile is 40MS, randowWrite shows 4ms
>
> What could be the reason for 10 times diff?
> My code does random put (Random string as a key), the size of put is
> hundreds of bytes.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message