hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-13420) RegionEnvironment.offerExecutionLatency Blocks Threads under Heavy Load
Date Tue, 21 Apr 2015 02:00:00 GMT

     [ https://issues.apache.org/jira/browse/HBASE-13420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Purtell updated HBASE-13420:
-----------------------------------
    Attachment: 1M-0.98.13-SNAPSHOT.svg
                1M-0.98.12.svg

I did a quick comparison using LoadTestTool on an all-localhost HDFS+HBase cluster between
0.98.12 and an "0.98.13-SNAPSHOT" which was .12 plus this patch.  The server has 32 GB of
RAM and 12 cores, Xeon E5-1660s running at 3.70GHz. All JVMs except the regionserver were
given 1 GB heap. The regionserver ran with 8 GB. (No particular reason for that heap size,
just reusing a setting from another test.) I installed the AccessController with "hbase.security.authorization"
set to false so every region would run with a coprocessor (largely inert) so we'd exercise
this change. CMS GC. LoadTestTool arguments: -read 100:10 -write 1:1024:10 -update 20:10 -num_keys
1000000

*0.98.12*
||read|| ||update|| ||write||	||
||keys_sec||latency_ms||keys_sec||latency_ms||keys_sec||latency_ms||
|19831.5102|0|786.3265306|5.285714286|3929.142857|2.102040816|

*0.98.13-SNAPSHOT*
||read|| ||update|| ||write||	||
||keys_sec||latency_ms||keys_sec||latency_ms||keys_sec||latency_ms||
|19377.10204|0|783.755102|5.265306122|3924.530612|2.102040816|

Profiles attached. They look almost identical with a quick glance.

I will run a longer comparison tomorrow with 25M keys.

> RegionEnvironment.offerExecutionLatency Blocks Threads under Heavy Load
> -----------------------------------------------------------------------
>
>                 Key: HBASE-13420
>                 URL: https://issues.apache.org/jira/browse/HBASE-13420
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: John Leach
>            Assignee: Andrew Purtell
>             Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
>
>         Attachments: 1M-0.98.12.svg, 1M-0.98.13-SNAPSHOT.svg, HBASE-13420.patch, HBASE-13420.txt,
hbase-13420.tar.gz, offerExecutionLatency.tiff
>
>   Original Estimate: 3h
>  Remaining Estimate: 3h
>
> The ArrayBlockingQueue blocks threads for 20s during a performance run focusing on creating
numerous small scans.  
> I see a buffer size of (100)
>     private final BlockingQueue<Long> coprocessorTimeNanos = new ArrayBlockingQueue<Long>(
>         LATENCY_BUFFER_SIZE);
> and then I see a drain coming from
>          MetricsRegionWrapperImpl with 45 second executor
>          HRegionMetricsWrapperRunable
>          RegionCoprocessorHost#getCoprocessorExecutionStatistics()   
>          RegionCoprocessorHost#getExecutionLatenciesNanos()
> Am I missing something?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message