hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10643) Failure in RS when using large size bucketcache
Date Fri, 28 Feb 2014 22:05:29 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13916482#comment-13916482
] 

Nick Dimiduk commented on HBASE-10643:
--------------------------------------

The RS initializes its connection to ZooKeeper very early in the startup process. The BucketCache
isn't created until after assignments have been received and the first storefile is opened.
Thus, the pause in allocating direct memory comes after the zk session is established.

One option would be to add a configuration point that, when enabled, will ask the RS to initialize
it's cache structures before the ZK session is established. That way the pause won't disrupt
the session. Other ideas?

Note that this should not be an issue with the BucketcCache running in file mode.

> Failure in RS when using large size bucketcache
> -----------------------------------------------
>
>                 Key: HBASE-10643
>                 URL: https://issues.apache.org/jira/browse/HBASE-10643
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.98.0, 0.96.0
>            Reporter: Biju Nair
>              Labels: bucketCache, regionserver
>
> When RS is brought up with XX:MaxDirectMemorySize of 22GB or higher, RS fails after a
successful start. From the RS logs it looks like the bucketCache memory allocation is taking
more time makes the RS considered dead by ZK. One option to fix the problem would be to allocate
the bucketCache before registering with ZK. 
> 2014-02-28 18:54:42,967 WARN  [regionserver60020.compactionChecker] util.Sleeper: We
slept 33496ms instead of 10000ms, this is likely due to a long garbage collecting pause and
it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2014-02-28 18:54:42,967 WARN  [regionserver60020.periodicFlusher] util.Sleeper: We slept
33496ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's
usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2014-02-28 18:54:42,967 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause
in JVM or host machine (eg GC): pause of approximately 23988ms
> GC pool 'ParNew' had collection(s): count=1 time=24432ms
> 2014-02-28 18:54:43,006 FATAL [regionserver60020] regionserver.HRegionServer: ABORTING
region server bbg-master2.bbg-test.hdp,60020,1393628951236: org.apache.hadoop.hbase.YouAreDeadException:
Server REPORT rejected; currently processing bbg-master2.bbg-test.hdp,60020,1393628951236
as dead server
>         at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:341)
>         at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:254)



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message