hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Biju Nair (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10643) Failure in RS when using large size bucketcache
Date Fri, 07 Mar 2014 22:19:43 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924403#comment-13924403
] 

Biju Nair commented on HBASE-10643:
-----------------------------------

Also the failures were intermittent and not consistent i.e one or two of the RS fails and
others were successful. There may be EC2 factors in play. But it would be good to allocate
the memory before registering with ZK. I am assuming this will also help in handling issues
like user configuring more memory than physically available.

> Failure in RS when using large size bucketcache
> -----------------------------------------------
>
>                 Key: HBASE-10643
>                 URL: https://issues.apache.org/jira/browse/HBASE-10643
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.98.0, 0.96.0
>            Reporter: Biju Nair
>              Labels: bucketCache, regionserver
>
> When RS is brought up with XX:MaxDirectMemorySize of 22GB or higher, RS fails after a
successful start. From the RS logs it looks like the bucketCache memory allocation is taking
more time makes the RS considered dead by ZK. One option to fix the problem would be to allocate
the bucketCache before registering with ZK. 
> 2014-02-28 18:54:42,967 WARN  [regionserver60020.compactionChecker] util.Sleeper: We
slept 33496ms instead of 10000ms, this is likely due to a long garbage collecting pause and
it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2014-02-28 18:54:42,967 WARN  [regionserver60020.periodicFlusher] util.Sleeper: We slept
33496ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's
usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2014-02-28 18:54:42,967 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause
in JVM or host machine (eg GC): pause of approximately 23988ms
> GC pool 'ParNew' had collection(s): count=1 time=24432ms
> 2014-02-28 18:54:43,006 FATAL [regionserver60020] regionserver.HRegionServer: ABORTING
region server bbg-master2.bbg-test.hdp,60020,1393628951236: org.apache.hadoop.hbase.YouAreDeadException:
Server REPORT rejected; currently processing bbg-master2.bbg-test.hdp,60020,1393628951236
as dead server
>         at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:341)
>         at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:254)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message