hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "schubert zhang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1060) RegionServer lease expire due to LogRoller take long time and the RegionServer cannot recovery
Date Thu, 18 Dec 2008 10:36:46 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657728#action_12657728
] 

schubert zhang commented on HBASE-1060:
---------------------------------------

Sometimes, when I do mapreduce job (eg. sorting) on a large dataset (eg. 30mil records), the
lease may exprie too. 

Now, I change the size of Region smaller (from 256MB to 64MB) and change the number of map
task smaller (from 8 per node to 3, there are 4codes pre node). The failure not happen.

> RegionServer lease expire  due to  LogRoller take long time and the RegionServer  cannot
recovery
> -------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1060
>                 URL: https://issues.apache.org/jira/browse/HBASE-1060
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.18.1
>         Environment: I have one master and 4 region servers.
> The test use batchUpdate() to insert many record to achieve big data sets.
>            Reporter: schubert zhang
>
> Usually, after tens GB of data, following exception occurs, it seems LogRoller take long
time and the RegionServer cannot report to master, and the lease expire. But it cannot recovery.
> ---------
> 2008-12-13 11:13:39,347 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling
hlog. Number of entries: 30015
> 2008-12-13 11:13:39,354 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer
created at /hbase/log_10.24.1.14_1229053900800_60020/hlog.dat.1229138019352
> 2008-12-13 11:13:47,991 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling
hlog. Number of entries: 30015
> 2008-12-13 11:13:49,138 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer
created at /hbase/log_10.24.1.14_1229053900800_60020/hlog.dat.1229138029136
> 2008-12-13 11:15:49,867 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 120318ms,
ten times longer than scheduled: 3000
> 2008-12-13 11:15:50,175 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: unable
to report to master for 120343 milliseconds - retrying
> 2008-12-13 11:15:50,271 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP
> 2008-12-13 11:15:50,419 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020,
call get([B@4b2a82b3, [B@53eadc52, [B@6026b688, -1, -1) from 10.24.1.18:58581: error: org.apache.hadoop.hbase.NotServingRegionException:
TableDes,,1228799531753
> org.apache.hadoop.hbase.NotServingRegionException: TableDes,,1228799531753
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1859)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1321)
>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-13 11:15:50,434 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020,
call get([B@26ee4c87, [B@66e24708, [B@11a592fd, -1, -1) from 10.24.1.12:46262: error: org.apache.hadoop.hbase.NotServingRegionException:
TableDes,,1228799531753
> org.apache.hadoop.hbase.NotServingRegionException: TableDes,,1228799531753
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1859)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1321)
>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-13 11:15:50,431 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020,
call get([B@519182c2, [B@281fb864, [B@3a82c1d0, -1, -1) from 10.24.1.16:34950: error: org.apache.hadoop.hbase.NotServingRegionException:
TableDes,,1228799531753
> org.apache.hadoop.hbase.NotServingRegionException: TableDes,,122879953175
> .................many exceptions.......................

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message