hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1061) Region Server throw NotServingRegionException when batchUpdate during compaction/spliting and then shutdown
Date Sun, 14 Dec 2008 12:44:48 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656385#action_12656385
] 

Andrew Purtell commented on HBASE-1061:
---------------------------------------

I've seen a failure case recently where delayed splits cause so much data to build that when
finally the compact/split happens, DFS is overwhelmed, and file errors result. See https://issues.apache.org/jira/browse/HBASE-1050?focusedCommentId=12656384#action_12656384
. Maybe this is another instance of that?

> Region Server  throw NotServingRegionException when batchUpdate during compaction/spliting
and then shutdown
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1061
>                 URL: https://issues.apache.org/jira/browse/HBASE-1061
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.18.1
>         Environment: 1 master, 4 region server
> Hadoop 0.18.1
> When about 30GB  30,000,000 rows loaded
>            Reporter: schubert zhang
>         Attachments: hbase-stephen-regionserver-nd4-rack0-cloud.log
>
>
> 2008-12-13 21:30:58,184 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling
hlog. Number of entries: 30015
> 2008-12-13 21:30:58,595 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction
on region GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:00,162 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer
created at /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229175060158
> 2008-12-13 21:31:00,162 INFO org.apache.hadoop.hbase.regionserver.HLog: removing old
log file /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229173244235 whose highest sequence/edit
id is 145008256
> 2008-12-13 21:31:00,165 INFO org.apache.hadoop.hbase.regionserver.HLog: removing old
log file /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229173248573 whose highest sequence/edit
id is 145038271
> 2008-12-13 21:31:05,571 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
completed on region GDR,138884579261228161625885,1229166027688 in 6sec
> 2008-12-13 21:31:05,571 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split
of region GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:05,735 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:05,863 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020,
call batchUpdate([B@4df611bd, row => 138884603141228853661885, {column => gdr:start_time,
value => '...', column => gdr:imsi, value => '...', column => gdr:msisdn, value
=> '...', column => gdr:sourceip, value => '...', column => gdr:destip, value
=> '...', column => gdr:gdrtype, value => '...', column => gdr:req_num, value
=> '...', column => gdr:apn, value => '...', column => gdr:gdrtime, value =>
'...', column => gdr:result, value => '...', column => gdr:gtpver, value => '...',
column => gdr:remoteno, value => '...', column => gdr:frontno, value => '...',
column => gdr:offset, value => '...', column => gdr:url, value => '...'}, -1)
from 10.24.1.10:39000: error: org.apache.hadoop.hbase.NotServingRegionException: Region GDR,138884579261228161625885,1229166027688
closed
> org.apache.hadoop.hbase.NotServingRegionException: Region GDR,138884579261228161625885,1229166027688
closed
>         at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1836)
>         at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1901)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1432)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1415)
>         at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-13 21:31:07,594 INFO org.apache.hadoop.hbase.regionserver.HRegion: region GDR,138884579261228161625885,1229175065574/642186371
available
> 2008-12-13 21:31:07,594 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed GDR,138884579261228161625885,1229175065574
> 2008-12-13 21:31:07,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020,
call batchUpdate([B@7792b8d, row => 138884603141228853661885, {column => gdr:start_time,
value => '...', column => gdr:imsi, value => '...', column => gdr:msisdn, value
=> '...', column => gdr:sourceip, value => '...', column => gdr:destip, value
=> '...', column => gdr:gdrtype, value => '...', column => gdr:req_num, value
=> '...', column => gdr:apn, value => '...', column => gdr:gdrtime, value =>
'...', column => gdr:result, value => '...', column => gdr:gtpver, value => '...',
column => gdr:remoteno, value => '...', column => gdr:frontno, value => '...',
column => gdr:offset, value => '...', column => gdr:url, value => '...'}, -1)
from 10.24.1.10:39000: error: org.apache.hadoop.hbase.NotServingRegionException: Region GDR,138884579261228161625885,1229166027688
closed
> org.apache.hadoop.hbase.NotServingRegionException: Region GDR,138884579261228161625885,1229166027688
closed
>         at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1836)
>         at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1901)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1432)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1415)
>         at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-13 21:31:08,346 INFO org.apache.hadoop.hbase.regionserver.HRegion: region GDR,138884596811227665528885,1229175065574/1240335718
available
> The many region throw such exception. and then the region server shutdown
> 2008-12-13 22:01:46,700 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer Exception:
java.io.IOException: Unable to create new block.
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2349)
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)
> 2008-12-13 22:01:46,701 WARN org.apache.hadoop.dfs.DFSClient: Error Recovery for block
blk_-1991134958879829135_36755 bad datanode[1]
> 2008-12-13 22:01:46,701 FATAL org.apache.hadoop.hbase.regionserver.Flusher: Replay of
hlog required. Forcing server shutdown
> and at last:
> 008-12-13 22:04:53,414 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
/10.24.1.18:60020. Already tried 6 time(s).
> 2008-12-13 22:04:54,417 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
/10.24.1.18:60020. Already tried 7 time(s).
> 2008-12-13 22:04:55,420 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
/10.24.1.18:60020. Already tried 8 time(s).
> 2008-12-13 22:04:56,423 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
/10.24.1.18:60020. Already tried 9 time(s).
> 2008-12-13 22:07:04,556 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread:
Compaction failed for region GDR,138883452271227827193885,1229167157710
> java.lang.NullPointerException
>         at org.apache.hadoop.hbase.client.ServerCallable.getRegionName(ServerCallable.java:71)
>         at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
>         at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
>         at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
>         at org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:167)
>         at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:85)
> 2008-12-13 22:07:04,558 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/0:0:0:0:0:0:0:0:60020.compactor exiting
> 2008-12-13 22:07:04,558 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/0:0:0:0:0:0:0:0:60020
exiting
> 2008-12-13 22:07:04,627 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Starting
shutdown thread.
> 2008-12-13 22:07:04,627 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown
thread complete

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message