hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1061) Region Server throw NotServingRegionException when batchUpdate during compaction/spliting and then shutdown
Date Sat, 13 Dec 2008 21:20:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656330#action_12656330
] 

stack commented on HBASE-1061:
------------------------------

Please enable DEBUG and repost (See FAQ for how).   What are the stats on your hdfs?  Tell
us about your uploader?  Looks like your cluster may be suffering overloading as seemed apparent
from the other JIRA you opened.

> Region Server  throw NotServingRegionException when batchUpdate during compaction/spliting
and then shutdown
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1061
>                 URL: https://issues.apache.org/jira/browse/HBASE-1061
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.18.1
>         Environment: 1 master, 4 region server
> Hadoop 0.18.1
> When about 30GB  30,000,000 rows loaded
>            Reporter: schubert zhang
>
> 2008-12-13 21:30:58,184 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling
hlog. Number of entries: 30015
> 2008-12-13 21:30:58,595 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction
on region GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:00,162 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer
created at /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229175060158
> 2008-12-13 21:31:00,162 INFO org.apache.hadoop.hbase.regionserver.HLog: removing old
log file /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229173244235 whose highest sequence/edit
id is 145008256
> 2008-12-13 21:31:00,165 INFO org.apache.hadoop.hbase.regionserver.HLog: removing old
log file /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229173248573 whose highest sequence/edit
id is 145038271
> 2008-12-13 21:31:05,571 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
completed on region GDR,138884579261228161625885,1229166027688 in 6sec
> 2008-12-13 21:31:05,571 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split
of region GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:05,735 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:05,863 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020,
call batchUpdate([B@4df611bd, row => 138884603141228853661885, {column => gdr:start_time,
value => '...', column => gdr:imsi, value => '...', column => gdr:msisdn, value
=> '...', column => gdr:sourceip, value => '...', column => gdr:destip, value
=> '...', column => gdr:gdrtype, value => '...', column => gdr:req_num, value
=> '...', column => gdr:apn, value => '...', column => gdr:gdrtime, value =>
'...', column => gdr:result, value => '...', column => gdr:gtpver, value => '...',
column => gdr:remoteno, value => '...', column => gdr:frontno, value => '...',
column => gdr:offset, value => '...', column => gdr:url, value => '...'}, -1)
from 10.24.1.10:39000: error: org.apache.hadoop.hbase.NotServingRegionException: Region GDR,138884579261228161625885,1229166027688
closed
> org.apache.hadoop.hbase.NotServingRegionException: Region GDR,138884579261228161625885,1229166027688
closed
>         at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1836)
>         at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1901)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1432)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1415)
>         at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-13 21:31:07,594 INFO org.apache.hadoop.hbase.regionserver.HRegion: region GDR,138884579261228161625885,1229175065574/642186371
available
> 2008-12-13 21:31:07,594 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed GDR,138884579261228161625885,1229175065574
> 2008-12-13 21:31:07,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020,
call batchUpdate([B@7792b8d, row => 138884603141228853661885, {column => gdr:start_time,
value => '...', column => gdr:imsi, value => '...', column => gdr:msisdn, value
=> '...', column => gdr:sourceip, value => '...', column => gdr:destip, value
=> '...', column => gdr:gdrtype, value => '...', column => gdr:req_num, value
=> '...', column => gdr:apn, value => '...', column => gdr:gdrtime, value =>
'...', column => gdr:result, value => '...', column => gdr:gtpver, value => '...',
column => gdr:remoteno, value => '...', column => gdr:frontno, value => '...',
column => gdr:offset, value => '...', column => gdr:url, value => '...'}, -1)
from 10.24.1.10:39000: error: org.apache.hadoop.hbase.NotServingRegionException: Region GDR,138884579261228161625885,1229166027688
closed
> org.apache.hadoop.hbase.NotServingRegionException: Region GDR,138884579261228161625885,1229166027688
closed
>         at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1836)
>         at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1901)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1432)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1415)
>         at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-13 21:31:08,346 INFO org.apache.hadoop.hbase.regionserver.HRegion: region GDR,138884596811227665528885,1229175065574/1240335718
available
> The many region throw such exception. and then the region server shutdown
> 2008-12-13 22:01:46,700 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer Exception:
java.io.IOException: Unable to create new block.
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2349)
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)
> 2008-12-13 22:01:46,701 WARN org.apache.hadoop.dfs.DFSClient: Error Recovery for block
blk_-1991134958879829135_36755 bad datanode[1]
> 2008-12-13 22:01:46,701 FATAL org.apache.hadoop.hbase.regionserver.Flusher: Replay of
hlog required. Forcing server shutdown
> and at last:
> 008-12-13 22:04:53,414 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
/10.24.1.18:60020. Already tried 6 time(s).
> 2008-12-13 22:04:54,417 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
/10.24.1.18:60020. Already tried 7 time(s).
> 2008-12-13 22:04:55,420 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
/10.24.1.18:60020. Already tried 8 time(s).
> 2008-12-13 22:04:56,423 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
/10.24.1.18:60020. Already tried 9 time(s).
> 2008-12-13 22:07:04,556 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread:
Compaction failed for region GDR,138883452271227827193885,1229167157710
> java.lang.NullPointerException
>         at org.apache.hadoop.hbase.client.ServerCallable.getRegionName(ServerCallable.java:71)
>         at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
>         at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
>         at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
>         at org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:167)
>         at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:85)
> 2008-12-13 22:07:04,558 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/0:0:0:0:0:0:0:0:60020.compactor exiting
> 2008-12-13 22:07:04,558 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/0:0:0:0:0:0:0:0:60020
exiting
> 2008-12-13 22:07:04,627 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Starting
shutdown thread.
> 2008-12-13 22:07:04,627 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown
thread complete

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message