hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From charan kumar <charan.ku...@gmail.com>
Subject Re: Region Server shutdown (Replay HLOg required)
Date Thu, 13 Jan 2011 06:54:22 GMT
Hi Stak,

We user hadoop-0.20.2 . I applied the patch HDFS-630 this morning, didnt
help. This is what I see before the line.. The file names might be little
different , I am collecting from different failures.

2011-01-12 07:01:32,336 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read:
java.io.IOException: Cannot open filename
/hbase/webtable/689554504/qa/8995572027002
725486
        at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1497)
        at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1824)
        at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1638)
        at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1767)
        at java.io.DataInputStream.read(DataInputStream.java:132)
        at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:105)
        at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:88)
        at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:81)
        at
org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:120)
        at
org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:66)
        at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)

Also I compiled the following log entries from Name Node , when the region
server shutsdown because of this exception. Hope this helps too.

11/01/12 19:45:43 INFO hdfs.StateChange: BLOCK* NameSystem.allocateBlock:
/hbase/webtable/compaction.dir/1326275218/2628290592728420519.
blk_-7100767084720175376_361638

11/01/12 19:45:48 INFO hdfs.StateChange: BLOCK* NameSystem.addStoredBlock:
addStoredBlock request received for blk_-7100767084720175376_361638 on
10.76.99.68:50010 size 23874613 But it does not belong to any file.
11/01/12 19:45:48 INFO hdfs.StateChange: BLOCK* NameSystem.addStoredBlock:
addStoredBlock request received for blk_-7100767084720175376_361638 on
10.76.99.57:50010 size 23874613 But it does not belong to any file.
11/01/12 19:45:48 INFO hdfs.StateChange: BLOCK* NameSystem.addStoredBlock:
addStoredBlock request received for blk_-7100767084720175376_361638 on
10.76.99.114:50010 size 23874613 But it does not belong to any file.


11/01/12 19:45:48 WARN hdfs.StateChange: DIR* NameSystem.completeFile:
failed to complete
/hbase/webtable/compaction.dir/1326275218/2628290592728420519 because
dir.getFileBlocks() is null  and pendingFile is null

11/01/12 19:45:48 INFO ipc.Server: IPC Server handler 54 on 8020, call
complete(/hbase/webtable/compaction.dir/1326275218/2628290592728420519,
DFSClient_880037874) from 1XX.XX.XX.XXXX :55925: error: java.io.IOException:
Could not complete write to file
/hbase/webtable/compaction.dir/1326275218/2628290592728420519 by
DFSClient_880037874
java.io.IOException: Could not complete write to file
/hbase/webtable/compaction.dir/1326275218/2628290592728420519 by
DFSClient_880037874
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:471)
        at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)




On Wed, Jan 12, 2011 at 10:40 PM, Stack <stack@duboce.net> wrote:

> On Wed, Jan 12, 2011 at 12:17 PM, charan kumar <charan.kumar@gmail.com>
> wrote:
> > 2011-01-11 16:00:27,489 FATAL
> > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog
> > required. Forcing server shutdown
> >
>
>
> What was in the log before this line?  It would say why the replay
> required.
>
> One thing you might do is grep 1138778035, the encoded name of the
> region that failed the write below in your master log.  Where was the
> region assigned?  Was it assigned two places concurrently?
>
> St.Ack
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message