Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Message-ID: <1469915837.1212607965553.JavaMail.jira@brutus>
Date: Wed, 4 Jun 2008 12:32:45 -0700 (PDT)
From: "stack (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Commented: (HBASE-667) Hung regionserver; hung on hdfs:
 writeChunk, DFSClient.java:2126, DataStreamer socketWrite
In-Reply-To: <1938233137.1212607965446.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602427#action_12602427 ] 

stack commented on HBASE-667:
-----------------------------

When in this state, regionserver will not go down.

> Hung regionserver; hung on hdfs: writeChunk, DFSClient.java:2126, DataStreamer socketWrite
> ------------------------------------------------------------------------------------------
>
>                 Key: HBASE-667
>                 URL: https://issues.apache.org/jira/browse/HBASE-667
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.1.2
>            Reporter: stack
>
> Internally, came across a hung regionserver.  Here is relevant excerpt from thread dump:
> {code}
> "ResponseProcessor for block blk_-6991279486194843565" daemon prio=1 tid=0x00002aab3ac13c50 nid=0x7ad7 runnable [0x0000000043080000..0x0000000043080d00]
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(Unknown Source)
>         at java.io.DataInputStream.readFully(Unknown Source)
>         at java.io.DataInputStream.readLong(Unknown Source)
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:1734)
> ...
> "IPC Server handler 3 on 60020" daemon prio=1 tid=0x00002aab3c05abf0 nid=0x6b61 waiting for monitor entry [0x0000000042878000..0x0000000042878d00]
>         at org.apache.hadoop.hbase.HLog.append(HLog.java:371)
>         - waiting to lock <0x00002aaab69d1180> (a java.lang.Integer)
>         at org.apache.hadoop.hbase.HRegion.update(HRegion.java:1629)
>         at org.apache.hadoop.hbase.HRegion.batchUpdate(HRegion.java:1432)
>         at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1552)
>         at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
> .....
> "IPC Server handler 1 on 60020" daemon prio=1 tid=0x00002aab3c3220a0 nid=0x6b5f waiting for monitor entry [0x0000000042676000..0x0000000042676c00]
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:2126)
>         - waiting to lock <0x00002aaab69d1a28> (a java.util.LinkedList)
>         at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:141)
>         at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
>         at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
>         - locked <0x00002aaab69d15b0> (a org.apache.hadoop.dfs.DFSClient$DFSOutputStream)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:41)
>         at java.io.DataOutputStream.write(Unknown Source)
>         - locked <0x00002aaab69d1228> (a org.apache.hadoop.fs.FSDataOutputStream)
>         at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:990)
>         - locked <0x00002aaab69d1050> (a org.apache.hadoop.io.SequenceFile$Writer)
>         at org.apache.hadoop.hbase.HLog.append(HLog.java:387)
>         - locked <0x00002aaab69d1180> (a java.lang.Integer)
>         at org.apache.hadoop.hbase.HRegion.update(HRegion.java:1629)
>         at org.apache.hadoop.hbase.HRegion.batchUpdate(HRegion.java:1432)
>         at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1552)
>         at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
> .....
> "DataStreamer for file /hbase/aa0-005-2.u.powerset.com/log_208.76.45.223_1212443824255_60020/hlog.dat.000 block blk_-6991279486194843565" daemon prio=1 tid=0x00002aab3c1b2e70 nid=0x6b50 runnable [0x0000000041969000..0x0000000041969c80]
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at java.net.SocketOutputStream.socketWrite(Unknown Source)
>         at java.net.SocketOutputStream.write(Unknown Source)
>         at java.io.BufferedOutputStream.write(Unknown Source)
>         - locked <0x00002aaab7652b70> (a java.io.BufferedOutputStream)
>         at java.io.DataOutputStream.write(Unknown Source)
>         - locked <0x00002aaab7652288> (a java.io.DataOutputStream)
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1631)
>         - locked <0x00002aaab69d1a28> (a java.util.LinkedList)
> {code}
> I've seen this before.  I saw this this morning where a pure hadoop client was hung in same way.  This is hadoop 0.16.4.  Seems like a pure hadoop prob.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.