Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Message-ID: <14703408.545481282693818782.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 19:50:18 -0400 (EDT)
From: "Jean-Daniel Cryans (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Subject: [jira] Commented: (HBASE-2861) regionserver's logsyncer thread
 hangs on DFSClient$DFSOutputStream.waitForAckedSeqno
In-Reply-To: <22344474.501291279738190080.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902167#action_12902167 ] 

Jean-Daniel Cryans commented on HBASE-2861:
-------------------------------------------

Kannan, did you ever see this issue again? Do you have more info from the DN's log or jstack?

> regionserver's logsyncer thread hangs on DFSClient$DFSOutputStream.waitForAckedSeqno
> ------------------------------------------------------------------------------------
>
>                 Key: HBASE-2861
>                 URL: https://issues.apache.org/jira/browse/HBASE-2861
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Kannan Muthukkaruppan
>            Priority: Blocker
>             Fix For: 0.90.0
>
>         Attachments: jstack.txt
>
>
> During loads into HBase, we are noticing that a RS is sometimes getting stuck.
> The logSyncer thread:
> {code}
>       at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3367)
>         - locked <0x00002aaac7fef748> (a java.util.LinkedList)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3301)
>         at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
>         at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:124)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.hflush(HLog.java:949)
> {code}
> A lot of other threads are stuck on:
> {code}
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.addToSyncQueue(HLog.java:916)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:936)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:828)
>         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1657)
>         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1425)
>         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1393)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1665)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2326)
> {code}
> Subsequently, trying to disable the table, which in turn attempts to close the region(s), caused internalFlushCache() also to get stuck here:
> {code}
>       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
>         at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
>         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:974)
>         at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:511)
>         - locked <0x00002aaab76af670> (a java.lang.Object)
>         at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:463)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:1468)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1329)
> {code}
> I'll attach the full jstack trace soon.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.