Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Wed, 10 Dec 2014 23:19:12 +0000 (UTC)
From: "Colin Patrick McCabe (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12760503.1418150220000.125582.1418253552741@Atlassian.JIRA>
In-Reply-To: <JIRA.12760503.1418150220000@Atlassian.JIRA>
References: <JIRA.12760503.1418150220000@Atlassian.JIRA>
 <JIRA.12760503.1418150220136@arcas>
Subject: [jira] [Commented] (ACCUMULO-3396) HDFS reads are hanging
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/ACCUMULO-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14241934#comment-14241934 ] 

Colin Patrick McCabe commented on ACCUMULO-3396:
------------------------------------------------

Basically, I am wondering if you are hitting HDFS-7489.  It should be easy to check, just drop this in your hdfs config.

> HDFS reads are hanging
> ----------------------
>
>                 Key: ACCUMULO-3396
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3396
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.6.0, 1.6.1
>         Environment: rhel6 linux 2.6.32-279 (x86_64)
> java 1.7.0_67-b01
> hadoop CDH5.1.2, HA (2) federated (2) NN configuration
> large production cluster
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>            Priority: Blocker
>
> On large clusters we are seeing various forms of HDFS reads hanging:
> Queries that never return.
> Major compactions that hang.
> Accumulo 1.6.1 incorporates detectors that report hanging major compactions and a monitor display that reports scans by age.
> Stack traces show readers in sun.nio.ch.EPollArrayWrapper.epollWait and in org.apache.hadoop.ipc.Client.Call(Client.java:1362).
> Netstat results for the tablet server shows many connections with a single byte waiting on the Recv-Q of the process, and no bytes waiting on the Send-Q.
> strace of the jvm shows the typical jvm thread noise (futex calls)
> jstack shows lots of read-requests to the NN.
> long-running MajC's do complete, albeit slowly.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)