Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of magnito@gmail.com designates
 209.85.160.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=xtpF6NWlblIHELOHxaaU5hBCFZa8OFICTWJGpOP7Ro9UVDyUp0FOTO6+CaPMhyb4YQ
         oN00aD1zYUr39qZAa3gqqWIM73ADbC+0CSI6ydX9yth6gZ7U5/MQOyXa8BgfY9ereAYG
         1GJ4UeE+2sQc523zqfDb2fzdWpHT/+LLonswY=
MIME-Version: 1.0
In-Reply-To: <BANLkTikSA4Hdi_5f=10jVroNwFC5_gxLTQ@mail.gmail.com>
References: <BANLkTimKXV0jaxQvW+7TAx5vrJLhfAcPVQ@mail.gmail.com>
	<COL117-W595BE076B1EF2006B98B4F8F9F0@phx.gbl>
	<BANLkTins7_+rAkJEjpTvT+GJ7=4Z+She0w@mail.gmail.com>
	<BANLkTikSA4Hdi_5f=10jVroNwFC5_gxLTQ@mail.gmail.com>
Date: Mon, 2 May 2011 12:56:08 -0700
Message-ID: <BANLkTikvW0utirD_OMTs438cGw_dGWwjTw@mail.gmail.com>
Subject: Re: one of our datanodes stops working after few hours
From: Jack Levin <magnito@gmail.com>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Tried removing yourkit and run on javasun, same thing.  We have some
threads blocked, does anyone know what they block on?

-Jack

On Mon, May 2, 2011 at 7:53 AM, Todd Lipcon <todd@cloudera.com> wrote:
> Hi Jack,
>
> Does this happen even if you aren't running Yourkit on the DN?
>
> Can you try using a Sun JDK instead of OpenJDK?
>
> -Todd
>
> On Sun, May 1, 2011 at 7:34 PM, Jack Levin <magnito@gmail.com> wrote:
>
>> Version: =A0 =A0 =A0 =A0 0.20.2+320 hdfs
>> .89 HBASE
>>
>> ulimit is 32k
>> xcievers is 5k
>>
>> Note from the jstack, I am not exceeding xcievers.
>>
>> -Jack
>>
>> On Sun, May 1, 2011 at 6:19 PM, Michael Segel <michael_segel@hotmail.com=
>
>> wrote:
>> >
>> >
>> > What's your xceivers set to?
>> > What's the ulimit -n =A0set for hdfs/hadoop user... (You didn't say wh=
ich
>> release/version you were using.)
>> >
>> >> Date: Sun, 1 May 2011 17:47:18 -0700
>> >> Subject: one of our datanodes stops working after few hours
>> >> From: magnito@gmail.com
>> >> To: user@hbase.apache.org
>> >>
>> >> I took a jstack (http://pastebin.com/5v6mHg3t). =A0 After few hours, =
its
>> >> literally staggers to a halt and gets very very slow... Any ideas
>> >> whats its blocking on?
>> >> (main issue is that fsreads for RS get really slow when that happens)=
.
>> >>
>> >> -Jack
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>