Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <49400BF2.3070702@duboce.net>
Date: Wed, 10 Dec 2008 10:35:30 -0800
From: stack <stack@duboce.net>
User-Agent: Thunderbird 2.0.0.18 (Macintosh/20081105)
MIME-Version: 1.0
To: hbase-user@hadoop.apache.org
Subject: Re: Sometimes hbase do not handle any request at all
References: <e3ac96540812100943k61635bf3k6c42772d53f9c7ca@mail.gmail.com>
In-Reply-To: <e3ac96540812100943k61635bf3k6c42772d53f9c7ca@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Which hbase Yunqing version?  When it is hung, can you see which region 
it fails on (Should be in the exception when the reduce fails).  Can you 
see why the reduce fails?  Is it the TaskTracker timing it out after 10 
minutes or is it timing out on a particular hbase region.  If you can 
figure the region, see which server its hosted on (Use UI or master logs 
to figure this).  Then go to that server, tail its logs.  Can you figure 
what its doing?   Is it stuck?  Thread dump it a few times and see if 
you can see where its blocked -- you can thread dump the server from the 
UI.  When you thread dump via the UI, it also outputs into the hbase 
regionserver log.  Post them to this list if you'd like us to look at 
them for you.

Thanks,
St.Ack

Zhou, Yunqing wrote:
> I'm using IdentityTableReducer to insert about 1M records into hbase
> on a 23 machines cluster.
> but I found that sometimes it got junked.
> everything suspended. all machine's work load are zeros.
>
> then some reducer failed, new reducer begin to insert some records.
> then the pheomenon appeared again.
>
> I've set all machines nofile limit to 32768.
> Do you know that's why?
> Thanks.
>