Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates
 74.125.44.28 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:sender:to:subject:in-reply-to:mime-version
         :content-type:references:x-google-sender-auth;
        b=u/s8A0j9BZ53lhCBslVsTr4z8PEDJvV5UKVEK8Rv8keg/tHp5LoQCcmD2JYDJlweUQ
         g8T7pFhjUGeCAA6XBDFyfxGSrbOE0tmExTIdhz1bvv+lT+23KAYWF9M0OsXvifYnBhH8
         8xeZoGTsgFj2liqZxr9Qdu3sp1TCQOd+cRw+M=
Message-ID: <31a243e70810231137x5d5b053cs35ec80a9fa2cb017@mail.gmail.com>
Date: Thu, 23 Oct 2008 14:37:39 -0400
From: "Jean-Daniel Cryans" <jdcryans@apache.org>
Sender: jdcryans@gmail.com
To: hbase-user@hadoop.apache.org
Subject: Re: NotServingRegionException - Map/Reduce process fails
In-Reply-To: <4900C2B7.3010008@duboce.net>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_42464_25233458.1224787059214"
References: <1FD66D81-F3FC-4F89-B0C8-D0B480872252@gmail.com>
	 <4900C2B7.3010008@duboce.net>

------=_Part_42464_25233458.1224787059214
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Dru.

See also if it's a case of
HBASE-921<https://issues.apache.org/jira/browse/HBASE-921>because it
would make sense if not using hbase 0.18.1 and under a heavy
load.

J-D

On Thu, Oct 23, 2008 at 2:30 PM, stack <stack@duboce.net> wrote:

> Find the MR task that failed.  Click through the UI to look at its logs.
>  It may have interesting info.  Its probably complaining about a region not
> being available (NSRE).  Figure which region it is.  Use the region
> historian or grep in the master logs -- 'grep -v metaScanner REGIONNAME' so
> you avoid the metaScanner noise -- to see if you can figure the regions
> history around the failure.  Look too at loading around failure time.  Were
> you swapping, etc. (Ganglia or some such helps here).
>
> You might also test table is still wholesome -- that the MR job didn't
> damage the table.  A quick check that all regions are onlined and accessible
> is to scan for a column whose column family does exist but whose qualifier
> you know is not present: e.g. if you have columnfamily 'page' and you know
> there is no column 'page:xyz', scan with that (Enable DEBUG in log4j so you
> can see regions being loaded as scan progresses): "scan 'TABLENAME',
> ['page:xyz']".
>
> You might need to up the timeouts/retries.
> St.Ack
>
>
>
> Dru Jensen wrote:
>
>> Hi hbase-users,
>>
>> During a fairly large MR process, on the Reduce cycle as its writing its
>> results to a table, I see org.apache.hadoop.hbase.NotServingRegionException
>> in the region server log several times and then I see a split reporting it
>> was successful.
>>
>> Eventually, the Reduce process fails with
>> org.apache.hadoop.hbase.client.RetriesExhaustedException after 10 failed
>> attempts.
>>
>> What can I do to fix it?
>>
>> Thanks,
>> Dru
>>
>>
>>
>>
>>
>

------=_Part_42464_25233458.1224787059214--