Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of vrAMAnATHAN00@aol.com
 designates 205.188.105.144 as permitted sender)
References: <8CDDF2B8EEAF4C9-1B14-35A43@webmail-m071.sysops.aol.com>
 <BANLkTi=uE4QwU2yt=s7N4-CE1KCQ_YR67Q@mail.gmail.com>
To: user@hbase.apache.org
Subject: Re: mapreduce job failure
Date: Mon, 16 May 2011 17:08:21 -0400
In-Reply-To: <BANLkTi=uE4QwU2yt=s7N4-CE1KCQ_YR67Q@mail.gmail.com>
MIME-Version: 1.0
From: Venkatesh <vramanathan00@aol.com>
Content-Type: multipart/alternative;
 boundary="--------MB_8CDE224854DB89D_838_279C_webmail-m093.sysops.aol.com"
Message-Id: <8CDE2248546947A-838-17AE@webmail-m093.sysops.aol.com>

----------MB_8CDE224854DB89D_838_279C_webmail-m093.sysops.aol.com
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="us-ascii"

Thanks J-D

Using hbase-0.20.6, 49 node cluster

The map reduce job involve a full table scan...(region size 4 gig)
The job runs great for 1 week..
Starts failing after 1 week of data accumulation (about 3000 regions)

 About 400 regions get created per day...

Can you suggest any tunables at the HBase level. or HDFS level.?

Also, I've one more issue..when region servers die..Errors below: (any sug=
gestion here is helpfull as well)

org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namen=
ode.LeaseExpiredException: No lease on /hbase_data_one_110425/.../compacti=
on.dir/249610074/4534752250560182124 File does not exist. Holder DFSClient=
_-398073404 does not have any open files.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(=
FSNamesystem.java:1332)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(=
FSNamesystem.java:1323)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAddition=
alBlock(FSNamesystem.java:1251)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNo=
de.java:422)
        at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMetho=
dAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
=20

=20


=20

=20

-----Original Message-----
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Sent: Fri, May 13, 2011 12:39 am
Subject: Re: mapreduce job failure


All that means is that the task stayed in map() for 10 minutes,

blocked on "something".


If you were scanning an hbase table, and didn't get a new row after 1

minute, then the scanner would expire. That's orthogonal tho.


You need to figure what you're blocking on, add logging and try to

jstack your Child processes for example.


J-D


On Thu, May 12, 2011 at 7:21 PM, Venkatesh <vramanathan00@aol.com> wrote:

>

>  Hi

> Using hbase-0.20.6

>

> mapreduce job started failing in the map phase (using hbase table as inp=
ut for=20

mapper)..(ran fine for a week or so starting with empty tables)..

>

> task tracker log:

>

>

> Task attempt_201105121141_0002_m_000452_0 failed to report status for 60=
0=20

seconds. Killing

>

>

> Region server log:

>

> 2011-05-12 18:27:39,919 INFO org.apache.hadoop.hbase.regionserver.HRegio=
nServer:=20

Scanner -7857209327501974146 lease expired

>

> 2011-05-12 18:28:29,716 ERROR org.apache.hadoop.hbase.regionserver.HRegi=
onServer:org.apache.hadoop.hbase.UnknownScannerException:=20

Name: -7857209327501974146

>        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegio=
nServer.java:1880)=20

       at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)   =
     at=20

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:25)

>        at java.lang.reflect.Method.invoke(Method.java:597)

>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java=
:657)

>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServe=
r.java:915)

>

>

> 2011-05-12 18:28:29,897 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serv=
er=20

handler 3 on 60020, call next(-78572093275019

> 74146, 1) from .....:35202: error: org.apache.hadoop.hbase.UnknownScanne=
rException:=20

Name: -7857209327501974146

> org.apache.hadoop.hbase.UnknownScannerException: Name: -7857209327501974=
146

>        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegio=
nServer.java:1880)

>        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)

>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth=
odAccessorImpl.java:25)=20

       at java.lang.reflect.Method.invoke(Method.java:597)

>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java=
:657)

>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServe=
r.java:915)

>

> I don't see any error in datanodes

>

> Appreciate any help

> thanks

> v

>

>

>


=20

----------MB_8CDE224854DB89D_838_279C_webmail-m093.sysops.aol.com--