hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikael Sitruk <mikael.sit...@gmail.com>
Subject Re: LeaseException despite high hbase.regionserver.lease.period
Date Sun, 12 Feb 2012 21:35:30 GMT
I had same wired problem last week when i upgraded from cdh3u0 to cdh3u2.
I got lease exception even when the  hbase.regionserver.lease.period was
set '300000'. The lease occurred after 1 mins or more.
Prior the upgrade everything was fine, i was executing with map reduce 8
slots.
The lease totally disappeared when i use only 1 slot, i thought there were
some concurrency problem with MR integration, but this weekend some of the
machines where restarted and suddenly everything is fine!! I'm back to 8
slot and the extract are running. (i run several time even without even one
lease exception)
In my case NO region server were stuck in transition.
It is really frustrating that i cannot point on what was the real problem.
Even log with debug did not point on problems (perhaps because it is also
missing some debug statement like when a scanner lease is added to the RS)

Mikael.S


On Sun, Feb 12, 2012 at 9:55 PM, Alex Baranau <alex.baranov.v@gmail.com>wrote:

> Hm, weird. Looks like now it goes well. What I did is once again restarted
> several regionservers (noticed that some where holding regions which were
> stuck in "in transition" state). And started back the job.
>
> Btw, forgot to mention that caching on Scanner was set to only 64 and I the
> estimate for records size is < 100KB. Anyhow, the weird thing was that the
> limits were set to 10min and it started to fail even before 10 min has
> passed (sometimes after 3-4 minutes).
>
> Have to go and can't watch the job to the end, will write whether it
> succeed or not. If face same problems again will try brutal variant: set
> caching = 10 (or even 1), set batch = 10 (or even 1).
>
> Alex
>
> On Sun, Feb 12, 2012 at 1:49 PM, Alex Baranau <alex.baranov.v@gmail.com
> >wrote:
>
> > Hi,
> >
> > 0.90.4-cdh3u2
> >
> > Alex
> >
> >
> > On Sun, Feb 12, 2012 at 1:44 PM, <yuzhihong@gmail.com> wrote:
> >
> >> Which version of hbase are you using ?
> >>
> >> Thanks
> >>
> >>
> >>
> >> On Feb 12, 2012, at 10:41 AM, Alex Baranau <alex.baranov.v@gmail.com>
> >> wrote:
> >>
> >> > Hello,
> >> >
> >> > I'm getting scanner lease exceptions during mapreduce job [1] after
> >> running
> >> > it for less than 7 minutes. Though I have set
> >> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
> >> > configuration on master and all regionservers and master (and
> restarted
> >> > all). Also set it in job's configuration.
> >> >
> >> > Is there anything else I could do to eliminate the exception?
> >> >
> >> > Thanks,
> >> > Alex
> >> >
> >> > [1]
> >> >
> >> > org.apache.hadoop.hbase.regionserver.LeaseException:
> >> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> >> > ʼ3854333525977315054ʼ does not exist
> >> >                  at
> >> >
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
> >> >
> >> >                  at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
> >> >
> >> >                  at
> sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
> >> > Source)
> >> >
> >> >                  at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > Last
> >> >                  at
> >> > java.lang.reflect.Method.invoke(Method.java:597)
> >> > 4KB
> >> > al FAILED         at
> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> >> > Last
> >> >                  at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> >> > 8KB
> >> >
> >> > All
> >> >                  at
> >> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> > Method)
> >> >
> >> >                  at
> >> >
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> >
> >> >                  at
> >> >
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> >
> >> >                  at
> >> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> >
> >> >                  at
> >> > org.ap
> >>
> >
> >
>



-- 
Mikael.S

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message