hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Baranau <alex.barano...@gmail.com>
Subject Re: LeaseException despite high hbase.regionserver.lease.period
Date Mon, 13 Feb 2012 09:38:10 GMT
Yeah, looks like the same thing.

The job went well eventually (many mappers, lasted for 13h +, no errors at
all).

Alex

On Sun, Feb 12, 2012 at 4:35 PM, Mikael Sitruk <mikael.sitruk@gmail.com>wrote:

> I had same wired problem last week when i upgraded from cdh3u0 to cdh3u2.
> I got lease exception even when the  hbase.regionserver.lease.period was
> set '300000'. The lease occurred after 1 mins or more.
> Prior the upgrade everything was fine, i was executing with map reduce 8
> slots.
> The lease totally disappeared when i use only 1 slot, i thought there were
> some concurrency problem with MR integration, but this weekend some of the
> machines where restarted and suddenly everything is fine!! I'm back to 8
> slot and the extract are running. (i run several time even without even one
> lease exception)
> In my case NO region server were stuck in transition.
> It is really frustrating that i cannot point on what was the real problem.
> Even log with debug did not point on problems (perhaps because it is also
> missing some debug statement like when a scanner lease is added to the RS)
>
> Mikael.S
>
>
> On Sun, Feb 12, 2012 at 9:55 PM, Alex Baranau <alex.baranov.v@gmail.com
> >wrote:
>
> > Hm, weird. Looks like now it goes well. What I did is once again
> restarted
> > several regionservers (noticed that some where holding regions which were
> > stuck in "in transition" state). And started back the job.
> >
> > Btw, forgot to mention that caching on Scanner was set to only 64 and I
> the
> > estimate for records size is < 100KB. Anyhow, the weird thing was that
> the
> > limits were set to 10min and it started to fail even before 10 min has
> > passed (sometimes after 3-4 minutes).
> >
> > Have to go and can't watch the job to the end, will write whether it
> > succeed or not. If face same problems again will try brutal variant: set
> > caching = 10 (or even 1), set batch = 10 (or even 1).
> >
> > Alex
> >
> > On Sun, Feb 12, 2012 at 1:49 PM, Alex Baranau <alex.baranov.v@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > 0.90.4-cdh3u2
> > >
> > > Alex
> > >
> > >
> > > On Sun, Feb 12, 2012 at 1:44 PM, <yuzhihong@gmail.com> wrote:
> > >
> > >> Which version of hbase are you using ?
> > >>
> > >> Thanks
> > >>
> > >>
> > >>
> > >> On Feb 12, 2012, at 10:41 AM, Alex Baranau <alex.baranov.v@gmail.com>
> > >> wrote:
> > >>
> > >> > Hello,
> > >> >
> > >> > I'm getting scanner lease exceptions during mapreduce job [1] after
> > >> running
> > >> > it for less than 7 minutes. Though I have set
> > >> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
> > >> > configuration on master and all regionservers and master (and
> > restarted
> > >> > all). Also set it in job's configuration.
> > >> >
> > >> > Is there anything else I could do to eliminate the exception?
> > >> >
> > >> > Thanks,
> > >> > Alex
> > >> >
> > >> > [1]
> > >> >
> > >> > org.apache.hadoop.hbase.regionserver.LeaseException:
> > >> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > >> > ʼ3854333525977315054ʼ does not exist
> > >> >                  at
> > >> >
> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
> > >> >
> > >> >                  at
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
> > >> >
> > >> >                  at
> > sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
> > >> > Source)
> > >> >
> > >> >                  at
> > >> >
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >> > Last
> > >> >                  at
> > >> > java.lang.reflect.Method.invoke(Method.java:597)
> > >> > 4KB
> > >> > al FAILED         at
> > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> > >> > Last
> > >> >                  at
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> > >> > 8KB
> > >> >
> > >> > All
> > >> >                  at
> > >> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > >> > Method)
> > >> >
> > >> >                  at
> > >> >
> > >>
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > >> >
> > >> >                  at
> > >> >
> > >>
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > >> >
> > >> >                  at
> > >> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > >> >
> > >> >                  at
> > >> > org.ap
> > >>
> > >
> > >
> >
>
>
>
> --
> Mikael.S
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message