hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: HRegionServer: Scanner, lease expired issue.
Date Wed, 11 Mar 2009 13:10:05 GMT
Schubert,

When you see INFO-level output, it means that it is normal operations.
A client scanner has a lease which lasts
hbase.regionserver.lease.period which by default is 1 minute. If your
client doesn't report back under that time (do a next() for example),
the scanner lease expires. This way, the region servers don't have to
support an infinite number of leases if the clients never close them.

That said, what are you doing in your map tasks that requires your
client to take more than 1 minute to do next()? If it is something
that requires more than 1 minute, do consider changing that value in
you hbase-site.xml to something like 120000 millis.

J-D

On Wed, Mar 11, 2009 at 5:30 AM, schubert zhang <zsongbo@gmail.com> wrote:
> I made more test. I think my guess may be the cause.
> In a latest test, there are 11 killed map tasks, and there are also 11
> expired leases.
>
> I think may be the MapReduce code should be improved.
>
> Schubert
> On Wed, Mar 11, 2009 at 4:38 PM, schubert zhang <zsongbo@gmail.com> wrote:
>
>> Hi all,
>> I am running a MapReduce job to read regions(row ranges) in Map task. The
>> job can work completely and correctly.
>>
>> But, usually, I find following INFO in hbase regionservre's log file.
>>
>> 2009-03-11 15:57:06,428 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>> -6710483098243538624 lease expired
>>
>> To debug this issue, I add some new log information in:
>> src/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
>> (1) in addScanner(), I add:
>>  LOG.info("----Schubert Add Scanner " + scannerName + " lease create!");
>> (2) in close(), I add:
>> LOG.info("----Schubert close " + scannerName + " lease cancel!");
>>
>> Then I redo my mapreduce job. And I found some leases/scanners are not
>> closed.
>>
>> I can make sure that my mapreduce RecordReader implements the close()
>> method and calls the close() method of the scanner.
>>
>> following log info is copy from one region server of my HBase clasuer, and
>> to make clear, I move the Add and Close together.
>>
>> 2009-03-11 15:52:41,406 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> 5785283094618320498 lease create!
>> 2009-03-11 15:54:08,423 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
>> 5785283094618320498 lease cancel!
>>
>> 2009-03-11 15:52:46,493 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> -3726092592914221244 lease create!
>> 2009-03-11 15:54:11,676 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
>> -3726092592914221244 lease cancel!
>>
>> 2009-03-11 15:52:51,984 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> 313013793996434532 lease create!
>> 2009-03-11 15:54:18,428 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
>> 313013793996434532 lease cancel!
>>
>> 2009-03-11 15:54:10,213 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> -5389407006476765661 lease create!
>> 2009-03-11 15:55:48,337 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
>> -5389407006476765661 lease cancel!
>>
>> 2009-03-11 15:54:13,505 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> 8463558669492796494 lease create!
>> 2009-03-11 15:55:59,152 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
>> 8463558669492796494 lease cancel!
>>
>> 2009-03-11 15:54:20,118 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> 8149590914348186380 lease create!
>> 2009-03-11 15:56:04,350 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
>> 8149590914348186380 lease cancel!
>>
>> 2009-03-11 15:54:29,466 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> 4137760362936278647 lease create!
>> 2009-03-11 15:56:10,765 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
>> 4137760362936278647 lease cancel!
>>
>> 2009-03-11 15:55:34,786 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> -5919619533272076177 lease create!
>> 2009-03-11 15:56:50,472 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>> -5919619533272076177 lease expired
>>
>> 2009-03-11 15:55:36,961 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> -7919803182075534481 lease create!
>> 2009-03-11 15:57:04,200 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>> -7919803182075534481 lease expired
>>
>> 2009-03-11 15:55:45,992 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> 2812856283312715383 lease create!
>> 2009-03-11 15:57:14,442 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>> 2812856283312715383 lease expired
>>
>> 2009-03-11 15:56:03,033 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
>> -6710483098243538624 lease create!
>> 2009-03-11 15:57:06,428 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>> -6710483098243538624 lease expired
>>
>> Wen can see, the
>> Scanners  -5919619533272076177, -7919803182075534481, 2812856283312715383
>> and -6710483098243538624 are not closed.
>>
>> I guess, it maybe caused by MapReduce kill task (speculative).
>>
>> Schubert
>>
>

Mime
View raw message