hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Rozendaal <dlrozend...@gmail.com>
Subject HBASE-1927 (was Re: HBase 0.20.1 scanners not closing properly (memory leak))
Date Thu, 22 Oct 2009 07:33:06 GMT
Issue created: HBASE-1927

On 21 okt 2009, at 23:55, stack wrote:

> Thanks for digging in Erik.  Nice one.  Would you mind making an  
> issue of
> your findings?  File it against 0.20.2 so we can roll out the fix in  
> next
> 0.20.x release.
> St.Ack
>
> On Wed, Oct 21, 2009 at 2:25 PM, Erik Rozendaal  
> <dlrozendaal@gmail.com>wrote:
>
>> I did some more investigation into this issue since after the  
>> original
>> issue stop occuring I noticed that MemStoreScanners where still  
>> being leaked
>> when scanning a store with an empty MemStore.
>>
>> The cause looks to be the KeyValueHeap constructor. It drops  
>> scanners when
>> the scanner's peek() method returns null (line 58 of KeyValueHeap).
>>
>> Unfortunately some scanners (like StoreScanner and MemStoreScanner)
>> register themselves to some global list when constructed and only  
>> deregister
>> on close().
>>
>> So a quick fix may be to add an
>>
>> } else {
>>   scanner.close()
>> }
>>
>> to the KeyValueHeap constructor when the scanner is not added. I'm  
>> not sure
>> if this is the cleanest fix though...
>>
>> Regards,
>> Erik
>>
>>
>> On 21 okt 2009, at 14:23, Guilherme Germoglio wrote:
>>
>> Hello Erik,
>>> I think your attachments were blocked. Could you please upload them
>>> somewhere else?
>>>
>>> On Wed, Oct 21, 2009 at 8:44 AM, Erik Rozendaal <dlrozendaal@gmail.com
>>>> wrote:
>>>
>>> Hi all,
>>>>
>>>> After some performance testing on my HBase 0.20.1 development  
>>>> environment
>>>> (running in pseudo- and full-distributed mode on a single laptop) I
>>>> noticed
>>>> that scanners do not get closed properly on the region server.  
>>>> After
>>>> creating a heap dump with Netbeans I can see the StoreScanner  
>>>> instances
>>>> are
>>>> still in the Store.changedReaderObservers collection.
>>>>
>>>> Each StoreScanner instance has the "closed" flag set to false, so  
>>>> it
>>>> looks
>>>> like the StoreScanner.close() method was never called.
>>>>
>>>> I double-checked my client code and counted the number of times I  
>>>> create
>>>> and close a scanner, and these counts match. I also found this is
>>>> repeatable
>>>> from the hbase shell. Open the shell, scan some table, take a  
>>>> heap dump
>>>> and
>>>> you'll find an unclosed StoreScanner objects in the
>>>> Store.changedReaderObservers collection.
>>>>
>>>> I've attached screenshots of the number of StoreScanner instances  
>>>> (after
>>>> 30.001 scans) and the Store.changedReaderObservers collection of  
>>>> one of
>>>> the
>>>> Stores (notice that the closed field's value is 0 => false).
>>>>
>>>> Ultimately the region server runs out of memory and crashes.
>>>>
>>>> Has anyone experience similar problems?
>>>>
>>>> Regards,
>>>> Erik
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Guilherme
>>>
>>> msn: guigermoglio@hotmail.com
>>> homepage: http://sites.google.com/site/germoglio/
>>>
>>
>>


Mime
View raw message