hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Gray <jl...@streamy.com>
Subject Re: HBASE-1927 (was Re: HBase 0.20.1 scanners not closing properly (memory leak))
Date Thu, 22 Oct 2009 17:12:20 GMT
Erik,

I just put up a patch with the fix you described and a unit test that 
replicates the behavior.

Please test to confirm it works.  If so, drop a note in the issue and I 
will commit.

Thanks for finding the bug.

JG

Erik Rozendaal wrote:
> Issue created: HBASE-1927
> 
> On 21 okt 2009, at 23:55, stack wrote:
> 
>> Thanks for digging in Erik.  Nice one.  Would you mind making an issue of
>> your findings?  File it against 0.20.2 so we can roll out the fix in next
>> 0.20.x release.
>> St.Ack
>>
>> On Wed, Oct 21, 2009 at 2:25 PM, Erik Rozendaal 
>> <dlrozendaal@gmail.com>wrote:
>>
>>> I did some more investigation into this issue since after the original
>>> issue stop occuring I noticed that MemStoreScanners where still being 
>>> leaked
>>> when scanning a store with an empty MemStore.
>>>
>>> The cause looks to be the KeyValueHeap constructor. It drops scanners 
>>> when
>>> the scanner's peek() method returns null (line 58 of KeyValueHeap).
>>>
>>> Unfortunately some scanners (like StoreScanner and MemStoreScanner)
>>> register themselves to some global list when constructed and only 
>>> deregister
>>> on close().
>>>
>>> So a quick fix may be to add an
>>>
>>> } else {
>>>   scanner.close()
>>> }
>>>
>>> to the KeyValueHeap constructor when the scanner is not added. I'm 
>>> not sure
>>> if this is the cleanest fix though...
>>>
>>> Regards,
>>> Erik
>>>
>>>
>>> On 21 okt 2009, at 14:23, Guilherme Germoglio wrote:
>>>
>>> Hello Erik,
>>>> I think your attachments were blocked. Could you please upload them
>>>> somewhere else?
>>>>
>>>> On Wed, Oct 21, 2009 at 8:44 AM, Erik Rozendaal <dlrozendaal@gmail.com
>>>>> wrote:
>>>>
>>>> Hi all,
>>>>>
>>>>> After some performance testing on my HBase 0.20.1 development 
>>>>> environment
>>>>> (running in pseudo- and full-distributed mode on a single laptop) I
>>>>> noticed
>>>>> that scanners do not get closed properly on the region server. After
>>>>> creating a heap dump with Netbeans I can see the StoreScanner 
>>>>> instances
>>>>> are
>>>>> still in the Store.changedReaderObservers collection.
>>>>>
>>>>> Each StoreScanner instance has the "closed" flag set to false, so it
>>>>> looks
>>>>> like the StoreScanner.close() method was never called.
>>>>>
>>>>> I double-checked my client code and counted the number of times I 
>>>>> create
>>>>> and close a scanner, and these counts match. I also found this is
>>>>> repeatable
>>>>> from the hbase shell. Open the shell, scan some table, take a heap 
>>>>> dump
>>>>> and
>>>>> you'll find an unclosed StoreScanner objects in the
>>>>> Store.changedReaderObservers collection.
>>>>>
>>>>> I've attached screenshots of the number of StoreScanner instances 
>>>>> (after
>>>>> 30.001 scans) and the Store.changedReaderObservers collection of 
>>>>> one of
>>>>> the
>>>>> Stores (notice that the closed field's value is 0 => false).
>>>>>
>>>>> Ultimately the region server runs out of memory and crashes.
>>>>>
>>>>> Has anyone experience similar problems?
>>>>>
>>>>> Regards,
>>>>> Erik
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> -- 
>>>> Guilherme
>>>>
>>>> msn: guigermoglio@hotmail.com
>>>> homepage: http://sites.google.com/site/germoglio/
>>>>
>>>
>>>
> 

Mime
View raw message