lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <>
Subject Re: memory use
Date Sun, 24 Jan 2010 03:04:02 GMT
On Sat, Jan 23, 2010 at 08:25:04PM -0600, Peter Karman wrote:
> I have a collection of about 90k+ docs totaling about 2.2G.
> When I index with KS 0.30_072 I see memory use climb to about 80M and hold 
> relatively steady.
> With svn trunk, I see memory use climb steadily throughout the 15 minutes 
> or so it takes to index, reaching 600M+.

We gots us a memory leak, then.  Not surprising.  It's hard to avoid making
refcounting errors while developing.  

That's why we have test_valgrind... only the suppressions file
trunk/devel/conf/kinoperl.supp has been wrong since the switch to
LockFreeRegistry, so test_valgrind is currently broken.

> Initially I thought this was due to the recent changes in trunk to 
> SortExternal and so didn't think much of it since I knew that feature was 
> currently receiving some attention to reduce memory use. But I didn't want 
> possible other leaks to go unreported.
> So I guess my question is: is that kind of memory use to be expected?

If you have sortable fields, then high memory usage is to be expected until we
fix SortWriter.  However, SortWriter hasn't changed fundamentally in between
0.30_72 and now.  So it would seem that this is a leak, almost certainly due
to a refcounting error.

Marvin Humphrey

View raw message