lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <dawid.we...@cs.put.poznan.pl>
Subject Re: FSTCompletionTest failure "At least 0.5MB RAM buffer is needed"
Date Thu, 02 Feb 2012 16:08:01 GMT
long freeHeap = Runtime.getRuntime().freeMemory();

Indeed, this doesn't look right; it'd have to be used in combination
with (maxMemory - totalMemory) because that's how much the heap can
grow? The problem is none of this is atomic, so the result can
unpredictable. There are other methods in management interface that
permit a somewhat more detailed checks. Don't know if they guarantee
atomicity of the returned snapshot, but I doubt it.

http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/MemoryMXBean.html#getHeapMemoryUsage()
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/MemoryPoolMXBean.html#getPeakUsage()

Dawid

On Thu, Feb 2, 2012 at 4:58 PM, Robert Muir <rcmuir@gmail.com> wrote:
> Doron, this sounds like something we should fix: I think we should
> open a JIRA issue for it.
>
> On Mon, Jan 30, 2012 at 12:34 PM, Doron Cohen <cdoronc@gmail.com> wrote:
>> Hi, this test consistently fails on Windows with an IBM JDK, with this
>> error:
>>
>>> java.lang.IllegalArgumentException: At least 0.5MB RAM buffer is needed:
>>> 432472
>>>  at org.apache.lucene.search.suggest.fst.Sort.<init>(Sort.java:159)
>>>  at org.apache.lucene.search.suggest.fst.Sort.<init>(Sort.java:150)
>>>  at
>>> org.apache.lucene.search.suggest.fst.FSTCompletionLookup.build(FSTCompletionLookup.java:181)
>>>  at
>>> org.apache.lucene.search.suggest.fst.FSTCompletionTest.testLargeInputConstantWeights(FSTCompletionTest.java:164)
>>>
>>> NOTE: reproduce with: ant test -Dtestcase=FSTCompletionTest
>>> -Dtestmethod=testLargeInputConstantWeights
>>> -Dtests.seed=63069fe8e90d25f1:-4459dd4f7ddf2b26:71f954eeb3888217 -Dargs="->
>>> Dfile.encoding=UTF-8"
>>> NOTE: test params are: locale=et_EE, timezone=America/Argentina/La_Rioja
>>> NOTE: all tests run in this JVM:
>>> [FSTCompletionTest]
>>> NOTE: Windows 7 6.1 build 7601 Service Pack 1 amd64/IBM Corporation 1.6.0
>>> (64-bit)/cpus=2,threads=4,free=330928,total=6291456
>>
>> The memory provided to sort is computed in
>> contrib/spell/.../suggest.fst.Sort.automatic():
>>
>> {code}
>>     public static BufferSize automatic() {
>>       long freeHeap = Runtime.getRuntime().freeMemory();
>>       return new BufferSize(Math.min(MIN_BUFFER_SIZE_MB * MB, freeHeap /
>> 2));
>>     }
>> {code}
>>
>> With Oracle's Java 6 the test passed.
>>
>> With IBM JDK, the fails even with -Xmx700m.   (Allow allocating at most
>> 177M.)
>> But It will pass with just -Xms10m.   (Allow allocating 10M at start.)
>>
>> So, if in a certain moment in a JVM's life the currently allocated memory is
>> almost exhausted, Sort will fail, even if the settings in effect allow to
>> allocate more heap.
>>
>> It seems "nice" that Sort attempts to behave "nice" - use as much as half of
>> the currently free heap.
>> This makes sense.
>> But perhaps in the situation that there's not enough free memory but the max
>> memory settings allow to allocate more, a reasonable minimum should be
>> returned, even the minimum of 0.5M. This will cause additional memory
>> allocation for heap, but I think in this case it is justified?
>>
>> Doron
>
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message