lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yura Smolsky <i...@altervision.biz>
Subject Re[2]: big index and multi threaded IndexSearcher
Date Wed, 16 Feb 2005 20:29:40 GMT
Hello, Erik.

EH> Are you using multiple IndexSearcher instances?    Or only one and
EH> sharing it across multiple threads?

EH> If using a single shared IndexSearcher instance doesn't help, it may be
EH> beneficial to port your code to Java and try it there.

I have single instance of IndexSearcher and I pass reference of it to each
thread. I will port code to Java if no other ideas will come my
mind...

EH> On Feb 16, 2005, at 3:04 PM, Yura Smolsky wrote:

>> Hello.
>>
>> I use PyLucene, python port of Lucene.
>>
>> I have problem about using big index (50Gb) with IndexSearcher
>> from many threads.
>> I use IndexSearcher from PyLucene's PythonThread. It's really a wrapper
>> around a Java/libgcj thread that python is tricked into thinking
>> it's one of its own.
>>
>> The core of problem:
>> When I have many threads (more than 5) I receive this exception:
>>   File "/usr/lib/python2.4/site-packages/PyLucene.py", line 2241, in
>> search
>>     def search(*args): return _PyLucene.Searcher_search(*args)
>> ValueError: java.lang.OutOfMemoryError
>>    <<No stacktrace available>>
>>
>> When I decrease number of threads to 3 or even 1 then search works.
>> How do many threads can affect to this exception?..
>>
>> I have 2 Gb of memory. So with one thread the process takes like
>> 1200-1300Mb.
>>
>> Andi Vajda suggested that "There may be overhead involved in having
>> multiple threads against a given index."
>>
>> Does anyone here have experience in handling big indexes with many
>> threads?


Yura Smolsky.



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message