lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yura Smolsky <i...@altervision.biz>
Subject big index and multi threaded IndexSearcher
Date Wed, 16 Feb 2005 20:04:56 GMT
Hello.

I use PyLucene, python port of Lucene.

I have problem about using big index (50Gb) with IndexSearcher
from many threads.
I use IndexSearcher from PyLucene's PythonThread. It's really a wrapper
around a Java/libgcj thread that python is tricked into thinking
it's one of its own.

The core of problem:
When I have many threads (more than 5) I receive this exception:
  File "/usr/lib/python2.4/site-packages/PyLucene.py", line 2241, in search
    def search(*args): return _PyLucene.Searcher_search(*args)
ValueError: java.lang.OutOfMemoryError
   <<No stacktrace available>>

When I decrease number of threads to 3 or even 1 then search works.
How do many threads can affect to this exception?..

I have 2 Gb of memory. So with one thread the process takes like
1200-1300Mb.

Andi Vajda suggested that "There may be overhead involved in having
multiple threads against a given index."

Does anyone here have experience in handling big indexes with many
threads?

Any ideas are appreciated.

Yura Smolsky.



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message