lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Hatcher <e...@ehatchersolutions.com>
Subject Re: big index and multi threaded IndexSearcher
Date Wed, 16 Feb 2005 20:14:26 GMT
Are you using multiple IndexSearcher instances?    Or only one and 
sharing it across multiple threads?

If using a single shared IndexSearcher instance doesn't help, it may be 
beneficial to port your code to Java and try it there.

I'm just now getting into PyLucene myself - building a demo for a Unix 
User's Group presentation I'm giving.

	Erik


On Feb 16, 2005, at 3:04 PM, Yura Smolsky wrote:

> Hello.
>
> I use PyLucene, python port of Lucene.
>
> I have problem about using big index (50Gb) with IndexSearcher
> from many threads.
> I use IndexSearcher from PyLucene's PythonThread. It's really a wrapper
> around a Java/libgcj thread that python is tricked into thinking
> it's one of its own.
>
> The core of problem:
> When I have many threads (more than 5) I receive this exception:
>   File "/usr/lib/python2.4/site-packages/PyLucene.py", line 2241, in 
> search
>     def search(*args): return _PyLucene.Searcher_search(*args)
> ValueError: java.lang.OutOfMemoryError
>    <<No stacktrace available>>
>
> When I decrease number of threads to 3 or even 1 then search works.
> How do many threads can affect to this exception?..
>
> I have 2 Gb of memory. So with one thread the process takes like
> 1200-1300Mb.
>
> Andi Vajda suggested that "There may be overhead involved in having
> multiple threads against a given index."
>
> Does anyone here have experience in handling big indexes with many
> threads?
>
> Any ideas are appreciated.
>
> Yura Smolsky.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message