lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson" <erickerick...@gmail.com>
Subject Re: How many Searches is a Searcher Worth?
Date Fri, 06 Apr 2007 02:04:55 GMT
Having to put a counter in and close/open your searcher should not
be necessary. I'm afraid I'm not going to be very helpful, because....

I took your test case and made some very minor modifications
to make it run in an environment I happen to have lying around
(mostly, just instantiated the QueryParser with a field known
to be in my index). I generated a list of all the terms in that field
in my index for the queries.txt file and let it rip.

It executed 1,599,586 queries. Admittedly, they were for one and
only one term each. Each search had just a few hits because of
the peculiarities of my index, usually less than 5.

Just for yucks, I tried instantiating the QueryParser with "", and
tacking my index field in front of the terms I read from queries.txt.
Still no problem.

I'm running Lucene 2.1, JDK 1.5, and running the code from inside
Eclipse on a Windows XP box. I think the Eclipse JVM only
allocates 128M to processes by default......

So, the only thing that comes to mind is the extremely sticky wicket
of the environment you're running in. I wonder if it's possible you're
CLASSPATH is getting incompatible jar files somehow? Perhaps
some jars are coming from from an older JVM? I'm grasping at straws
here.....

We have production systems running complex queries for days at
a time and are processing far more than 40 queries, with no problems
so I'm pretty sure it's not Lucene, but something peculiar to your
environment.

If you're running a Windows machine, depending on how it was
installed, sometimes the java installation process
will put files in one of the Windows system directories so it's possible
that you're hitting something in there even though you aren't trying....

Kudos for writing that test program, by the way. It sure removed
a lot of variables from the situation.

Best
Erick@ItWorksOnMyMachine......

On 4/5/07, Craig W Conway <craigwconway@yahoo.com> wrote:
>
> So, forgetting the RMI stuff, I put together a test client very similar to
> the one in the book "Lucene in Action" page 182.
>
> The  client:
>
> 1. instantiates a IndexSearcher
> 2. loops through queries, searches, prints hit count, saves nothing
>
> I am only able to run through about 40 searches before I get an
> OutOfMemoryException.  JDK 1.5
>
> Because of this, I have put a counter in my search server to close and
> re-open the IndexSearcher after a certian number of searches. But this
> shouldn't be necessary right? What's eating up all the memory?
>
> Source code @ http://urbanmarsupial.com/share/TestLuceneMemory.java
>
> Any hints would be greatly appreciated!
>
> Thanks,
>
> Craig
>
> ----- Original Message ----
> From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
> To: java-user@lucene.apache.org
> Sent: Wednesday, April 4, 2007 10:39:50 AM
> Subject: Re: How many Searches is a Searcher Worth?
>
> No reason that I can think of.  What makes you think the problem is with
> the IndexSearcher?  Maybe it's something else in your code, for instance.
> Make sure you have the same version of Java on both ends of the
> call.  Also, Java 6 made our RMI calls a lot more stable than even 1.5.
>
> Otis
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share
>
> ----- Original Message ----
> From: Craig W Conway <craigwconway@yahoo.com>
> To: java-user@lucene.apache.org
> Sent: Wednesday, April 4, 2007 1:25:27 PM
> Subject: How many Searches is a Searcher Worth?
>
> I am using an RMI architecture for calling a remote service which uses an
> IndexSearcher in its own JVM. I am starting the service with the following
> provisions for memory allocation and garbage collection: java -server
> -Xmx1024m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>
> After about 1000 search calls I start to run out of memory, and I have to
> close and re-open the IndexSearcher, loosing any cached data and filters...
> Is there any reason why I shouldn't be able to use my IndexSearcher forever,
> until I want to close it?
>
> Thanks!
>
> Craig
>
>
>
>
>
>
> ____________________________________________________________________________________
> Now that's room service!  Choose from over 150,000 hotels
> in 45,000 destinations on Yahoo! Travel to find your fit.
> http://farechase.yahoo.com/promo-generic-14795097
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
>
>
>
>
>
> ____________________________________________________________________________________
> No need to miss a message. Get email on-the-go
> with Yahoo! Mail for Mobile. Get started.
> http://mobile.yahoo.com/mail

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message