lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastin <sebasmt...@gmail.com>
Subject Re: Java Heap Space -Out Of Memory Error
Date Sat, 15 Sep 2007 05:19:38 GMT

HI testn,

it gives performance improvement while optimizing the Index. 

Now i seprate the IndexStore on a daily basis.(ie) 
For Every Day it create a new Index store ,sep- 08-2007,sep-09-2007 like
wise it will minimize the size of the IndexStore.could you give me an idea
on how to open every day folders for every search.

Query I use here is,

9840836588 AND dateSc:[070901 TO 070910] 

07---->year (2007)
09---->month(september)
01----->day

i restrict for 15 days that it is possible to search 15 days record in my
application.at a time 10 users aare going to search every store.is there any
other better way to improve the search performance to avoid memory problem
as well as speed of the search.






testn wrote:
> 
> So did you see any improvement in performance?
> 
> Sebastin wrote:
>> 
>> It works finally .i use Lucene 2.2  in my application.thanks testn and
>> Mike
>> 
>> Michael McCandless-2 wrote:
>>> 
>>> 
>>> It sounds like there may be a Lucene version mismatch?  When Luke was
>>> used
>>> it was likely based on Lucene 2.2, but it sounds like an older version
>>> of
>>> Lucene is now being used to open the index?
>>> 
>>> Mike
>>> 
>>> "testn" <test1@doramail.com> wrote:
>>>> 
>>>> Should the file be "segments_8" and "segments.gen"? Why is it
>>>> "Segment"?
>>>> The
>>>> case is different.
>>>> 
>>>> 
>>>> Sebastin wrote:
>>>> > 
>>>> > java.io.IoException:File Not Found- Segments  is the error message
>>>> > 
>>>> > testn wrote:
>>>> >> 
>>>> >> What is the error message? Probably Mike, Erick or Yonik can help
>>>> you
>>>> >> better on this since I'm no one in index area.
>>>> >> 
>>>> >> Sebastin wrote:
>>>> >>> 
>>>> >>> HI testn,
>>>> >>>              1.I optimize the Large Indexes of size 10 GB using
>>>> Luke.it
>>>> >>> optimize all the content into a single CFS file and it generates
>>>> >>> segments.gen and segments_8 file when i search the item it shows
an
>>>> >>> error that segments file is not there.could you help me in this

>>>> >>> 
>>>> >>> testn wrote:
>>>> >>>> 
>>>> >>>> 1. You can close the searcher once you're done. If you want
to
>>>> reopen
>>>> >>>> the index, you can close and reopen only the updated 3 readers
and
>>>> keep
>>>> >>>> the 2 old indexreaders and reuse it. It should reduce the
time to
>>>> >>>> reopen it.
>>>> >>>> 2. Make sure that you optimize it every once in a while
>>>> >>>> 3. You might consider separating indices in separated storage
and
>>>> use
>>>> >>>> ParallelReader
>>>> >>>> 
>>>> >>>> 
>>>> >>>> 
>>>> >>>> Sebastin wrote:
>>>> >>>>> 
>>>> >>>>> The problem in my pplication are as follows:
>>>> >>>>>                  1.I am not able to see the updated
records in my
>>>> >>>>> index store because i instantiate 
>>>> >>>>> IndexReader and IndexSearcher class once that is in
the first
>>>> >>>>> search.further searches use the same IndexReaders(5
Directories)
>>>> and
>>>> >>>>> IndexSearcher with different queries.
>>>> >>>>> 
>>>> >>>>>                 2.My search is very very slow First
2 Directories
>>>> of
>>>> >>>>> size 10 GB each which are having old index records and
no update
>>>> in
>>>> >>>>> that remaining 3 Diretories are updated every second.
>>>> >>>>> 
>>>> >>>>>                 3.i am Indexing 20 million records per
day so the
>>>> >>>>> Index store gets growing and it makes search very very
slower.
>>>> >>>>>  
>>>> >>>>>                4.I am using searcherOne class as the
global
>>>> >>>>> application helper class ,with the scope as APPLICATION
it
>>>> consists of
>>>> >>>>> one IndexReader and IndexSearcher get set method which
will hold
>>>> the
>>>> >>>>> IndexReader and IndexSearcher object after the First
Search.it is
>>>> used
>>>> >>>>> for all other searches.
>>>> >>>>> 
>>>> >>>>>               5.I am using Lucene 2.2.0 version, in
a WEB
>>>> Application
>>>> >>>>> which index 15 fields per document and Index 5 Fieds,store
10
>>>> Fields.i
>>>> >>>>> am not using any sort in my query.for a single query
upto the
>>>> maximum
>>>> >>>>> it fetches 600 records from the index store(5 direcories)
   
>>>> >>>>>                 
>>>> >>>>> 
>>>> >>>>> hossman wrote:
>>>> >>>>>> 
>>>> >>>>>> 
>>>> >>>>>> : I set IndexSearcher as the application Object
after the first
>>>> >>>>>> search.
>>>> >>>>>> 	...
>>>> >>>>>> : how can i reconstruct the new IndexSearcher for
every hour to
>>>> see
>>>> >>>>>> the
>>>> >>>>>> : updated records .
>>>> >>>>>> 
>>>> >>>>>> i'm confused ... my understanding based on the comments
you made
>>>> >>>>>> below 
>>>> >>>>>> (in an earlier message) was that you already *were*
constructing
>>>> a
>>>> >>>>>> new  
>>>> >>>>>> IndexSearcher once an hour -- but every time you
do that, your
>>>> memory 
>>>> >>>>>> usage grows, and and that sometimes you got OOM
Errors.
>>>> >>>>>> 
>>>> >>>>>> if that's not what you said, then i think you need
to explain,
>>>> in
>>>> >>>>>> detail, 
>>>> >>>>>> in one message, exactly what your problem is.  And
don't assume
>>>> we 
>>>> >>>>>> understand anything -- tell us *EVERYTHING* (like,
for example,
>>>> what
>>>> >>>>>> the 
>>>> >>>>>> word "crore" means, how "searcherOne" is implemented,
and the
>>>> answer
>>>> >>>>>> to 
>>>> >>>>>> the specfic question i asked in my last message:
does your
>>>> >>>>>> application, 
>>>> >>>>>> contain anywhere in it, any code that will close
anything
>>>> >>>>>> (IndexSearchers 
>>>> >>>>>> or IndexReaders) ?
>>>> >>>>>> 
>>>> >>>>>> 
>>>> >>>>>> : > : I use StandardAnalyzer.the records daily
ranges from 5
>>>> crore to
>>>> >>>>>> 6 crore.
>>>> >>>>>> : > for
>>>> >>>>>> : > : every second i am updating my Index. i
instantiate
>>>> >>>>>> IndexSearcher object
>>>> >>>>>> : > one
>>>> >>>>>> : > : time for all the searches. for an hour
can i see the
>>>> updated
>>>> >>>>>> records in
>>>> >>>>>> : > the
>>>> >>>>>> : > : indexstore by reinstantiating IndexSearcher
object.but the
>>>> >>>>>> problem when
>>>> >>>>>> : > i
>>>> >>>>>> : > : reinstantiate IndexSearcher ,my RAM memory
gets
>>>> appended.is
>>>> >>>>>> there any
>>>> >>>>>> 
>>>> >>>>>> 
>>>> >>>>>> : > IndexSearcher are you explicitly closing
both the old
>>>> >>>>>> IndexSearcher as 
>>>> >>>>>> : > well as all of 4 of those old IndexReaders
and the
>>>> MultiReader?
>>>> >>>>>> 
>>>> >>>>>> 
>>>> >>>>>> 
>>>> >>>>>> 
>>>> >>>>>> -Hoss
>>>> >>>>>> 
>>>> >>>>>> 
>>>> >>>>>>
>>>> ---------------------------------------------------------------------
>>>> >>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> >>>>>> For additional commands, e-mail:
>>>> java-user-help@lucene.apache.org
>>>> >>>>>> 
>>>> >>>>>> 
>>>> >>>>>> 
>>>> >>>>> 
>>>> >>>>> 
>>>> >>>> 
>>>> >>>> 
>>>> >>> 
>>>> >>> 
>>>> >> 
>>>> >> 
>>>> > 
>>>> > 
>>>> 
>>>> -- 
>>>> View this message in context:
>>>> http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12655880
>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12687410
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message