lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From testn <te...@doramail.com>
Subject Re: Java Heap Space -Out Of Memory Error
Date Sat, 15 Sep 2007 13:01:40 GMT

If you know that there are only 15 days of indexes you need to search on, you
just need to open only the latest 15 indexes at a time right? You can simply
create a wrapper that return MultiReader which you can cache for a while and
close the oldest index once the date rolls.


Sebastin wrote:
> 
> HI testn,
> 
> it gives performance improvement while optimizing the Index. 
> 
> Now i seprate the IndexStore on a daily basis.(ie) 
> For Every Day it create a new Index store ,sep- 08-2007,sep-09-2007 like
> wise it will minimize the size of the IndexStore.could you give me an idea
> on how to open every day folders for every search.
> 
> Query I use here is,
> 
> 9840836588 AND dateSc:[070901 TO 070910] 
> 
> 07---->year (2007)
> 09---->month(september)
> 01----->day
> 
> i restrict for 15 days that it is possible to search 15 days record in my
> application.at a time 10 users aare going to search every store.is there
> any other better way to improve the search performance to avoid memory
> problem as well as speed of the search.
> 
> 
> 
> 
> 
> 
> testn wrote:
>> 
>> So did you see any improvement in performance?
>> 
>> Sebastin wrote:
>>> 
>>> It works finally .i use Lucene 2.2  in my application.thanks testn and
>>> Mike
>>> 
>>> Michael McCandless-2 wrote:
>>>> 
>>>> 
>>>> It sounds like there may be a Lucene version mismatch?  When Luke was
>>>> used
>>>> it was likely based on Lucene 2.2, but it sounds like an older version
>>>> of
>>>> Lucene is now being used to open the index?
>>>> 
>>>> Mike
>>>> 
>>>> "testn" <test1@doramail.com> wrote:
>>>>> 
>>>>> Should the file be "segments_8" and "segments.gen"? Why is it
>>>>> "Segment"?
>>>>> The
>>>>> case is different.
>>>>> 
>>>>> 
>>>>> Sebastin wrote:
>>>>> > 
>>>>> > java.io.IoException:File Not Found- Segments  is the error message
>>>>> > 
>>>>> > testn wrote:
>>>>> >> 
>>>>> >> What is the error message? Probably Mike, Erick or Yonik can
help
>>>>> you
>>>>> >> better on this since I'm no one in index area.
>>>>> >> 
>>>>> >> Sebastin wrote:
>>>>> >>> 
>>>>> >>> HI testn,
>>>>> >>>              1.I optimize the Large Indexes of size 10 GB
using
>>>>> Luke.it
>>>>> >>> optimize all the content into a single CFS file and it generates
>>>>> >>> segments.gen and segments_8 file when i search the item
it shows
>>>>> an
>>>>> >>> error that segments file is not there.could you help me
in this 
>>>>> >>> 
>>>>> >>> testn wrote:
>>>>> >>>> 
>>>>> >>>> 1. You can close the searcher once you're done. If you
want to
>>>>> reopen
>>>>> >>>> the index, you can close and reopen only the updated
3 readers
>>>>> and keep
>>>>> >>>> the 2 old indexreaders and reuse it. It should reduce
the time to
>>>>> >>>> reopen it.
>>>>> >>>> 2. Make sure that you optimize it every once in a while
>>>>> >>>> 3. You might consider separating indices in separated
storage and
>>>>> use
>>>>> >>>> ParallelReader
>>>>> >>>> 
>>>>> >>>> 
>>>>> >>>> 
>>>>> >>>> Sebastin wrote:
>>>>> >>>>> 
>>>>> >>>>> The problem in my pplication are as follows:
>>>>> >>>>>                  1.I am not able to see the updated
records in
>>>>> my
>>>>> >>>>> index store because i instantiate 
>>>>> >>>>> IndexReader and IndexSearcher class once that is
in the first
>>>>> >>>>> search.further searches use the same IndexReaders(5
Directories)
>>>>> and
>>>>> >>>>> IndexSearcher with different queries.
>>>>> >>>>> 
>>>>> >>>>>                 2.My search is very very slow First
2
>>>>> Directories of
>>>>> >>>>> size 10 GB each which are having old index records
and no update
>>>>> in
>>>>> >>>>> that remaining 3 Diretories are updated every second.
>>>>> >>>>> 
>>>>> >>>>>                 3.i am Indexing 20 million records
per day so
>>>>> the
>>>>> >>>>> Index store gets growing and it makes search very
very slower.
>>>>> >>>>>  
>>>>> >>>>>                4.I am using searcherOne class as
the global
>>>>> >>>>> application helper class ,with the scope as APPLICATION
it
>>>>> consists of
>>>>> >>>>> one IndexReader and IndexSearcher get set method
which will hold
>>>>> the
>>>>> >>>>> IndexReader and IndexSearcher object after the First
Search.it
>>>>> is used
>>>>> >>>>> for all other searches.
>>>>> >>>>> 
>>>>> >>>>>               5.I am using Lucene 2.2.0 version,
in a WEB
>>>>> Application
>>>>> >>>>> which index 15 fields per document and Index 5 Fieds,store
10
>>>>> Fields.i
>>>>> >>>>> am not using any sort in my query.for a single query
upto the
>>>>> maximum
>>>>> >>>>> it fetches 600 records from the index store(5 direcories)
   
>>>>> >>>>>                 
>>>>> >>>>> 
>>>>> >>>>> hossman wrote:
>>>>> >>>>>> 
>>>>> >>>>>> 
>>>>> >>>>>> : I set IndexSearcher as the application Object
after the first
>>>>> >>>>>> search.
>>>>> >>>>>> 	...
>>>>> >>>>>> : how can i reconstruct the new IndexSearcher
for every hour to
>>>>> see
>>>>> >>>>>> the
>>>>> >>>>>> : updated records .
>>>>> >>>>>> 
>>>>> >>>>>> i'm confused ... my understanding based on the
comments you
>>>>> made
>>>>> >>>>>> below 
>>>>> >>>>>> (in an earlier message) was that you already
*were*
>>>>> constructing a
>>>>> >>>>>> new  
>>>>> >>>>>> IndexSearcher once an hour -- but every time
you do that, your
>>>>> memory 
>>>>> >>>>>> usage grows, and and that sometimes you got
OOM Errors.
>>>>> >>>>>> 
>>>>> >>>>>> if that's not what you said, then i think you
need to explain,
>>>>> in
>>>>> >>>>>> detail, 
>>>>> >>>>>> in one message, exactly what your problem is.
 And don't assume
>>>>> we 
>>>>> >>>>>> understand anything -- tell us *EVERYTHING*
(like, for example,
>>>>> what
>>>>> >>>>>> the 
>>>>> >>>>>> word "crore" means, how "searcherOne" is implemented,
and the
>>>>> answer
>>>>> >>>>>> to 
>>>>> >>>>>> the specfic question i asked in my last message:
does your
>>>>> >>>>>> application, 
>>>>> >>>>>> contain anywhere in it, any code that will close
anything
>>>>> >>>>>> (IndexSearchers 
>>>>> >>>>>> or IndexReaders) ?
>>>>> >>>>>> 
>>>>> >>>>>> 
>>>>> >>>>>> : > : I use StandardAnalyzer.the records
daily ranges from 5
>>>>> crore to
>>>>> >>>>>> 6 crore.
>>>>> >>>>>> : > for
>>>>> >>>>>> : > : every second i am updating my Index.
i instantiate
>>>>> >>>>>> IndexSearcher object
>>>>> >>>>>> : > one
>>>>> >>>>>> : > : time for all the searches. for an hour
can i see the
>>>>> updated
>>>>> >>>>>> records in
>>>>> >>>>>> : > the
>>>>> >>>>>> : > : indexstore by reinstantiating IndexSearcher
object.but
>>>>> the
>>>>> >>>>>> problem when
>>>>> >>>>>> : > i
>>>>> >>>>>> : > : reinstantiate IndexSearcher ,my RAM
memory gets
>>>>> appended.is
>>>>> >>>>>> there any
>>>>> >>>>>> 
>>>>> >>>>>> 
>>>>> >>>>>> : > IndexSearcher are you explicitly closing
both the old
>>>>> >>>>>> IndexSearcher as 
>>>>> >>>>>> : > well as all of 4 of those old IndexReaders
and the
>>>>> MultiReader?
>>>>> >>>>>> 
>>>>> >>>>>> 
>>>>> >>>>>> 
>>>>> >>>>>> 
>>>>> >>>>>> -Hoss
>>>>> >>>>>> 
>>>>> >>>>>> 
>>>>> >>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> >>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> >>>>>> For additional commands, e-mail:
>>>>> java-user-help@lucene.apache.org
>>>>> >>>>>> 
>>>>> >>>>>> 
>>>>> >>>>>> 
>>>>> >>>>> 
>>>>> >>>>> 
>>>>> >>>> 
>>>>> >>>> 
>>>>> >>> 
>>>>> >>> 
>>>>> >> 
>>>>> >> 
>>>>> > 
>>>>> > 
>>>>> 
>>>>> -- 
>>>>> View this message in context:
>>>>> http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12655880
>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12690100
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message