lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From testn <te...@doramail.com>
Subject Re: Java Heap Space -Out Of Memory Error
Date Mon, 17 Sep 2007 12:55:10 GMT

As I mentioned, IndexReader is the one that holds the memory. You should
explicitly close the underlying IndexReader to make sure that the reader
releases the memory. 



Sebastin wrote:
> 
> Hi testn,
>               Every IndexFolder is of size 1.5 GB of size,eventhough when
> i used to Open and close the IndexSearcher it wont release the memory for
> all the searches.
>               When i set the IndexSearcher object as the Application Scope
> object its not possile for me to see current day records.
> 
>                  Could you give me an Idea how to trigger out this
> problem. 
>       
> 
> testn wrote:
>> 
>> If you know that there are only 15 days of indexes you need to search on,
>> you just need to open only the latest 15 indexes at a time right? You can
>> simply create a wrapper that return MultiReader which you can cache for a
>> while and close the oldest index once the date rolls.
>> 
>> 
>> Sebastin wrote:
>>> 
>>> HI testn,
>>> 
>>> it gives performance improvement while optimizing the Index. 
>>> 
>>> Now i seprate the IndexStore on a daily basis.(ie) 
>>> For Every Day it create a new Index store ,sep- 08-2007,sep-09-2007 like
>>> wise it will minimize the size of the IndexStore.could you give me an
>>> idea on how to open every day folders for every search.
>>> 
>>> Query I use here is,
>>> 
>>> 9840836588 AND dateSc:[070901 TO 070910] 
>>> 
>>> 07---->year (2007)
>>> 09---->month(september)
>>> 01----->day
>>> 
>>> i restrict for 15 days that it is possible to search 15 days record in
>>> my application.at a time 10 users aare going to search every store.is
>>> there any other better way to improve the search performance to avoid
>>> memory problem as well as speed of the search.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> testn wrote:
>>>> 
>>>> So did you see any improvement in performance?
>>>> 
>>>> Sebastin wrote:
>>>>> 
>>>>> It works finally .i use Lucene 2.2  in my application.thanks testn and
>>>>> Mike
>>>>> 
>>>>> Michael McCandless-2 wrote:
>>>>>> 
>>>>>> 
>>>>>> It sounds like there may be a Lucene version mismatch?  When Luke
was
>>>>>> used
>>>>>> it was likely based on Lucene 2.2, but it sounds like an older
>>>>>> version of
>>>>>> Lucene is now being used to open the index?
>>>>>> 
>>>>>> Mike
>>>>>> 
>>>>>> "testn" <test1@doramail.com> wrote:
>>>>>>> 
>>>>>>> Should the file be "segments_8" and "segments.gen"? Why is it
>>>>>>> "Segment"?
>>>>>>> The
>>>>>>> case is different.
>>>>>>> 
>>>>>>> 
>>>>>>> Sebastin wrote:
>>>>>>> > 
>>>>>>> > java.io.IoException:File Not Found- Segments  is the error
message
>>>>>>> > 
>>>>>>> > testn wrote:
>>>>>>> >> 
>>>>>>> >> What is the error message? Probably Mike, Erick or Yonik
can help
>>>>>>> you
>>>>>>> >> better on this since I'm no one in index area.
>>>>>>> >> 
>>>>>>> >> Sebastin wrote:
>>>>>>> >>> 
>>>>>>> >>> HI testn,
>>>>>>> >>>              1.I optimize the Large Indexes of size
10 GB using
>>>>>>> Luke.it
>>>>>>> >>> optimize all the content into a single CFS file
and it generates
>>>>>>> >>> segments.gen and segments_8 file when i search the
item it shows
>>>>>>> an
>>>>>>> >>> error that segments file is not there.could you
help me in this 
>>>>>>> >>> 
>>>>>>> >>> testn wrote:
>>>>>>> >>>> 
>>>>>>> >>>> 1. You can close the searcher once you're done.
If you want to
>>>>>>> reopen
>>>>>>> >>>> the index, you can close and reopen only the
updated 3 readers
>>>>>>> and keep
>>>>>>> >>>> the 2 old indexreaders and reuse it. It should
reduce the time
>>>>>>> to
>>>>>>> >>>> reopen it.
>>>>>>> >>>> 2. Make sure that you optimize it every once
in a while
>>>>>>> >>>> 3. You might consider separating indices in
separated storage
>>>>>>> and use
>>>>>>> >>>> ParallelReader
>>>>>>> >>>> 
>>>>>>> >>>> 
>>>>>>> >>>> 
>>>>>>> >>>> Sebastin wrote:
>>>>>>> >>>>> 
>>>>>>> >>>>> The problem in my pplication are as follows:
>>>>>>> >>>>>                  1.I am not able to see
the updated records in
>>>>>>> my
>>>>>>> >>>>> index store because i instantiate 
>>>>>>> >>>>> IndexReader and IndexSearcher class once
that is in the first
>>>>>>> >>>>> search.further searches use the same IndexReaders(5
>>>>>>> Directories) and
>>>>>>> >>>>> IndexSearcher with different queries.
>>>>>>> >>>>> 
>>>>>>> >>>>>                 2.My search is very very
slow First 2
>>>>>>> Directories of
>>>>>>> >>>>> size 10 GB each which are having old index
records and no
>>>>>>> update in
>>>>>>> >>>>> that remaining 3 Diretories are updated
every second.
>>>>>>> >>>>> 
>>>>>>> >>>>>                 3.i am Indexing 20 million
records per day so
>>>>>>> the
>>>>>>> >>>>> Index store gets growing and it makes search
very very slower.
>>>>>>> >>>>>  
>>>>>>> >>>>>                4.I am using searcherOne
class as the global
>>>>>>> >>>>> application helper class ,with the scope
as APPLICATION it
>>>>>>> consists of
>>>>>>> >>>>> one IndexReader and IndexSearcher get set
method which will
>>>>>>> hold the
>>>>>>> >>>>> IndexReader and IndexSearcher object after
the First Search.it
>>>>>>> is used
>>>>>>> >>>>> for all other searches.
>>>>>>> >>>>> 
>>>>>>> >>>>>               5.I am using Lucene 2.2.0
version, in a WEB
>>>>>>> Application
>>>>>>> >>>>> which index 15 fields per document and Index
5 Fieds,store 10
>>>>>>> Fields.i
>>>>>>> >>>>> am not using any sort in my query.for a
single query upto the
>>>>>>> maximum
>>>>>>> >>>>> it fetches 600 records from the index store(5
direcories)    
>>>>>>> >>>>>                 
>>>>>>> >>>>> 
>>>>>>> >>>>> hossman wrote:
>>>>>>> >>>>>> 
>>>>>>> >>>>>> 
>>>>>>> >>>>>> : I set IndexSearcher as the application
Object after the
>>>>>>> first
>>>>>>> >>>>>> search.
>>>>>>> >>>>>> 	...
>>>>>>> >>>>>> : how can i reconstruct the new IndexSearcher
for every hour
>>>>>>> to see
>>>>>>> >>>>>> the
>>>>>>> >>>>>> : updated records .
>>>>>>> >>>>>> 
>>>>>>> >>>>>> i'm confused ... my understanding based
on the comments you
>>>>>>> made
>>>>>>> >>>>>> below 
>>>>>>> >>>>>> (in an earlier message) was that you
already *were*
>>>>>>> constructing a
>>>>>>> >>>>>> new  
>>>>>>> >>>>>> IndexSearcher once an hour -- but every
time you do that,
>>>>>>> your memory 
>>>>>>> >>>>>> usage grows, and and that sometimes
you got OOM Errors.
>>>>>>> >>>>>> 
>>>>>>> >>>>>> if that's not what you said, then i
think you need to
>>>>>>> explain, in
>>>>>>> >>>>>> detail, 
>>>>>>> >>>>>> in one message, exactly what your problem
is.  And don't
>>>>>>> assume we 
>>>>>>> >>>>>> understand anything -- tell us *EVERYTHING*
(like, for
>>>>>>> example, what
>>>>>>> >>>>>> the 
>>>>>>> >>>>>> word "crore" means, how "searcherOne"
is implemented, and the
>>>>>>> answer
>>>>>>> >>>>>> to 
>>>>>>> >>>>>> the specfic question i asked in my last
message: does your
>>>>>>> >>>>>> application, 
>>>>>>> >>>>>> contain anywhere in it, any code that
will close anything
>>>>>>> >>>>>> (IndexSearchers 
>>>>>>> >>>>>> or IndexReaders) ?
>>>>>>> >>>>>> 
>>>>>>> >>>>>> 
>>>>>>> >>>>>> : > : I use StandardAnalyzer.the
records daily ranges from 5
>>>>>>> crore to
>>>>>>> >>>>>> 6 crore.
>>>>>>> >>>>>> : > for
>>>>>>> >>>>>> : > : every second i am updating
my Index. i instantiate
>>>>>>> >>>>>> IndexSearcher object
>>>>>>> >>>>>> : > one
>>>>>>> >>>>>> : > : time for all the searches.
for an hour can i see the
>>>>>>> updated
>>>>>>> >>>>>> records in
>>>>>>> >>>>>> : > the
>>>>>>> >>>>>> : > : indexstore by reinstantiating
IndexSearcher object.but
>>>>>>> the
>>>>>>> >>>>>> problem when
>>>>>>> >>>>>> : > i
>>>>>>> >>>>>> : > : reinstantiate IndexSearcher
,my RAM memory gets
>>>>>>> appended.is
>>>>>>> >>>>>> there any
>>>>>>> >>>>>> 
>>>>>>> >>>>>> 
>>>>>>> >>>>>> : > IndexSearcher are you explicitly
closing both the old
>>>>>>> >>>>>> IndexSearcher as 
>>>>>>> >>>>>> : > well as all of 4 of those old
IndexReaders and the
>>>>>>> MultiReader?
>>>>>>> >>>>>> 
>>>>>>> >>>>>> 
>>>>>>> >>>>>> 
>>>>>>> >>>>>> 
>>>>>>> >>>>>> -Hoss
>>>>>>> >>>>>> 
>>>>>>> >>>>>> 
>>>>>>> >>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >>>>>> To unsubscribe, e-mail:
>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>> >>>>>> For additional commands, e-mail:
>>>>>>> java-user-help@lucene.apache.org
>>>>>>> >>>>>> 
>>>>>>> >>>>>> 
>>>>>>> >>>>>> 
>>>>>>> >>>>> 
>>>>>>> >>>>> 
>>>>>>> >>>> 
>>>>>>> >>>> 
>>>>>>> >>> 
>>>>>>> >>> 
>>>>>>> >> 
>>>>>>> >> 
>>>>>>> > 
>>>>>>> > 
>>>>>>> 
>>>>>>> -- 
>>>>>>> View this message in context:
>>>>>>> http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12655880
>>>>>>> Sent from the Lucene - Java Users mailing list archive at
>>>>>>> Nabble.com.
>>>>>>> 
>>>>>>> 
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>> 
>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12735206
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message