lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastin <sebasmt...@gmail.com>
Subject Re: Java Heap Space -Out Of Memory Error
Date Wed, 19 Sep 2007 11:31:02 GMT

HI testn,
              I used to search now 3 folders of total size 1.5.GB.still it
consumes lot of memory for every search.i used to close all the IndexReader
once i finish the search.i optimize the FIles using Luke.when i set the
IndexSearcher  object as application level object,its not possible to see
the updated records.
could you guide me how to resolve this memory problem.

testn wrote:
> 
> As I mentioned, IndexReader is the one that holds the memory. You should
> explicitly close the underlying IndexReader to make sure that the reader
> releases the memory. 
> 
> 
> 
> Sebastin wrote:
>> 
>> Hi testn,
>>               Every IndexFolder is of size 1.5 GB of size,eventhough when
>> i used to Open and close the IndexSearcher it wont release the memory for
>> all the searches.
>>               When i set the IndexSearcher object as the Application
>> Scope object its not possile for me to see current day records.
>> 
>>                  Could you give me an Idea how to trigger out this
>> problem. 
>>       
>> 
>> testn wrote:
>>> 
>>> If you know that there are only 15 days of indexes you need to search
>>> on, you just need to open only the latest 15 indexes at a time right?
>>> You can simply create a wrapper that return MultiReader which you can
>>> cache for a while and close the oldest index once the date rolls.
>>> 
>>> 
>>> Sebastin wrote:
>>>> 
>>>> HI testn,
>>>> 
>>>> it gives performance improvement while optimizing the Index. 
>>>> 
>>>> Now i seprate the IndexStore on a daily basis.(ie) 
>>>> For Every Day it create a new Index store ,sep- 08-2007,sep-09-2007
>>>> like wise it will minimize the size of the IndexStore.could you give me
>>>> an idea on how to open every day folders for every search.
>>>> 
>>>> Query I use here is,
>>>> 
>>>> 9840836588 AND dateSc:[070901 TO 070910] 
>>>> 
>>>> 07---->year (2007)
>>>> 09---->month(september)
>>>> 01----->day
>>>> 
>>>> i restrict for 15 days that it is possible to search 15 days record in
>>>> my application.at a time 10 users aare going to search every store.is
>>>> there any other better way to improve the search performance to avoid
>>>> memory problem as well as speed of the search.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> testn wrote:
>>>>> 
>>>>> So did you see any improvement in performance?
>>>>> 
>>>>> Sebastin wrote:
>>>>>> 
>>>>>> It works finally .i use Lucene 2.2  in my application.thanks testn
>>>>>> and Mike
>>>>>> 
>>>>>> Michael McCandless-2 wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> It sounds like there may be a Lucene version mismatch?  When
Luke
>>>>>>> was used
>>>>>>> it was likely based on Lucene 2.2, but it sounds like an older
>>>>>>> version of
>>>>>>> Lucene is now being used to open the index?
>>>>>>> 
>>>>>>> Mike
>>>>>>> 
>>>>>>> "testn" <test1@doramail.com> wrote:
>>>>>>>> 
>>>>>>>> Should the file be "segments_8" and "segments.gen"? Why is
it
>>>>>>>> "Segment"?
>>>>>>>> The
>>>>>>>> case is different.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Sebastin wrote:
>>>>>>>> > 
>>>>>>>> > java.io.IoException:File Not Found- Segments  is the
error
>>>>>>>> message
>>>>>>>> > 
>>>>>>>> > testn wrote:
>>>>>>>> >> 
>>>>>>>> >> What is the error message? Probably Mike, Erick
or Yonik can
>>>>>>>> help you
>>>>>>>> >> better on this since I'm no one in index area.
>>>>>>>> >> 
>>>>>>>> >> Sebastin wrote:
>>>>>>>> >>> 
>>>>>>>> >>> HI testn,
>>>>>>>> >>>              1.I optimize the Large Indexes
of size 10 GB using
>>>>>>>> Luke.it
>>>>>>>> >>> optimize all the content into a single CFS file
and it
>>>>>>>> generates
>>>>>>>> >>> segments.gen and segments_8 file when i search
the item it
>>>>>>>> shows an
>>>>>>>> >>> error that segments file is not there.could
you help me in this 
>>>>>>>> >>> 
>>>>>>>> >>> testn wrote:
>>>>>>>> >>>> 
>>>>>>>> >>>> 1. You can close the searcher once you're
done. If you want to
>>>>>>>> reopen
>>>>>>>> >>>> the index, you can close and reopen only
the updated 3 readers
>>>>>>>> and keep
>>>>>>>> >>>> the 2 old indexreaders and reuse it. It
should reduce the time
>>>>>>>> to
>>>>>>>> >>>> reopen it.
>>>>>>>> >>>> 2. Make sure that you optimize it every
once in a while
>>>>>>>> >>>> 3. You might consider separating indices
in separated storage
>>>>>>>> and use
>>>>>>>> >>>> ParallelReader
>>>>>>>> >>>> 
>>>>>>>> >>>> 
>>>>>>>> >>>> 
>>>>>>>> >>>> Sebastin wrote:
>>>>>>>> >>>>> 
>>>>>>>> >>>>> The problem in my pplication are as
follows:
>>>>>>>> >>>>>                  1.I am not able to
see the updated records
>>>>>>>> in my
>>>>>>>> >>>>> index store because i instantiate 
>>>>>>>> >>>>> IndexReader and IndexSearcher class
once that is in the first
>>>>>>>> >>>>> search.further searches use the same
IndexReaders(5
>>>>>>>> Directories) and
>>>>>>>> >>>>> IndexSearcher with different queries.
>>>>>>>> >>>>> 
>>>>>>>> >>>>>                 2.My search is very
very slow First 2
>>>>>>>> Directories of
>>>>>>>> >>>>> size 10 GB each which are having old
index records and no
>>>>>>>> update in
>>>>>>>> >>>>> that remaining 3 Diretories are updated
every second.
>>>>>>>> >>>>> 
>>>>>>>> >>>>>                 3.i am Indexing 20 million
records per day so
>>>>>>>> the
>>>>>>>> >>>>> Index store gets growing and it makes
search very very
>>>>>>>> slower.
>>>>>>>> >>>>>  
>>>>>>>> >>>>>                4.I am using searcherOne
class as the global
>>>>>>>> >>>>> application helper class ,with the scope
as APPLICATION it
>>>>>>>> consists of
>>>>>>>> >>>>> one IndexReader and IndexSearcher get
set method which will
>>>>>>>> hold the
>>>>>>>> >>>>> IndexReader and IndexSearcher object
after the First
>>>>>>>> Search.it is used
>>>>>>>> >>>>> for all other searches.
>>>>>>>> >>>>> 
>>>>>>>> >>>>>               5.I am using Lucene 2.2.0
version, in a WEB
>>>>>>>> Application
>>>>>>>> >>>>> which index 15 fields per document and
Index 5 Fieds,store 10
>>>>>>>> Fields.i
>>>>>>>> >>>>> am not using any sort in my query.for
a single query upto the
>>>>>>>> maximum
>>>>>>>> >>>>> it fetches 600 records from the index
store(5 direcories)    
>>>>>>>> >>>>>                 
>>>>>>>> >>>>> 
>>>>>>>> >>>>> hossman wrote:
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> : I set IndexSearcher as the application
Object after the
>>>>>>>> first
>>>>>>>> >>>>>> search.
>>>>>>>> >>>>>> 	...
>>>>>>>> >>>>>> : how can i reconstruct the new
IndexSearcher for every hour
>>>>>>>> to see
>>>>>>>> >>>>>> the
>>>>>>>> >>>>>> : updated records .
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> i'm confused ... my understanding
based on the comments you
>>>>>>>> made
>>>>>>>> >>>>>> below 
>>>>>>>> >>>>>> (in an earlier message) was that
you already *were*
>>>>>>>> constructing a
>>>>>>>> >>>>>> new  
>>>>>>>> >>>>>> IndexSearcher once an hour -- but
every time you do that,
>>>>>>>> your memory 
>>>>>>>> >>>>>> usage grows, and and that sometimes
you got OOM Errors.
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> if that's not what you said, then
i think you need to
>>>>>>>> explain, in
>>>>>>>> >>>>>> detail, 
>>>>>>>> >>>>>> in one message, exactly what your
problem is.  And don't
>>>>>>>> assume we 
>>>>>>>> >>>>>> understand anything -- tell us *EVERYTHING*
(like, for
>>>>>>>> example, what
>>>>>>>> >>>>>> the 
>>>>>>>> >>>>>> word "crore" means, how "searcherOne"
is implemented, and
>>>>>>>> the answer
>>>>>>>> >>>>>> to 
>>>>>>>> >>>>>> the specfic question i asked in
my last message: does your
>>>>>>>> >>>>>> application, 
>>>>>>>> >>>>>> contain anywhere in it, any code
that will close anything
>>>>>>>> >>>>>> (IndexSearchers 
>>>>>>>> >>>>>> or IndexReaders) ?
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> : > : I use StandardAnalyzer.the
records daily ranges from 5
>>>>>>>> crore to
>>>>>>>> >>>>>> 6 crore.
>>>>>>>> >>>>>> : > for
>>>>>>>> >>>>>> : > : every second i am updating
my Index. i instantiate
>>>>>>>> >>>>>> IndexSearcher object
>>>>>>>> >>>>>> : > one
>>>>>>>> >>>>>> : > : time for all the searches.
for an hour can i see the
>>>>>>>> updated
>>>>>>>> >>>>>> records in
>>>>>>>> >>>>>> : > the
>>>>>>>> >>>>>> : > : indexstore by reinstantiating
IndexSearcher object.but
>>>>>>>> the
>>>>>>>> >>>>>> problem when
>>>>>>>> >>>>>> : > i
>>>>>>>> >>>>>> : > : reinstantiate IndexSearcher
,my RAM memory gets
>>>>>>>> appended.is
>>>>>>>> >>>>>> there any
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> : > IndexSearcher are you explicitly
closing both the old
>>>>>>>> >>>>>> IndexSearcher as 
>>>>>>>> >>>>>> : > well as all of 4 of those
old IndexReaders and the
>>>>>>>> MultiReader?
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> -Hoss
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> >>>>>> To unsubscribe, e-mail:
>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>> >>>>>> For additional commands, e-mail:
>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> 
>>>>>>>> >>>>>> 
>>>>>>>> >>>>> 
>>>>>>>> >>>>> 
>>>>>>>> >>>> 
>>>>>>>> >>>> 
>>>>>>>> >>> 
>>>>>>>> >>> 
>>>>>>>> >> 
>>>>>>>> >> 
>>>>>>>> > 
>>>>>>>> > 
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12655880
>>>>>>>> Sent from the Lucene - Java Users mailing list archive at
>>>>>>>> Nabble.com.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>> 
>>>>>>> 
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12775620
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message