lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastin <sebasmt...@gmail.com>
Subject Re: Java Heap Space -Out Of Memory Error
Date Mon, 17 Sep 2007 04:03:01 GMT

Hi testn,
              Every IndexFolder is of size 1.5 GB of size,eventhough when i
used to Open and close the IndexSearcher it wont release the memory for all
the searches.
              When i set the IndexSearcher object as the Application Scope
object its not possile for me to see current day records.

                 Could you give me an Idea how to trigger out this problem. 
      

testn wrote:
> 
> If you know that there are only 15 days of indexes you need to search on,
> you just need to open only the latest 15 indexes at a time right? You can
> simply create a wrapper that return MultiReader which you can cache for a
> while and close the oldest index once the date rolls.
> 
> 
> Sebastin wrote:
>> 
>> HI testn,
>> 
>> it gives performance improvement while optimizing the Index. 
>> 
>> Now i seprate the IndexStore on a daily basis.(ie) 
>> For Every Day it create a new Index store ,sep- 08-2007,sep-09-2007 like
>> wise it will minimize the size of the IndexStore.could you give me an
>> idea on how to open every day folders for every search.
>> 
>> Query I use here is,
>> 
>> 9840836588 AND dateSc:[070901 TO 070910] 
>> 
>> 07---->year (2007)
>> 09---->month(september)
>> 01----->day
>> 
>> i restrict for 15 days that it is possible to search 15 days record in my
>> application.at a time 10 users aare going to search every store.is there
>> any other better way to improve the search performance to avoid memory
>> problem as well as speed of the search.
>> 
>> 
>> 
>> 
>> 
>> 
>> testn wrote:
>>> 
>>> So did you see any improvement in performance?
>>> 
>>> Sebastin wrote:
>>>> 
>>>> It works finally .i use Lucene 2.2  in my application.thanks testn and
>>>> Mike
>>>> 
>>>> Michael McCandless-2 wrote:
>>>>> 
>>>>> 
>>>>> It sounds like there may be a Lucene version mismatch?  When Luke was
>>>>> used
>>>>> it was likely based on Lucene 2.2, but it sounds like an older version
>>>>> of
>>>>> Lucene is now being used to open the index?
>>>>> 
>>>>> Mike
>>>>> 
>>>>> "testn" <test1@doramail.com> wrote:
>>>>>> 
>>>>>> Should the file be "segments_8" and "segments.gen"? Why is it
>>>>>> "Segment"?
>>>>>> The
>>>>>> case is different.
>>>>>> 
>>>>>> 
>>>>>> Sebastin wrote:
>>>>>> > 
>>>>>> > java.io.IoException:File Not Found- Segments  is the error message
>>>>>> > 
>>>>>> > testn wrote:
>>>>>> >> 
>>>>>> >> What is the error message? Probably Mike, Erick or Yonik
can help
>>>>>> you
>>>>>> >> better on this since I'm no one in index area.
>>>>>> >> 
>>>>>> >> Sebastin wrote:
>>>>>> >>> 
>>>>>> >>> HI testn,
>>>>>> >>>              1.I optimize the Large Indexes of size
10 GB using
>>>>>> Luke.it
>>>>>> >>> optimize all the content into a single CFS file and
it generates
>>>>>> >>> segments.gen and segments_8 file when i search the item
it shows
>>>>>> an
>>>>>> >>> error that segments file is not there.could you help
me in this 
>>>>>> >>> 
>>>>>> >>> testn wrote:
>>>>>> >>>> 
>>>>>> >>>> 1. You can close the searcher once you're done.
If you want to
>>>>>> reopen
>>>>>> >>>> the index, you can close and reopen only the updated
3 readers
>>>>>> and keep
>>>>>> >>>> the 2 old indexreaders and reuse it. It should reduce
the time
>>>>>> to
>>>>>> >>>> reopen it.
>>>>>> >>>> 2. Make sure that you optimize it every once in
a while
>>>>>> >>>> 3. You might consider separating indices in separated
storage
>>>>>> and use
>>>>>> >>>> ParallelReader
>>>>>> >>>> 
>>>>>> >>>> 
>>>>>> >>>> 
>>>>>> >>>> Sebastin wrote:
>>>>>> >>>>> 
>>>>>> >>>>> The problem in my pplication are as follows:
>>>>>> >>>>>                  1.I am not able to see the
updated records in
>>>>>> my
>>>>>> >>>>> index store because i instantiate 
>>>>>> >>>>> IndexReader and IndexSearcher class once that
is in the first
>>>>>> >>>>> search.further searches use the same IndexReaders(5
>>>>>> Directories) and
>>>>>> >>>>> IndexSearcher with different queries.
>>>>>> >>>>> 
>>>>>> >>>>>                 2.My search is very very slow
First 2
>>>>>> Directories of
>>>>>> >>>>> size 10 GB each which are having old index records
and no
>>>>>> update in
>>>>>> >>>>> that remaining 3 Diretories are updated every
second.
>>>>>> >>>>> 
>>>>>> >>>>>                 3.i am Indexing 20 million records
per day so
>>>>>> the
>>>>>> >>>>> Index store gets growing and it makes search
very very slower.
>>>>>> >>>>>  
>>>>>> >>>>>                4.I am using searcherOne class
as the global
>>>>>> >>>>> application helper class ,with the scope as
APPLICATION it
>>>>>> consists of
>>>>>> >>>>> one IndexReader and IndexSearcher get set method
which will
>>>>>> hold the
>>>>>> >>>>> IndexReader and IndexSearcher object after the
First Search.it
>>>>>> is used
>>>>>> >>>>> for all other searches.
>>>>>> >>>>> 
>>>>>> >>>>>               5.I am using Lucene 2.2.0 version,
in a WEB
>>>>>> Application
>>>>>> >>>>> which index 15 fields per document and Index
5 Fieds,store 10
>>>>>> Fields.i
>>>>>> >>>>> am not using any sort in my query.for a single
query upto the
>>>>>> maximum
>>>>>> >>>>> it fetches 600 records from the index store(5
direcories)    
>>>>>> >>>>>                 
>>>>>> >>>>> 
>>>>>> >>>>> hossman wrote:
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> : I set IndexSearcher as the application
Object after the
>>>>>> first
>>>>>> >>>>>> search.
>>>>>> >>>>>> 	...
>>>>>> >>>>>> : how can i reconstruct the new IndexSearcher
for every hour
>>>>>> to see
>>>>>> >>>>>> the
>>>>>> >>>>>> : updated records .
>>>>>> >>>>>> 
>>>>>> >>>>>> i'm confused ... my understanding based
on the comments you
>>>>>> made
>>>>>> >>>>>> below 
>>>>>> >>>>>> (in an earlier message) was that you already
*were*
>>>>>> constructing a
>>>>>> >>>>>> new  
>>>>>> >>>>>> IndexSearcher once an hour -- but every
time you do that, your
>>>>>> memory 
>>>>>> >>>>>> usage grows, and and that sometimes you
got OOM Errors.
>>>>>> >>>>>> 
>>>>>> >>>>>> if that's not what you said, then i think
you need to explain,
>>>>>> in
>>>>>> >>>>>> detail, 
>>>>>> >>>>>> in one message, exactly what your problem
is.  And don't
>>>>>> assume we 
>>>>>> >>>>>> understand anything -- tell us *EVERYTHING*
(like, for
>>>>>> example, what
>>>>>> >>>>>> the 
>>>>>> >>>>>> word "crore" means, how "searcherOne" is
implemented, and the
>>>>>> answer
>>>>>> >>>>>> to 
>>>>>> >>>>>> the specfic question i asked in my last
message: does your
>>>>>> >>>>>> application, 
>>>>>> >>>>>> contain anywhere in it, any code that will
close anything
>>>>>> >>>>>> (IndexSearchers 
>>>>>> >>>>>> or IndexReaders) ?
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> : > : I use StandardAnalyzer.the records
daily ranges from 5
>>>>>> crore to
>>>>>> >>>>>> 6 crore.
>>>>>> >>>>>> : > for
>>>>>> >>>>>> : > : every second i am updating my Index.
i instantiate
>>>>>> >>>>>> IndexSearcher object
>>>>>> >>>>>> : > one
>>>>>> >>>>>> : > : time for all the searches. for
an hour can i see the
>>>>>> updated
>>>>>> >>>>>> records in
>>>>>> >>>>>> : > the
>>>>>> >>>>>> : > : indexstore by reinstantiating IndexSearcher
object.but
>>>>>> the
>>>>>> >>>>>> problem when
>>>>>> >>>>>> : > i
>>>>>> >>>>>> : > : reinstantiate IndexSearcher ,my
RAM memory gets
>>>>>> appended.is
>>>>>> >>>>>> there any
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> : > IndexSearcher are you explicitly
closing both the old
>>>>>> >>>>>> IndexSearcher as 
>>>>>> >>>>>> : > well as all of 4 of those old IndexReaders
and the
>>>>>> MultiReader?
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> -Hoss
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> >>>>>> To unsubscribe, e-mail:
>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>> >>>>>> For additional commands, e-mail:
>>>>>> java-user-help@lucene.apache.org
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>> 
>>>>>> >>>>> 
>>>>>> >>>> 
>>>>>> >>>> 
>>>>>> >>> 
>>>>>> >>> 
>>>>>> >> 
>>>>>> >> 
>>>>>> > 
>>>>>> > 
>>>>>> 
>>>>>> -- 
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12655880
>>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>> 
>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12729051
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message