Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <12690737.post@talk.nabble.com>
Date: Sat, 15 Sep 2007 07:33:31 -0700 (PDT)
From: Sebastin <sebasmtech@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Java Heap Space -Out Of Memory Error
In-Reply-To: <12690100.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <12475468.post@talk.nabble.com> <12476607.post@talk.nabble.com>
 <12478804.post@talk.nabble.com> <12484208.post@talk.nabble.com>
 <12492218.post@talk.nabble.com> <12496515.post@talk.nabble.com>
 <12500824.post@talk.nabble.com>
 <Pine.LNX.4.62.0709051624210.1421@radix.cryptio.net>
 <12515624.post@talk.nabble.com>
 <Pine.LNX.4.62.0709052208180.31701@radix.cryptio.net>
 <12528983.post@talk.nabble.com> <12595489.post@talk.nabble.com>
 <12650012.post@talk.nabble.com> <12652816.post@talk.nabble.com>
 <12655013.post@talk.nabble.com> <12655880.post@talk.nabble.com>
 <1189695196.8650.1210472501@webmail.messagingengine.com>
 <12668746.post@talk.nabble.com> <12679970.post@talk.nabble.com>
 <12687410.post@talk.nabble.com> <12690100.post@talk.nabble.com>


 HI testn,
               could you trigger me out the You can simply create a wrapper
that return MultiReader which you can cache for a while and close the oldest
index once the date rolls.,this point in detail.i am not able to get that. 

testn wrote:
> 
> If you know that there are only 15 days of indexes you need to search on,
> you just need to open only the latest 15 indexes at a time right? You can
> simply create a wrapper that return MultiReader which you can cache for a
> while and close the oldest index once the date rolls.
> 
> 
> Sebastin wrote:
>> 
>> HI testn,
>> 
>> it gives performance improvement while optimizing the Index. 
>> 
>> Now i seprate the IndexStore on a daily basis.(ie) 
>> For Every Day it create a new Index store ,sep- 08-2007,sep-09-2007 like
>> wise it will minimize the size of the IndexStore.could you give me an
>> idea on how to open every day folders for every search.
>> 
>> Query I use here is,
>> 
>> 9840836588 AND dateSc:[070901 TO 070910] 
>> 
>> 07---->year (2007)
>> 09---->month(september)
>> 01----->day
>> 
>> i restrict for 15 days that it is possible to search 15 days record in my
>> application.at a time 10 users aare going to search every store.is there
>> any other better way to improve the search performance to avoid memory
>> problem as well as speed of the search.
>> 
>> 
>> 
>> 
>> 
>> 
>> testn wrote:
>>> 
>>> So did you see any improvement in performance?
>>> 
>>> Sebastin wrote:
>>>> 
>>>> It works finally .i use Lucene 2.2  in my application.thanks testn and
>>>> Mike
>>>> 
>>>> Michael McCandless-2 wrote:
>>>>> 
>>>>> 
>>>>> It sounds like there may be a Lucene version mismatch?  When Luke was
>>>>> used
>>>>> it was likely based on Lucene 2.2, but it sounds like an older version
>>>>> of
>>>>> Lucene is now being used to open the index?
>>>>> 
>>>>> Mike
>>>>> 
>>>>> "testn" <test1@doramail.com> wrote:
>>>>>> 
>>>>>> Should the file be "segments_8" and "segments.gen"? Why is it
>>>>>> "Segment"?
>>>>>> The
>>>>>> case is different.
>>>>>> 
>>>>>> 
>>>>>> Sebastin wrote:
>>>>>> > 
>>>>>> > java.io.IoException:File Not Found- Segments  is the error message
>>>>>> > 
>>>>>> > testn wrote:
>>>>>> >> 
>>>>>> >> What is the error message? Probably Mike, Erick or Yonik can help
>>>>>> you
>>>>>> >> better on this since I'm no one in index area.
>>>>>> >> 
>>>>>> >> Sebastin wrote:
>>>>>> >>> 
>>>>>> >>> HI testn,
>>>>>> >>>              1.I optimize the Large Indexes of size 10 GB using
>>>>>> Luke.it
>>>>>> >>> optimize all the content into a single CFS file and it generates
>>>>>> >>> segments.gen and segments_8 file when i search the item it shows
>>>>>> an
>>>>>> >>> error that segments file is not there.could you help me in this 
>>>>>> >>> 
>>>>>> >>> testn wrote:
>>>>>> >>>> 
>>>>>> >>>> 1. You can close the searcher once you're done. If you want to
>>>>>> reopen
>>>>>> >>>> the index, you can close and reopen only the updated 3 readers
>>>>>> and keep
>>>>>> >>>> the 2 old indexreaders and reuse it. It should reduce the time
>>>>>> to
>>>>>> >>>> reopen it.
>>>>>> >>>> 2. Make sure that you optimize it every once in a while
>>>>>> >>>> 3. You might consider separating indices in separated storage
>>>>>> and use
>>>>>> >>>> ParallelReader
>>>>>> >>>> 
>>>>>> >>>> 
>>>>>> >>>> 
>>>>>> >>>> Sebastin wrote:
>>>>>> >>>>> 
>>>>>> >>>>> The problem in my pplication are as follows:
>>>>>> >>>>>                  1.I am not able to see the updated records in
>>>>>> my
>>>>>> >>>>> index store because i instantiate 
>>>>>> >>>>> IndexReader and IndexSearcher class once that is in the first
>>>>>> >>>>> search.further searches use the same IndexReaders(5
>>>>>> Directories) and
>>>>>> >>>>> IndexSearcher with different queries.
>>>>>> >>>>> 
>>>>>> >>>>>                 2.My search is very very slow First 2
>>>>>> Directories of
>>>>>> >>>>> size 10 GB each which are having old index records and no
>>>>>> update in
>>>>>> >>>>> that remaining 3 Diretories are updated every second.
>>>>>> >>>>> 
>>>>>> >>>>>                 3.i am Indexing 20 million records per day so
>>>>>> the
>>>>>> >>>>> Index store gets growing and it makes search very very slower.
>>>>>> >>>>>  
>>>>>> >>>>>                4.I am using searcherOne class as the global
>>>>>> >>>>> application helper class ,with the scope as APPLICATION it
>>>>>> consists of
>>>>>> >>>>> one IndexReader and IndexSearcher get set method which will
>>>>>> hold the
>>>>>> >>>>> IndexReader and IndexSearcher object after the First Search.it
>>>>>> is used
>>>>>> >>>>> for all other searches.
>>>>>> >>>>> 
>>>>>> >>>>>               5.I am using Lucene 2.2.0 version, in a WEB
>>>>>> Application
>>>>>> >>>>> which index 15 fields per document and Index 5 Fieds,store 10
>>>>>> Fields.i
>>>>>> >>>>> am not using any sort in my query.for a single query upto the
>>>>>> maximum
>>>>>> >>>>> it fetches 600 records from the index store(5 direcories)    
>>>>>> >>>>>                 
>>>>>> >>>>> 
>>>>>> >>>>> hossman wrote:
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> : I set IndexSearcher as the application Object after the
>>>>>> first
>>>>>> >>>>>> search.
>>>>>> >>>>>> 	...
>>>>>> >>>>>> : how can i reconstruct the new IndexSearcher for every hour
>>>>>> to see
>>>>>> >>>>>> the
>>>>>> >>>>>> : updated records .
>>>>>> >>>>>> 
>>>>>> >>>>>> i'm confused ... my understanding based on the comments you
>>>>>> made
>>>>>> >>>>>> below 
>>>>>> >>>>>> (in an earlier message) was that you already *were*
>>>>>> constructing a
>>>>>> >>>>>> new  
>>>>>> >>>>>> IndexSearcher once an hour -- but every time you do that, your
>>>>>> memory 
>>>>>> >>>>>> usage grows, and and that sometimes you got OOM Errors.
>>>>>> >>>>>> 
>>>>>> >>>>>> if that's not what you said, then i think you need to explain,
>>>>>> in
>>>>>> >>>>>> detail, 
>>>>>> >>>>>> in one message, exactly what your problem is.  And don't
>>>>>> assume we 
>>>>>> >>>>>> understand anything -- tell us *EVERYTHING* (like, for
>>>>>> example, what
>>>>>> >>>>>> the 
>>>>>> >>>>>> word "crore" means, how "searcherOne" is implemented, and the
>>>>>> answer
>>>>>> >>>>>> to 
>>>>>> >>>>>> the specfic question i asked in my last message: does your
>>>>>> >>>>>> application, 
>>>>>> >>>>>> contain anywhere in it, any code that will close anything
>>>>>> >>>>>> (IndexSearchers 
>>>>>> >>>>>> or IndexReaders) ?
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> : > : I use StandardAnalyzer.the records daily ranges from 5
>>>>>> crore to
>>>>>> >>>>>> 6 crore.
>>>>>> >>>>>> : > for
>>>>>> >>>>>> : > : every second i am updating my Index. i instantiate
>>>>>> >>>>>> IndexSearcher object
>>>>>> >>>>>> : > one
>>>>>> >>>>>> : > : time for all the searches. for an hour can i see the
>>>>>> updated
>>>>>> >>>>>> records in
>>>>>> >>>>>> : > the
>>>>>> >>>>>> : > : indexstore by reinstantiating IndexSearcher object.but
>>>>>> the
>>>>>> >>>>>> problem when
>>>>>> >>>>>> : > i
>>>>>> >>>>>> : > : reinstantiate IndexSearcher ,my RAM memory gets
>>>>>> appended.is
>>>>>> >>>>>> there any
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> : > IndexSearcher are you explicitly closing both the old
>>>>>> >>>>>> IndexSearcher as 
>>>>>> >>>>>> : > well as all of 4 of those old IndexReaders and the
>>>>>> MultiReader?
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> -Hoss
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> >>>>>> To unsubscribe, e-mail:
>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>> >>>>>> For additional commands, e-mail:
>>>>>> java-user-help@lucene.apache.org
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>>> 
>>>>>> >>>>> 
>>>>>> >>>>> 
>>>>>> >>>> 
>>>>>> >>>> 
>>>>>> >>> 
>>>>>> >>> 
>>>>>> >> 
>>>>>> >> 
>>>>>> > 
>>>>>> > 
>>>>>> 
>>>>>> -- 
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12655880
>>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>> 
>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12690737
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org