lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastin <sebasmt...@gmail.com>
Subject Re: Java Heap Space -Out Of Memory Error
Date Wed, 05 Sep 2007 15:46:39 GMT

I use StandardAnalyzer.the records daily ranges from 5 crore to 6 crore. for
every second i am updating my Index. i instantiate IndexSearcher object one
time for all the searches. for an hour can i see the updated records in the
indexstore by reinstantiating IndexSearcher object.but the problem when i
reinstantiate IndexSearcher ,my RAM memory gets appended.is there any
possibility to control the memory at that time without affecting
performance.

testn wrote:
> 
> A couple things to make sure:
> 1. When you open IndexWriter, what is the analyzer you use?
> StandardAnalyzer?
> 2. How many records are there?
> 3. Could you also check number of terms in your indices? If there are too
> many terms, you could consider chop something in smaller piece for
> example... store area code and phone number separately if the numbers are
> pretty distributed.
> 
> 
> Sebastin wrote:
>> 
>> Hi testn,
>>            here is my index details:
>>                             Index fields :5 fields
>>                             Store Fileds:10 fields
>>            
>> 
>> Index code:              
>> 
>>   contents=new StringBuilder().append(compCallingPartyNumber).append("
>> ").append(compCalledPartyNumber).append("
>> ").append(compImsiNumber).append(" ").append(callType).toString();
>> 
>> 
>> records=new StringBuilder().append(compCallingPartyNumber).append("
>> ").append(compCalledPartyNumber).append("
>> ").append(compchargDur).append(" ").append(compTimeSc).append("
>> ").append(compImsiNumber).append(" ").append(outgoingRoute).append("
>> ").append(incomingRoute).append(" ").append(cgiLocation).toString();
>> 
>> 
>>                                Document document = new Document();
>>                                     document.add(new Field("contents",
>>                                                            contents,
>>                                                           
>> Field.Store.NO,
>>                                                           
>> Field.Index.TOKENIZED));
>>                                     document.add(new Field("fil", filen,
>>                                                           
>> Field.Store.NO,
>>                                                           
>> Field.Index.TOKENIZED));
>>                                     document.add(new Field("records",
>> records,
>>                                                           
>> Field.Store.YES,
>>                                                           
>> Field.Index.NO));
>>                                     document.add(new Field("dateSc",
>> dateSc,
>>                                                           
>> Field.Store.YES,
>>                                                           
>> Field.Index.TOKENIZED));
>>                                     
>>                                     indexWriter.addDocument(document);
>> 
>> inputs for the document:
>> 
>> compCallingPartyNumber="9840836588";
>> compCalledPartyNumber="9840861114";
>> compImsiNumber="984510005469874";
>> callType="1";
>> compChargDur="98456";
>> compTimeSc="984";
>> outgoingRoute="i987j";
>> incomingRoute="poi09";
>> cgiLocation="dft1234567";
>> 
>> here is my search code:
>> 
>>                                          Directory indexDir2 = 
>>                       FSDirectory.getDirectory(indexSourceDir02,false);
>>                                          IndexReader indexSource2 =
>> IndexReader.open(indexDir2);
>>                                        Directory indexDir3 = 
>>                                             
>> FSDirectory.getDirectory(indexSourceDir03,false);
>>                                          IndexReader indexSource3 =
>> IndexReader.open(indexDir3);
>>                                          Directory indexDir4 = 
>>                                             
>> FSDirectory.getDirectory(indexSourceDir04,false);
>>                                          IndexReader indexSource4 =
>> IndexReader.open(indexDir4); 
>>                                         
>>            
>>                                          
>>         IndexReader[] readArray =
>> {indexSource2,indexSource3,indexSource4};
>>         //merged reader
>>         IndexReader mergedReader = new MultiReader(readArray);
>>         IndexSearcher is = new IndexSearcher(mergedReader);
>>         
>>                                 
>>                                  QueryParser parser = 
>>                                      new QueryParser("contents" ,new
>> StandardAnalyzer());
>>                                  
>>                                  
>>                                 String searchQuery= 
>>                                        new
>> StringBuffer().append(inputNo).append(" AND dateSc:["
>> ).append(fromDate).append(" TO ").append(toDate).append("]").append("
>> ").append("AND").append(" ").append(callTyp).toString();
>>                                    
>>                                  
>>                                  
>>                                  Query callDetailquery =
>> parser.parse(searchQuery);
>>                                  
>>                                  hits = is.search(callDetailquery); 
>> 
>> 
>> 
>> 
>>                                              
>>                                        
>>                                           
>>          
>> 
>> testn wrote:
>>> 
>>> Can you provide more info about your index? How many documents, fields
>>> and what is the average document length?
>>> 
>>> 
>>> Sebastin wrote:
>>>> 
>>>> Hi testn,
>>>>            i index the dateSc as 070904(2007/09/04) format.i am not
>>>> using any timestamp here.how can we effectively reopen the
>>>> IndexSearcher  for an hour and save the memory because my index gets
>>>> updated every minute.
>>>> 
>>>> testn wrote:
>>>>> 
>>>>> Check out Wiki for more information at
>>>>> http://wiki.apache.org/jakarta-lucene/LargeScaleDateRangeProcessing
>>>>> 
>>>>> 
>>>>> 
>>>>> Sebastin wrote:
>>>>>> 
>>>>>> Hi All,
>>>>>>        i used to search 3 Lucene Index store of size 6 GB,10 GB,10
GB
>>>>>> of records using MultiReader class.
>>>>>> 
>>>>>> here is the following code snippet:
>>>>>> 
>>>>>> 
>>>>>>      
>>>>>>                                          Directory indexDir2 = 
>>>>>>                      
>>>>>> FSDirectory.getDirectory(indexSourceDir02,false);
>>>>>>                                          IndexReader indexSource2
=
>>>>>> IndexReader.open(indexDir2);
>>>>>>                                        Directory indexDir3 = 
>>>>>>                                             
>>>>>> FSDirectory.getDirectory(indexSourceDir03,false);
>>>>>>                                          IndexReader indexSource3
=
>>>>>> IndexReader.open(indexDir3);
>>>>>>                                          Directory indexDir4 = 
>>>>>>                                             
>>>>>> FSDirectory.getDirectory(indexSourceDir04,false);
>>>>>>                                          IndexReader indexSource4
=
>>>>>> IndexReader.open(indexDir4); 
>>>>>>                                         
>>>>>>            
>>>>>>                                          
>>>>>>         IndexReader[] readArray =
>>>>>> {indexSource2,indexSource3,indexSource4};
>>>>>>         //merged reader
>>>>>>         IndexReader mergedReader = new MultiReader(readArray);
>>>>>>         IndexSearcher is = new IndexSearcher(mergedReader);
>>>>>>         
>>>>>>                                 
>>>>>>                                  QueryParser parser = 
>>>>>>                                      new QueryParser("contents" ,new
>>>>>> StandardAnalyzer());
>>>>>>                                  
>>>>>>                                  
>>>>>>                                 String searchQuery= 
>>>>>>                                        new
>>>>>> StringBuffer().append(inputNo).append(" AND dateSc:["
>>>>>> ).append(fromDate).append(" TO ").append(toDate).append("]").append("
>>>>>> ").append("AND").append(" ").append(callTyp).toString();
>>>>>>                                    
>>>>>>                                  
>>>>>>                                  
>>>>>>                                  Query callDetailquery =
>>>>>> parser.parse(searchQuery);
>>>>>>                                  
>>>>>>                                  hits = is.search(callDetailquery);

>>>>>> 
>>>>>> 
>>>>>> it takes 300 MB of RAM for every search and it is very very slow
is
>>>>>> there any other way to control the Memory and to make search faster.i
>>>>>> use SINGLETON  to use the IndexSearcher as a one time used object
for
>>>>>> all the instances.
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12500824
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message