lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sergiu gordea <gser...@ifit.uni-klu.ac.at>
Subject Re: OutOfMemory example
Date Mon, 13 Sep 2004 15:54:43 GMT
then probably is my mistake ...I havn't read all the emails in the thread.
So ... your goal is to produce errors ... I try to avoid them :))

   All the best,

  Sergiu

  

Jiří Kuhn wrote:

>You don't see the point of my post. I sent an application which can everyone run only
with lucene jar and in deterministic way produce OutOfMemoryError.
>
>That's all.
>
>Jiri.
>
>
>-----Original Message-----
>From: sergiu gordea [mailto:gsergiu@ifit.uni-klu.ac.at]
>Sent: Monday, September 13, 2004 5:16 PM
>To: Lucene Users List
>Subject: Re: OutOfMemory example
>
>
>I have a few comments regarding your code ...
>1. Why do you use RamDirectory and not the hard disk?
>2. as John said, you should reuse the index instead of creating it each 
>time in the main function
>    if(!indexExists(File indexFile))
>         IndexWriter writer = new IndexWriter(directory, new 
>StandardAnalyzer(), true);
>    else
>         IndexWriter writer = new IndexWriter(directory, new 
>StandardAnalyzer(), false);
>    (in some cases indexExists can be as simple as verifying if the file 
>exits on the hard disk)
>
>3. you iterate in a loop over 10.000 times and you create a lot of objects
>   
>
>for (int i = 0; i < 365 * 30; i++) {
>            Document doc = new Document();
>
>            doc.add(Field.Keyword("date", df.format(new 
>Date(c.getTimeInMillis()))));
>            doc.add(Field.Keyword("id", "AB" + String.valueOf(i)));
>            doc.add(Field.Text("text", "Tohle je text " + i));
>            writer.addDocument(doc);
>
>            c.add(Calendar.DAY_OF_YEAR, 1);
>        }
>all the underlined lines of code create new  ojects, and all of them are 
>kept in memory.
>This is a lot of memory allocated only by this loop. I think that you 
>create more than 100.000 object in this loop ...
>What do you think?
>And none of them cannot be realeased (collected by gc) untill you close 
>the index writer.
>
>None says that your code is complicated, but all programmers should 
>understand that this is a poor design...
>And ... more then that your information is kept in a RamDirectory 
>....when you will close the writer you will still keep the information 
>in memory ...
>
>Sory if I was too agressive with my comments .... but ... I cannot see 
>what were you thinking when you wrote that code ...
>
>If you are trying to make a test .... then I sugest you to replace the 
>hard codded 365 value ... with a variable, to iterate over it and to 
>test the power of your machine
>(PC + JVM) :))
>
>I wish you luck,
>
> Sergiu
>
>
>
>
>
>
>Jiří Kuhn wrote:
>
>  
>
>>I disagree or I don't understand. 
>>
>>I can change the code as it is shown below. Now I must reopen the index to see the
changes, but the memory problem remains. I realy don't know what I'm doing wrong, the code
is so simple.
>>
>>Jiri.
>>
>>	...
>>
>>   public static void main(String[] args) throws IOException
>>   {
>>       Directory directory = create_index();
>>
>>       for (int i = 1; i < 100; i++) {
>>           System.err.println("loop " + i + ", index version: " + IndexReader.getCurrentVersion(directory));
>>           search_index(directory);
>>           add_to_index(directory, i);
>>       }
>>   }
>>
>>   private static void add_to_index(Directory directory, int i) throws IOException
>>   {
>>       IndexWriter writer = new IndexWriter(directory, new StandardAnalyzer(), false);
>>
>>       SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd");
>>       Document doc = new Document();
>>
>>       doc.add(Field.Keyword("date", df.format(new Date(System.currentTimeMillis()))));
>>       doc.add(Field.Keyword("id", "CD" + String.valueOf(i)));
>>       doc.add(Field.Text("text", "Tohle neni text " + i));
>>       writer.addDocument(doc);
>>
>>       System.err.println("index size: " + writer.docCount());
>>       writer.close();
>>   }
>>
>>	...
>>
>>-----Original Message-----
>>From: John Moylan [mailto:johnm@rte.ie]
>>Sent: Monday, September 13, 2004 3:25 PM
>>To: Lucene Users List
>>Subject: Re: OutOfMemory example
>>
>>
>>You should reuse your old index (as eg an application variable) unless 
>>it has changed - use getCurrentVersion to check the index for updates. 
>>This has come up before.
>>
>>John
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>
>> 
>>
>>    
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>  
>


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message