lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sergiu gordea <gser...@ifit.uni-klu.ac.at>
Subject Re: OutOfMemory example
Date Mon, 13 Sep 2004 15:15:53 GMT
I have a few comments regarding your code ...
1. Why do you use RamDirectory and not the hard disk?
2. as John said, you should reuse the index instead of creating it each 
time in the main function
    if(!indexExists(File indexFile))
         IndexWriter writer = new IndexWriter(directory, new 
StandardAnalyzer(), true);
    else
         IndexWriter writer = new IndexWriter(directory, new 
StandardAnalyzer(), false);
    (in some cases indexExists can be as simple as verifying if the file 
exits on the hard disk)

3. you iterate in a loop over 10.000 times and you create a lot of objects
   

for (int i = 0; i < 365 * 30; i++) {
            Document doc = new Document();

            doc.add(Field.Keyword("date", df.format(new 
Date(c.getTimeInMillis()))));
            doc.add(Field.Keyword("id", "AB" + String.valueOf(i)));
            doc.add(Field.Text("text", "Tohle je text " + i));
            writer.addDocument(doc);

            c.add(Calendar.DAY_OF_YEAR, 1);
        }
all the underlined lines of code create new  ojects, and all of them are 
kept in memory.
This is a lot of memory allocated only by this loop. I think that you 
create more than 100.000 object in this loop ...
What do you think?
And none of them cannot be realeased (collected by gc) untill you close 
the index writer.

None says that your code is complicated, but all programmers should 
understand that this is a poor design...
And ... more then that your information is kept in a RamDirectory 
....when you will close the writer you will still keep the information 
in memory ...

Sory if I was too agressive with my comments .... but ... I cannot see 
what were you thinking when you wrote that code ...

If you are trying to make a test .... then I sugest you to replace the 
hard codded 365 value ... with a variable, to iterate over it and to 
test the power of your machine
(PC + JVM) :))

I wish you luck,

 Sergiu






Jiří Kuhn wrote:

>I disagree or I don't understand. 
>
>I can change the code as it is shown below. Now I must reopen the index to see the changes,
but the memory problem remains. I realy don't know what I'm doing wrong, the code is so simple.
>
>Jiri.
>
>	...
>
>    public static void main(String[] args) throws IOException
>    {
>        Directory directory = create_index();
>
>        for (int i = 1; i < 100; i++) {
>            System.err.println("loop " + i + ", index version: " + IndexReader.getCurrentVersion(directory));
>            search_index(directory);
>            add_to_index(directory, i);
>        }
>    }
>
>    private static void add_to_index(Directory directory, int i) throws IOException
>    {
>        IndexWriter writer = new IndexWriter(directory, new StandardAnalyzer(), false);
>
>        SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd");
>        Document doc = new Document();
>
>        doc.add(Field.Keyword("date", df.format(new Date(System.currentTimeMillis()))));
>        doc.add(Field.Keyword("id", "CD" + String.valueOf(i)));
>        doc.add(Field.Text("text", "Tohle neni text " + i));
>        writer.addDocument(doc);
>
>        System.err.println("index size: " + writer.docCount());
>        writer.close();
>    }
>
>	...
>
>-----Original Message-----
>From: John Moylan [mailto:johnm@rte.ie]
>Sent: Monday, September 13, 2004 3:25 PM
>To: Lucene Users List
>Subject: Re: OutOfMemory example
>
>
>You should reuse your old index (as eg an application variable) unless 
>it has changed - use getCurrentVersion to check the index for updates. 
>This has come up before.
>
>John
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>  
>


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message