lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harald Stowasser <stowasse...@idowa.de>
Subject Re: OutOfMemory when indexing
Date Mon, 13 Jun 2005 13:16:36 GMT
Stanislav Jordanov schrieb:

> High guys,
> Building some huge index (about 500,000 docs totaling to 10megs of plain
> text) we've run into the following problem:
> Most of the time the IndexWriter process consumes a fairly small amount
> of memory (about 32 megs).
> However, as the index size grows, the memory usage sporadically bursts
> to levels of (say) 1000 gigs and then falls back to its level.
> The problem is that unless te process is started with some option like
> -Xmx1000m this situation causes an OutOfMemoryException which terminates
> the indexing process.
> 
> My question is - is there a way to avoid it?


1.
I start my programm with:
java -Xms256M -Xmx512M -jar Suchmaschine.jar &

This protect me now from OutOfMemoryException. After I use
iterative-subroutines.

2.
Free your variables as soon as possible.
like "term=null;"
This will help your Garbage-Collector!

3.
Maybe you should watch totalMemory and R.freeMemory() from
Runtime.getRuntime()
That will help you to find the "Memory-dissipater"

4.
I had the problem when deleting Documents from Index. I used a
Subroutine to delete single Documents.
It runs much better when I replaced it into a "iterative" subroutine
like this:

  public int deleteMany(String keywords)
  {
    int anzahl=0;
    try
    {
      openReader();
      String[] temp = keywords.split(",");
      //Runtime R = Runtime.getRuntime();
      for (int i = 0 ; i < temp.length ; i++)
      {
        Term term =new Term("keyword",temp[i]);
        anzahl+= mReader.delete(term);
        term=null;
        /*System.out.println("deleted " + temp[i]
                   +" t:"+R.totalMemory()
                   +" f:"+R.freeMemory()
                   +" m"+R.maxMemory());
        */
      }
      close();
    } catch (Exception e){
      cIdowa.error( "Could not delete Documents:" + keywords
            +". Because:"+ e.getMessage() + "\n" +e.toString() );
    }
    return anzahl;
  }




Mime
View raw message