Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (hermes.apache.org: local policy)
Message-ID: <42AD8734.9070205@idowa.de>
Date: Mon, 13 Jun 2005 15:16:36 +0200
From: Harald Stowasser <stowasser.h@idowa.de>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; de-AT;
 rv:1.7.2) Gecko/20040803
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: OutOfMemory when indexing
References: <42AD7F8F.5070205@sirma.bg>
In-Reply-To: <42AD7F8F.5070205@sirma.bg>
Content-Type: multipart/signed; micalg=pgp-sha1;
 protocol="application/pgp-signature";
 boundary="------------enig2B14CF68AD60B3A86BEFEE80"

--------------enig2B14CF68AD60B3A86BEFEE80
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Stanislav Jordanov schrieb:

> High guys,
> Building some huge index (about 500,000 docs totaling to 10megs of plain
> text) we've run into the following problem:
> Most of the time the IndexWriter process consumes a fairly small amount
> of memory (about 32 megs).
> However, as the index size grows, the memory usage sporadically bursts
> to levels of (say) 1000 gigs and then falls back to its level.
> The problem is that unless te process is started with some option like
> -Xmx1000m this situation causes an OutOfMemoryException which terminates
> the indexing process.
> 
> My question is - is there a way to avoid it?


1.
I start my programm with:
java -Xms256M -Xmx512M -jar Suchmaschine.jar &

This protect me now from OutOfMemoryException. After I use
iterative-subroutines.

2.
Free your variables as soon as possible.
like "term=null;"
This will help your Garbage-Collector!

3.
Maybe you should watch totalMemory and R.freeMemory() from
Runtime.getRuntime()
That will help you to find the "Memory-dissipater"

4.
I had the problem when deleting Documents from Index. I used a
Subroutine to delete single Documents.
It runs much better when I replaced it into a "iterative" subroutine
like this:

  public int deleteMany(String keywords)
  {
    int anzahl=0;
    try
    {
      openReader();
      String[] temp = keywords.split(",");
      //Runtime R = Runtime.getRuntime();
      for (int i = 0 ; i < temp.length ; i++)
      {
        Term term =new Term("keyword",temp[i]);
        anzahl+= mReader.delete(term);
        term=null;
        /*System.out.println("deleted " + temp[i]
                   +" t:"+R.totalMemory()
                   +" f:"+R.freeMemory()
                   +" m"+R.maxMemory());
        */
      }
      close();
    } catch (Exception e){
      cIdowa.error( "Could not delete Documents:" + keywords
            +". Because:"+ e.getMessage() + "\n" +e.toString() );
    }
    return anzahl;
  }


--------------enig2B14CF68AD60B3A86BEFEE80
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFCrYc4d2ByLviqWmYRAvHcAJ4rJZBZ28nchkB9U5fCpm+Fp5bNgQCcDmlK
+B0jZ52nRIw3wy5Fl9NP0IA=
=jnmG
-----END PGP SIGNATURE-----

--------------enig2B14CF68AD60B3A86BEFEE80--