lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Memory use and Lucene
Date Fri, 02 Apr 2010 09:13:05 GMT
OS level tools (top, ps, activity monitor, task manager) aren't great
ways to measure Java's memory usage, since they only see how much heap
java has allocated from the OS.  Within that heap, java can have lots
of free space that it knows about but the OS does not (this is
Runtime.freeMemory()).

You can try restricting the heap size java is allowed to use (java
-Xmx) -- keep reducing it until you hit OOME to see how much "real"
memory is required.  But: don't run, in general, with such a tight
heap.  First off it's a real performance hit (GC runs "hot").  Second
off, there's always the risk that something needs to temporarily
allocate something big and then free it and that thing will hit OOME.
It'd be like an airplane flying over a mountain range just a wee bit
over what it thinks is the highest peak...

Note that Runtime.gc() does not guarantee it actually runs.

If you want to dig, use a memory profiler or heap dump to see what's
still reachable...

Mike

On Thu, Apr 1, 2010 at 7:41 PM, John Viviano <john@intercorpinc.com> wrote:
> All -
>
>
>
> I have a question is about memory use and Lucene.  I'm not sure if I'm
> dealing with a leak, or if I'm seeing expected behavior.  I'll preface this
> by acknowledging that the "error" could be in my understanding of things.
>
>
>
> I've included a lot of information below.  There is a demo program that
> follows a pattern pretty close to what one segment of my application does.
> There's output from 'top' as well as some info about my os and jvm.
>
>
>
> My primary concern:  according to 'top' at startup, the application is
> taking about 22MB.  After indexing, closing the indexWriter and running
> garbage collection, 'top' reports that the program is about 170MB.  So is
> this a leak?  Is this expected behavior?  Is it something where an allocator
> doesn't return memory to the OS, perhaps for efficiency?  Something else?
>
>
>
> -----------------  STARTING VALUES IN TOP --------------------
>
> Mem:   1048576k total,   305252k used,   743324k free,        0k buffers
>
> Swap:        0k total,        0k used,        0k free,        0k cached
>
>
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>
> 5422 jviviano  17   0  332m  21m 8320 S    0  2.1   0:00.28 java
>
>
>
> -----------------  ENDING VALUES IN TOP --------------------
>
> Mem:   1048576k total,   443572k used,   605004k free,        0k buffers
>
> Swap:        0k total,        0k used,        0k free,        0k cached
>
>
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>
> 5422 jviviano  16   0  463m 166m 8568 S    0 16.2   0:03.97 java
>
>
>
> I also use the java runtime values to report what java knows about memory
> inside my program, as follows:
>
>
>
> Starting free memory:   84,882,328 bytes
>
>      Released by GC:    3,849,488 bytes
>
>  Ending free memory:   84,562,360 bytes
>
>  Unexplained losses:      319,968 bytes
>
>
>
> By the way, if anyone has a better approach to understanding what's going on
> with memory, I'm all ears.
>
>
>
> Here's my sample code, followed by my environment:
>
> ----------------------------------------------------------------------------
> -----------------
>
>
>
> import java.io.IOException;
>
> import java.util.*;
>
> import java.io.File;
>
>
>
> // Developed for Lucene-3.0.1
>
> import org.apache.lucene.store.Directory;
>
> import org.apache.lucene.store.FSDirectory;
>
> import org.apache.lucene.index.IndexWriter;
>
> import org.apache.lucene.analysis.Analyzer;
>
> import org.apache.lucene.analysis.standard.StandardAnalyzer;
>
> import org.apache.lucene.document.Document;
>
> import org.apache.lucene.document.Field;
>
> import org.apache.lucene.util.Version;
>
> // No querying going on in this sample code
>
> //import org.apache.lucene.search.Query;
>
> //import org.apache.lucene.queryParser.QueryParser;
>
> //import org.apache.lucene.queryParser.ParseException;
>
>
>
> class MemoryQuestion
>
> {
>
>     static final int MAX_REPS = 1000;
>
>     static final String TEST_INDEX = "./testIndex/";
>
>
>
>     private static void forceGarbageCollection(Runtime r)
>
>     {
>
>           r.gc();
>
>     }
>
>
>
>    private static long checkAvailableMemory(Runtime r)
>
>     {
>
>           long availableMemory = r.maxMemory() - (r.totalMemory() -
> r.freeMemory());
>
>           return availableMemory;
>
>     }
>
>
>
>     public static void main(String[] args) throws IOException
>
>     {
>
>        // Get a Runtime object to help manage garbage collection
>
>        Runtime r = Runtime.getRuntime();
>
>
>
>           String[] testData = new String[] {
>
>                           "Lorem ipsum dolor sit amet, consectetur
> adipiscing elit.",
>
>                           "Praesent blandit lacinia sem vel pulvinar.",
>
>                           "Ut pellentesque tellus sit amet felis mollis
> semper.",
>
>                           "Sed eleifend, nisi ac placerat aliquam, elit
> ligula fermentum purus, ut placerat mauris tortor vitae nisl.",
>
>                           "Curabitur nec urna sed sem dapibus rutrum.",
>
>                           "Curabitur sodales venenatis tellus.",
>
>                           "Donec lacus metus, euismod eget dictum quis,
> feugiat ut magna.",
>
>                           "Ut eget faucibus lorem.",
>
>                           "Pellentesque habitant morbi tristique senectus
> et netus et malesuada fames ac turpis egestas.",
>
>                           "Nulla urna nisi, eleifend id consectetur at,
> ornare id metus.",
>
>                           "Phasellus dictum sodales augue, vitae auctor
> tortor ultricies ut.",
>
>                           "Nam ullamcorper varius est, vel ultrices turpis
> laoreet eu. Nunc id leo vitae felis faucibus ornare."
>
>                     };
>
>
>
>
>
>           // create a directory called testIndex in the working directory
>
>           // and an IndexWriter
>
>           Directory dir = FSDirectory.open(new File(TEST_INDEX));
>
>           IndexWriter indexWriter  = new IndexWriter(dir,
>
>                                                          
 new
> StandardAnalyzer(Version.LUCENE_CURRENT),
>
>
> IndexWriter.MaxFieldLength.LIMITED);
>
>           // if you want to experiment with different buffer sizes
>
>           // indexWriter.setRAMBufferSizeMB(8.0);
>
>
>
>           // start as clean as you can
>
>           forceGarbageCollection(r);
>
>           long startingMemory = checkAvailableMemory(r);
>
>
>
>           // loop through the array of testData strings MAX_REP times
>
>           //     create a new document for each string and add it to the
> index
>
>           //     commit the index at the end of each loop
>
>           // close the index
>
>           for (int i=0; i< MAX_REPS ; i++ )
>
>           {
>
>                System.out.print('.');
>
>
>
>                for (String text : testData) {
>
>                     Document doc = new Document();
>
>                     doc.add(new Field("text",text,
>
>                                     Field.Store.YES,Field.Index.ANALYZED));
>
>                     indexWriter.addDocument(doc);
>
>                     doc = null;
>
>                }
>
>                indexWriter.commit();
>
>           }
>
>           indexWriter.close();
>
>           indexWriter = null;
>
>
>
>           // get memory stats before and after forcing garbage collection
>
>           long pre_gcMemory = checkAvailableMemory(r);
>
>           forceGarbageCollection(r);
>
>           long endingMemory = checkAvailableMemory(r);
>
>
>
>           Formatter f = new Formatter();
>
>           System.out.println("");
>
>           System.out.printf("Starting free memory: %,12d bytes %n",
> startingMemory);
>
>           System.out.printf("      Released by GC: %,12d bytes %n",
> endingMemory - pre_gcMemory);
>
>           System.out.printf("  Ending free memory: %,12d bytes %n",
> endingMemory);
>
>           System.out.printf("  Unexplained losses: %,12d bytes %n",
> startingMemory - endingMemory);
>
>           System.out.println("Ctrl-C to exit -- you may want to check
> memory use as reported by 'top'");
>
>           while (true)
>
>           {
>
>                try {
>
>                     Thread.sleep(100);
>
>                } catch(Exception e)
>
>                {
>
>                     System.out.println(e);
>
>                }
>
>           }
>
>
>
>     }
>
> }
>
>
>
> ----------------------------------------------------------------------------
> -----------------
>
> My OS and JVM versions are as follows:
>
>
>
> Linux version 2.6.18-028stab066.10 (root@rhel5-64-build) (gcc version 4.1.2
> 20070626 (Red Hat 4.1.2-14)) #1 SMP Fri Dec 4 15:49:04 MSK 2009
>
>
>
> java version "1.6.0_17"
>
> Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
>
> Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)
>
>
>
>
>
> Thanks,
>
> John Viviano
>
>  <mailto:john@intercorpinc.com> john@intercorpinc.com
>
>
>
>
>
> InterCorp, Inc.
>
> 11469 Olive Blvd, Ste. 225
>
> St. Louis, MO 63141
>
> 314.610.5276
>
>
>
>  <http://www.intercorpinc.com/> www.intercorpinc.com
>
> "The problems that exist in the world today cannot
> be solved by the level of thinking that created them."
> - Albert Einstein
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message