lucene-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Solr Wiki] Update of "ShawnHeisey" by ShawnHeisey
Date Mon, 05 Jan 2015 06:50:57 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification.

The "ShawnHeisey" page has been changed by ShawnHeisey:
https://wiki.apache.org/solr/ShawnHeisey?action=diff&rev1=25&rev2=26

Comment:
Added a note saying that a few humongous allocations are perfectly normal.

  
  /!\ Some notes about the !G1HeapRegionSize parameter:  I was seeing a large number of "humongous
allocations" in the GC log, which means that they are larger than 50 percent of a G1 heap
region.  Those allocations were about 2MB in size, and my 4GB minimum heap results in a default
region size of 2MB, which is calculated by dividing the -Xms value by 2048.  I suspect that
those allocations were for filterCache entries, because the indexes on that instance are each
about 16.5 million documents, and the bitset memory structure of a filter for an index that
size is a little over 2MB.  Setting the region size to 8MB ensures that those allocations
are not categorized as humongous.  See "Humongous Allocations" section on this [[http://www.infoq.com/articles/tuning-tips-G1-GC|blog
entry]] for more information.
  
- Garbage collection for Allocations that are tagged as humongous is not fully handled in
the concurrent and low-pause parts of the process.  They can only be properly handled by a
full garbage collection, which will always be slow.  Depending on the size of your indexes
and the size of your heap, the region size may require adjustment from the 8MB that I am using
above.  If you have an index with about 100 million documents in it, you'll want to use a
region size of 32MB, which is the maximum possible size.  Because of this limitation of the
G1 collector, we recommend always keeping a Solr index below a maxDoc value of around 100
to 120 million.
+ Garbage collection for allocations that are tagged as humongous is not fully handled in
the concurrent and low-pause parts of the process.  They can only be properly handled by a
full garbage collection, which will always be slow.  Depending on the size of your indexes
and the size of your heap, the region size may require adjustment from the 8MB that I am using
above.  If you have an index with about 100 million documents in it, you'll want to use a
region size of 32MB, which is the maximum possible size.  Because of this limitation of the
G1 collector, we recommend always keeping a Solr index below a maxDoc value of around 100
to 120 million.
  
  Further down on this page, I have included GC logging options that will create a log you
can search for "humongous" to determine if there are a lot of these allocations.  I have been
informed that with Java 8u40, the !G1HeapRegionSize parameter is not required, because Java
8 manages large allocations a lot better than Java 7 does.
  
@@ -69, +69 @@

  
  === Garbage Collection Logging ===
  
- Here are the GC logging options that I use.  If you've decided to use the G1 options above
and would like to know if you're having a lot of humongous allocations, the logfile created
by these options will tell you.  You can also use the logfile to create a graph of your garbage
collection activity to compare different GC tuning options.  Be sure that you use a valid
logfile path on the -Xloggc parameter:
+ Here are the GC logging options that I use.  If you've decided to use the G1 options above
and would like to know if you're having a lot of humongous allocations, the logfile created
by these options will tell you.  A few humongous allocations are perfectly normal ... but
there should not be very many of them.  You can also use the logfile to create a graph of
your garbage collection activity to compare different GC tuning options.  Be sure that you
use a valid logfile path on the -Xloggc parameter:
  
  {{{
  GCLOG_OPTS=" \

Mime
View raw message