Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of markrmiller@gmail.com
 designates 209.85.212.48 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1251.1)
Subject: Re: [Solr Wiki] Update of "SolrPerformanceFactors" by RobertMuir
From: Mark Miller <markrmiller@gmail.com>
In-Reply-To: <alpine.DEB.2.00.1111041054450.13130@bester>
Date: Sat, 5 Nov 2011 08:42:33 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <17C8FE34-1FA3-457F-B73A-27573B1218B4@gmail.com>
References: <20111103232612.64519.76402@eos.apache.org>
 <alpine.DEB.2.00.1111041054450.13130@bester>
To: dev@lucene.apache.org

+1 - this is good info. The first thing it says is "You may want to =
optimize an index whenever practical -- ie: if you build your index =
once, and then never modify it."

True stuff. At worst we should clarify practical a bit more - but I =
still don't think it's bad as is.

Then it gives you further good info.

What is the point of removing it?

- Mark
=20
On Nov 4, 2011, at 1:57 PM, Chris Hostetter wrote:

>=20
> Completley removing all of this info seems like more harm then good -- =
it=20
> actually advises against doing an optimize except when you know you're=20=

> never going to modify your index, and it explains the downsides of=20
> optimizing.
>=20
> i would suggest we add most of this back, but perhaps change the title=20=

> (since many pieces of info in this section aren't specific to=20
> optimizing, they're just about segments) and be more vigorous in =
warning=20
> about the costs of optimize.
>=20
> : The "SolrPerformanceFactors" page has been changed by RobertMuir:
> : =
http://wiki.apache.org/solr/SolrPerformanceFactors?action=3Ddiff&rev1=3D28=
&rev2=3D29
> :=20
> : Comment:
> : die optimize die
> :=20
> :  =20
> :      * Memory usage during indexing
> :      * Segment merge time
> : -    * Optimization times
> :      * Index size
> :  =20
> :   These impacts can be reduced by the use of `omitNorms=3D"true"`
> : @@ -74, +73 @@
> :=20
> :   =3D=3D=3D Explicit Warming of Sort Fields =3D=3D=3D
> :  =20
> :   If you do a lot of field based sorting, it is advantageous to add =
explicitly warming queries to the "newSearcher" and "firstSearcher" =
event listeners in your solrconfig which sort on those fields, so the =
!FieldCache is populated prior to any queries being executed by your =
users.
> : -=20
> : - =3D=3D Optimization Considerations =3D=3D
> : -=20
> : - You may want to optimize an index whenever practical -- ie: if you =
build your index once, and then never modify it.
> : -=20
> : - If your index is receiving a steady stream of modifications, then =
consider the following factors...
> : -=20
> : -    * As more segments are added to the index, query performace =
will degrade slightly.  Automatic segment merging by Lucene will set an =
upper bound on the number of segments created though.
> : -    * Auto-warming time will grow since it's normally dependent on =
doing searches.=20
> : -    * The first distribution after an optimization will take longer =
than subsequent ones. See [[CollectionDistribution|Collection =
Distribution]] for more information.
> : -    * During optimization the file size of the index doubles, but =
returns to it's original size or even slightly less.
> : -    * If you can, make sure that you do not have multiple =
concurrent producers of documents calling commit(). Multiple concurrent =
commits will cause a large performance degradation.=20
> : -=20
> : - Since optimizing an index saves all the segments in an index =
(about 7 files per segment) into a single segment, optimizing an index =
helps avoid the "too many open files" problem, i.e. running out of file =
descriptors, which is mentioned in an =
[[http://www.onjava.com/pub/a/onjava/2003/03/05/lucene.html#indexing_speed=
|ONJava Article]].
> :  =20
> :   =3D=3D Updates and Commit Frequency Tradeoffs =3D=3D
> :  =20
> :=20
>=20
> -Hoss
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>=20

- Mark Miller
lucidimagination.com


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org