lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Klaas <mike.kl...@gmail.com>
Subject Re: [jira] Created: (LUCENE-1172) Small speedups to DocumentsWriter
Date Mon, 11 Feb 2008 07:07:27 GMT
Certainly others do agree with you to some degree that this case is  
on the cost/benefit borderline.  Again, this case wasn't really the  
point.

My point was it feels to me that you have, on occasion, been over- 
quick to criticize without paying sufficiently respectful attention  
to the details of what is being discussed.  For instance, the  
criticism of "these tests should be done on a *nix platform" to  
someone who has repeated the tests on osx (yes, a nix) and windows.   
Or that the test is too short and the index in memory (it was 10MM  
docs with term vecs on FSDirectory.  It is possible that some of the  
index wasn't fsync'd at the end of each test, I suppose, but I would  
expect this to be a small amount and equivalent in pre- and post- 
patch scenarios).  Or calling a full index run of 10MM docs a "micro  
benchmark".

I do think that I was unchill in sending the original post to the  
list instead of to you via personal mail.  I shouldn't have.

regards,
-Mike

On 10-Feb-08, at 7:33 PM, robert engels wrote:

> Please chill. You are inferring something that was not implied. You  
> may think it lacks perspective and respect (I disagree on both),  
> but it certainly doesn't lack in correctness.
>
> First, depending on how you measure it, 2x speedup equates to a 50%  
> reduction in time. In my review of the changes that brought about  
> the biggest performance gains from 1.9 on, almost all were related  
> to avoiding disk accesses by buffering more documents and doing  
> more processing in memory.  I don't think many of the micro- 
> benchmarks mattered much, and with a JVM environment it is very  
> difficult to prove as it is going to be heavily JVM and  
> configuration dependent.
>
> The main point was that ANY disk access is going to be ORDERS OF  
> MAGNITUDE slower than any of these sort of optimizations.
>
> So either you are loading the index completely in memory (only  
> small indexes, so the difference in speed is not going to matter  
> much), or you might be using a federated system of memory indices  
> (to form a large index), but USUALLY at some point the index must  
> be first created in a persistent store (that which is covered  
> here), in order to provide realistic restart times, etc.
>
> The author of the patch and timings gives no information as to disk  
> speed, IO speed, controllers, raid configuration , etc. When  
> creating an index in persistent store, these factors matter more  
> than a 2-4% speed up.  Creating an index completely in memory is  
> then bound by the reading of the data from the disk, and/or the  
> network - all much slower than the actual indexing.
>
> Usually optimizations like this only matter in areas of development  
> where the data set is small, but the processing large (a lot of  
> numerical analysis).  In some cases the data set may also be  
> "large", but then usually the processing is exponentially larger.   
> The building of the index in Lucene in not very computationally  
> expensive.
>
> If you are going spend hundreds of hours "optimizing", you best be  
> optimizing the right things. That was the point of the link I sent  
> (the quotes are from people far more capable than I).
>
> I was trying to make the point that a 2-4 % speed up probably  
> doesn't amount to much in a real environment given all of the other  
> factors, so it is probably better for the project/community to err  
> on the side of code clarity and ease of maintenance.
>
> The project can continue to do what it wants (obviously) - but what  
> I was pointing out should be nothing new to experienced designers/ 
> developers - I only offering a reminder. It is my observation  
> (others will disagree !), but I think a lot of Lucene has some  
> unneeded esoteric code, where the benefit doesn't match the cost.
>
> On Feb 10, 2008, at 5:48 PM, Mike Klaas wrote:
>
>> While I agree in general that excessive optimization at the  
>> expense of code clarity is undesirable, you are overstating the  
>> point.  2X is a ridiculous threshold to apply to something as  
>> performance critical as a full text search engine.  If search was  
>> twice as slow, lucene would be utterly unusable for me.  Indexing  
>> less important than search, of course, but a 2X slowdown with be  
>> quite painful there.
>>
>> I don't have an opinion in this case: I believe that there is a  
>> tradeoff but that it is the responsibility of the commiter(s) to  
>> achieve the correct balance--they are the ones who will be  
>> maintaining the code, after all.  I find your persistence  
>> surprising and your tone dangerously near condescending.  Telling  
>> the guy who has spent hundreds of hours carefully optimizing this  
>> code that "Almost always there is a better bottleneck somewhere"  
>> shows an astonishing lack of perspective and respect.
>>
>> -Mike
>>
>> On 10-Feb-08, at 12:15 PM, robert engels wrote:
>>
>>> I am not sure these numbers matter. I think they are skewed  
>>> because you are probably running too short a test, and the index  
>>> is in memory (or OS cache).
>>>
>>> Once you use a real index that needs to read/write from the disk,  
>>> the percentage change will be negligible.
>>>
>>> This is the problem with many of these "performance changes" -  
>>> they just aren't real world enough.  Even if they were, I would  
>>> argue that code simplicity/maintainability is worth more than 6  
>>> seconds on a operation that takes 4 minutes to run...
>>>
>>> There are many people that believe micro benchmarks are next to  
>>> worthless. A good rule of thumb is that if the optimization  
>>> doesn't result in 2x speedup, it probably shouldn't be done. In  
>>> most cases any efficiency gains are later lost in maintainability  
>>> issues.
>>>
>>> See http://en.wikipedia.org/wiki/Optimization_(computer_science)
>>>
>>> Almost always there is a better bottleneck somewhere.
>>>
>>> On Feb 10, 2008, at 1:37 PM, Michael McCandless wrote:
>>>
>>>>
>>>> Yonik Seeley wrote:
>>>>
>>>>> I wonder how well a single generic quickSort(Object[] arr, int  
>>>>> low,
>>>>> int high) would perform vs the type-specific ones?  I guess the  
>>>>> main
>>>>> overhead would be a cast from Object to the specific class to  
>>>>> do the
>>>>> compare?  Too bad Java doesn't have true generics/templates.
>>>>
>>>>
>>>> OK I tested this.
>>>>
>>>> Starting from the patch on LUCENE-1172, which has 3 quickSort  
>>>> methods
>>>> (one per type), I created a single quickSort method on Object[]  
>>>> that
>>>> takes a Comparator, and made 3 Comparators instead.
>>>>
>>>> Mac OS X 10.4 (JVM 1.5):
>>>>
>>>>     original patch --> 247.1
>>>>   simplified patch --> 254.9 (3.2% slower)
>>>>
>>>> Windows Server 2003 R64 (JVM 1.6):
>>>>
>>>>     original patch --> 440.6
>>>>   simplified patch --> 452.7 (2.7% slower)
>>>>
>>>> The times are best in 10 runs.  I'm running all tests with these  
>>>> JVM
>>>> args:
>>>>
>>>>   -Xms1024M -Xmx1024M -Xbatch -server
>>>>
>>>> I think this is a big enough difference in performance that it's
>>>> worth keeping 3 separate quickSorts in DocumentsWriter.
>>>>
>>>> Mike
>>>>
>>>> ------------------------------------------------------------------- 
>>>> --
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>
>>>
>>>
>>> -------------------------------------------------------------------- 
>>> -
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message