lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: [jira] Created: (LUCENE-1172) Small speedups to DocumentsWriter
Date Mon, 11 Feb 2008 08:39:30 GMT
The reason it needs (or should be done) on Unix, is that it is much  
easier (and better I think) at reporting the "real" timings. What the  
reporter stated was in (most likely) real time? which is not the best  
way to measure performance - especially on multi user/tasking OSes.

The unix time facilities give a better picture of exactly why the  
program took the amount of time to execute.


On Feb 11, 2008, at 1:07 AM, Mike Klaas wrote:

> Certainly others do agree with you to some degree that this case is  
> on the cost/benefit borderline.  Again, this case wasn't really the  
> point.
>
> My point was it feels to me that you have, on occasion, been over- 
> quick to criticize without paying sufficiently respectful attention  
> to the details of what is being discussed.  For instance, the  
> criticism of "these tests should be done on a *nix platform" to  
> someone who has repeated the tests on osx (yes, a nix) and  
> windows.  Or that the test is too short and the index in memory (it  
> was 10MM docs with term vecs on FSDirectory.  It is possible that  
> some of the index wasn't fsync'd at the end of each test, I  
> suppose, but I would expect this to be a small amount and  
> equivalent in pre- and post-patch scenarios).  Or calling a full  
> index run of 10MM docs a "micro benchmark".
>
> I do think that I was unchill in sending the original post to the  
> list instead of to you via personal mail.  I shouldn't have.
>
> regards,
> -Mike
>
> On 10-Feb-08, at 7:33 PM, robert engels wrote:
>
>> Please chill. You are inferring something that was not implied.  
>> You may think it lacks perspective and respect (I disagree on  
>> both), but it certainly doesn't lack in correctness.
>>
>> First, depending on how you measure it, 2x speedup equates to a  
>> 50% reduction in time. In my review of the changes that brought  
>> about the biggest performance gains from 1.9 on, almost all were  
>> related to avoiding disk accesses by buffering more documents and  
>> doing more processing in memory.  I don't think many of the micro- 
>> benchmarks mattered much, and with a JVM environment it is very  
>> difficult to prove as it is going to be heavily JVM and  
>> configuration dependent.
>>
>> The main point was that ANY disk access is going to be ORDERS OF  
>> MAGNITUDE slower than any of these sort of optimizations.
>>
>> So either you are loading the index completely in memory (only  
>> small indexes, so the difference in speed is not going to matter  
>> much), or you might be using a federated system of memory indices  
>> (to form a large index), but USUALLY at some point the index must  
>> be first created in a persistent store (that which is covered  
>> here), in order to provide realistic restart times, etc.
>>
>> The author of the patch and timings gives no information as to  
>> disk speed, IO speed, controllers, raid configuration , etc. When  
>> creating an index in persistent store, these factors matter more  
>> than a 2-4% speed up.  Creating an index completely in memory is  
>> then bound by the reading of the data from the disk, and/or the  
>> network - all much slower than the actual indexing.
>>
>> Usually optimizations like this only matter in areas of  
>> development where the data set is small, but the processing large  
>> (a lot of numerical analysis).  In some cases the data set may  
>> also be "large", but then usually the processing is exponentially  
>> larger.  The building of the index in Lucene in not very  
>> computationally expensive.
>>
>> If you are going spend hundreds of hours "optimizing", you best be  
>> optimizing the right things. That was the point of the link I sent  
>> (the quotes are from people far more capable than I).
>>
>> I was trying to make the point that a 2-4 % speed up probably  
>> doesn't amount to much in a real environment given all of the  
>> other factors, so it is probably better for the project/community  
>> to err on the side of code clarity and ease of maintenance.
>>
>> The project can continue to do what it wants (obviously) - but  
>> what I was pointing out should be nothing new to experienced  
>> designers/developers - I only offering a reminder. It is my  
>> observation (others will disagree !), but I think a lot of Lucene  
>> has some unneeded esoteric code, where the benefit doesn't match  
>> the cost.
>>
>> On Feb 10, 2008, at 5:48 PM, Mike Klaas wrote:
>>
>>> While I agree in general that excessive optimization at the  
>>> expense of code clarity is undesirable, you are overstating the  
>>> point.  2X is a ridiculous threshold to apply to something as  
>>> performance critical as a full text search engine.  If search was  
>>> twice as slow, lucene would be utterly unusable for me.  Indexing  
>>> less important than search, of course, but a 2X slowdown with be  
>>> quite painful there.
>>>
>>> I don't have an opinion in this case: I believe that there is a  
>>> tradeoff but that it is the responsibility of the commiter(s) to  
>>> achieve the correct balance--they are the ones who will be  
>>> maintaining the code, after all.  I find your persistence  
>>> surprising and your tone dangerously near condescending.  Telling  
>>> the guy who has spent hundreds of hours carefully optimizing this  
>>> code that "Almost always there is a better bottleneck somewhere"  
>>> shows an astonishing lack of perspective and respect.
>>>
>>> -Mike
>>>
>>> On 10-Feb-08, at 12:15 PM, robert engels wrote:
>>>
>>>> I am not sure these numbers matter. I think they are skewed  
>>>> because you are probably running too short a test, and the index  
>>>> is in memory (or OS cache).
>>>>
>>>> Once you use a real index that needs to read/write from the  
>>>> disk, the percentage change will be negligible.
>>>>
>>>> This is the problem with many of these "performance changes" -  
>>>> they just aren't real world enough.  Even if they were, I would  
>>>> argue that code simplicity/maintainability is worth more than 6  
>>>> seconds on a operation that takes 4 minutes to run...
>>>>
>>>> There are many people that believe micro benchmarks are next to  
>>>> worthless. A good rule of thumb is that if the optimization  
>>>> doesn't result in 2x speedup, it probably shouldn't be done. In  
>>>> most cases any efficiency gains are later lost in  
>>>> maintainability issues.
>>>>
>>>> See http://en.wikipedia.org/wiki/Optimization_(computer_science)
>>>>
>>>> Almost always there is a better bottleneck somewhere.
>>>>
>>>> On Feb 10, 2008, at 1:37 PM, Michael McCandless wrote:
>>>>
>>>>>
>>>>> Yonik Seeley wrote:
>>>>>
>>>>>> I wonder how well a single generic quickSort(Object[] arr, int  
>>>>>> low,
>>>>>> int high) would perform vs the type-specific ones?  I guess  
>>>>>> the main
>>>>>> overhead would be a cast from Object to the specific class to  
>>>>>> do the
>>>>>> compare?  Too bad Java doesn't have true generics/templates.
>>>>>
>>>>>
>>>>> OK I tested this.
>>>>>
>>>>> Starting from the patch on LUCENE-1172, which has 3 quickSort  
>>>>> methods
>>>>> (one per type), I created a single quickSort method on Object[]  
>>>>> that
>>>>> takes a Comparator, and made 3 Comparators instead.
>>>>>
>>>>> Mac OS X 10.4 (JVM 1.5):
>>>>>
>>>>>     original patch --> 247.1
>>>>>   simplified patch --> 254.9 (3.2% slower)
>>>>>
>>>>> Windows Server 2003 R64 (JVM 1.6):
>>>>>
>>>>>     original patch --> 440.6
>>>>>   simplified patch --> 452.7 (2.7% slower)
>>>>>
>>>>> The times are best in 10 runs.  I'm running all tests with  
>>>>> these JVM
>>>>> args:
>>>>>
>>>>>   -Xms1024M -Xmx1024M -Xbatch -server
>>>>>
>>>>> I think this is a big enough difference in performance that it's
>>>>> worth keeping 3 separate quickSorts in DocumentsWriter.
>>>>>
>>>>> Mike
>>>>>
>>>>> ------------------------------------------------------------------ 
>>>>> ---
>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------- 
>>>> --
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>
>>>
>>>
>>> -------------------------------------------------------------------- 
>>> -
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message