lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Engels" <reng...@ix.netcom.com>
Subject RE: FieldsReader synchronized access vs. ThreadLocal ?
Date Fri, 19 May 2006 07:50:45 GMT
Given that Lucene is generally VERY CPU bound, having stalled processors
implies that those threads (and more) are blocked, either by IO, or by a
synchronized block - as long as you have more threads than processors.

If the machine has a POOR disk subsystem in comparison to the CPU speed, and
the OS disk cache is too small, you can easily stall the threads.

-----Original Message-----
From: Peter Keegan [mailto:peterlkeegan@gmail.com]
Sent: Thursday, May 18, 2006 1:32 PM
To: java-dev@lucene.apache.org; rengels@ix.netcom.com
Subject: Re: FieldsReader synchronized access vs. ThreadLocal ?


I'm returning 20 results (about .5Kb each). In fact, I had to reduce that
from 50 because the network was becoming the bottleneck.

On the 16-cpu server, I ran tests using 8, 16 and 32 query threads, but
there was no improvement with more threads. I still believe the hardware was
to blame.

Peter

On 5/18/06, Robert Engels <rengels@ix.netcom.com> wrote:
>
> As someone else pointed out, the proposed mods will only affect queries
> the
> return a lot of Documents. If your test is only set up to return a few
> documents (or possible none at all), then you will see no difference.
>
> The fact that some of the CPUs were far less than 100%, and others were at
> 100% may be a good sign. How any query threads were you testing with?
>
> -----Original Message-----
> From: Peter Keegan [mailto:peterlkeegan@gmail.com]
> Sent: Thursday, May 18, 2006 1:01 PM
> To: java-dev@lucene.apache.org; rengels@ix.netcom.com
> Subject: Re: FieldsReader synchronized access vs. ThreadLocal ?
>
>
> Robert,
>
> Sorry I missed your questions.
>
> The test results seem hard to believe. Doubling the CPUs only increased
> > through put by 20%??? Seems rather low for primarily a "read only" test.
>
>
> I think this refers to the test I did on a 16 cpu (32 hyperthreaded)
> server.
> This system was actually two 8 cpu systems cabled together on their
> backplanes. I suspect that some tradeoffs were made in its design that
> allowed for this flexibility which resulted in the minimal improvement in
> the tests.
>
> Peter did not seem to answer many of the follow-up questions (at least I
> > could not find the answers) regarding whether or not the CPU usage was
> > 100%.
>
>
> On the 16-cpu system I noticed that load was not distributed very evenly -
> some were near 100%, others were less than 10%. On the AMD Opteron
> servers,
> the distribution was quite even and between 75-100%.
>
> I look forward to your thoughts, and others - hopefully someone can run
> the
> > test on a multiple CPU machine.
> >
> >
> I built Lucene with your mod's and ran my test on the 8 cpu AMD Linux
> server, but noticed no difference in query throughput. It would seem that
> ThreadLocal could improve performance, but I think my bottlenecks are
> elsewhere, like IndexInput.readVInt and inserting results in priority
> queues.
>
> Peter
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message