lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Engels" <reng...@ix.netcom.com>
Subject RE: FieldsReader synchronized access vs. ThreadLocal ?
Date Fri, 19 May 2006 08:17:31 GMT
As an aside,

On my VERY crappy 1.2 ghz single CPU P4, using a index of 300k documents, I
can perform 50 searches per second (returning 20 document matches each).
This includes the time to serialize and send the results to the client
(although the client is on the same machine, but it also competes for cpu
with the search server).

Based on some informal viewing of the CPU usage, the client consume 50-70%
of the cpu, so I would assume that moving the client off the server should
double the queries per second (although there would be additional delay due
to network transmission). So even a single CPU P4 could easily do 100
queries per second.

Even though we are comparing apples and oranges, unless you are performing
some really expensive queries, I would expect your configuration to be MUCH
faster.

-----Original Message-----
From: Peter Keegan [mailto:peterlkeegan@gmail.com]
Sent: Thursday, May 18, 2006 1:32 PM
To: java-dev@lucene.apache.org; rengels@ix.netcom.com
Subject: Re: FieldsReader synchronized access vs. ThreadLocal ?


I'm returning 20 results (about .5Kb each). In fact, I had to reduce that
from 50 because the network was becoming the bottleneck.

On the 16-cpu server, I ran tests using 8, 16 and 32 query threads, but
there was no improvement with more threads. I still believe the hardware was
to blame.

Peter

On 5/18/06, Robert Engels <rengels@ix.netcom.com> wrote:
>
> As someone else pointed out, the proposed mods will only affect queries
> the
> return a lot of Documents. If your test is only set up to return a few
> documents (or possible none at all), then you will see no difference.
>
> The fact that some of the CPUs were far less than 100%, and others were at
> 100% may be a good sign. How any query threads were you testing with?
>
> -----Original Message-----
> From: Peter Keegan [mailto:peterlkeegan@gmail.com]
> Sent: Thursday, May 18, 2006 1:01 PM
> To: java-dev@lucene.apache.org; rengels@ix.netcom.com
> Subject: Re: FieldsReader synchronized access vs. ThreadLocal ?
>
>
> Robert,
>
> Sorry I missed your questions.
>
> The test results seem hard to believe. Doubling the CPUs only increased
> > through put by 20%??? Seems rather low for primarily a "read only" test.
>
>
> I think this refers to the test I did on a 16 cpu (32 hyperthreaded)
> server.
> This system was actually two 8 cpu systems cabled together on their
> backplanes. I suspect that some tradeoffs were made in its design that
> allowed for this flexibility which resulted in the minimal improvement in
> the tests.
>
> Peter did not seem to answer many of the follow-up questions (at least I
> > could not find the answers) regarding whether or not the CPU usage was
> > 100%.
>
>
> On the 16-cpu system I noticed that load was not distributed very evenly -
> some were near 100%, others were less than 10%. On the AMD Opteron
> servers,
> the distribution was quite even and between 75-100%.
>
> I look forward to your thoughts, and others - hopefully someone can run
> the
> > test on a multiple CPU machine.
> >
> >
> I built Lucene with your mod's and ran my test on the 8 cpu AMD Linux
> server, but noticed no difference in query throughput. It would seem that
> ThreadLocal could improve performance, but I think my bottlenecks are
> elsewhere, like IndexInput.readVInt and inserting results in priority
> queues.
>
> Peter
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message