lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Engels" <reng...@ix.netcom.com>
Subject RE: FieldsReader synchronized access vs. ThreadLocal ?
Date Fri, 19 May 2006 14:17:41 GMT
If you can get much higher throughput from a single threaded client, then
you should queue the requests in the server and process them from a single
thread (or a small pool of threads).

If you can get much higher throughput from a single threaded client seems to
also imply that Lucene (or at least your packaging) is NOT very concurrent
(since more threads actually reduce the efficiency).

-----Original Message-----
From: Peter Keegan [mailto:peterlkeegan@gmail.com]
Sent: Friday, May 19, 2006 9:03 AM
To: java-dev@lucene.apache.org; rengels@ix.netcom.com
Subject: Re: FieldsReader synchronized access vs. ThreadLocal ?


The queries are mostly boolean (all AND'd terms), no. of terms varies
anywhere from a few to 25 or more, with 1 or 2 sort fields.

My tests are designed to measure total query throughput, not just raw search
speed. The client test program blasts queries from >50 threads over a socket
and runs on a separate server from Lucene. I can get much higher rates by
just blasting from a single thread in the client, but this doesn't simulate
the real use model.

Peter

On 5/19/06, Robert Engels <rengels@ix.netcom.com> wrote:
>
> As an aside,
>
> On my VERY crappy 1.2 ghz single CPU P4, using a index of 300k documents,
> I
> can perform 50 searches per second (returning 20 document matches each).
> This includes the time to serialize and send the results to the client
> (although the client is on the same machine, but it also competes for cpu
> with the search server).
>
> Based on some informal viewing of the CPU usage, the client consume 50-70%
> of the cpu, so I would assume that moving the client off the server should
> double the queries per second (although there would be additional delay
> due
> to network transmission). So even a single CPU P4 could easily do 100
> queries per second.
>
> Even though we are comparing apples and oranges, unless you are performing
> some really expensive queries, I would expect your configuration to be
> MUCH
> faster.
>
> -----Original Message-----
> From: Peter Keegan [mailto:peterlkeegan@gmail.com]
> Sent: Thursday, May 18, 2006 1:32 PM
> To: java-dev@lucene.apache.org; rengels@ix.netcom.com
> Subject: Re: FieldsReader synchronized access vs. ThreadLocal ?
>
>
> I'm returning 20 results (about .5Kb each). In fact, I had to reduce that
> from 50 because the network was becoming the bottleneck.
>
> On the 16-cpu server, I ran tests using 8, 16 and 32 query threads, but
> there was no improvement with more threads. I still believe the hardware
> was
> to blame.
>
> Peter
>
> On 5/18/06, Robert Engels <rengels@ix.netcom.com> wrote:
> >
> > As someone else pointed out, the proposed mods will only affect queries
> > the
> > return a lot of Documents. If your test is only set up to return a few
> > documents (or possible none at all), then you will see no difference.
> >
> > The fact that some of the CPUs were far less than 100%, and others were
> at
> > 100% may be a good sign. How any query threads were you testing with?
> >
> > -----Original Message-----
> > From: Peter Keegan [mailto:peterlkeegan@gmail.com]
> > Sent: Thursday, May 18, 2006 1:01 PM
> > To: java-dev@lucene.apache.org; rengels@ix.netcom.com
> > Subject: Re: FieldsReader synchronized access vs. ThreadLocal ?
> >
> >
> > Robert,
> >
> > Sorry I missed your questions.
> >
> > The test results seem hard to believe. Doubling the CPUs only increased
> > > through put by 20%??? Seems rather low for primarily a "read only"
> test.
> >
> >
> > I think this refers to the test I did on a 16 cpu (32 hyperthreaded)
> > server.
> > This system was actually two 8 cpu systems cabled together on their
> > backplanes. I suspect that some tradeoffs were made in its design that
> > allowed for this flexibility which resulted in the minimal improvement
> in
> > the tests.
> >
> > Peter did not seem to answer many of the follow-up questions (at least I
> > > could not find the answers) regarding whether or not the CPU usage was
> > > 100%.
> >
> >
> > On the 16-cpu system I noticed that load was not distributed very evenly
> -
> > some were near 100%, others were less than 10%. On the AMD Opteron
> > servers,
> > the distribution was quite even and between 75-100%.
> >
> > I look forward to your thoughts, and others - hopefully someone can run
> > the
> > > test on a multiple CPU machine.
> > >
> > >
> > I built Lucene with your mod's and ran my test on the 8 cpu AMD Linux
> > server, but noticed no difference in query throughput. It would seem
> that
> > ThreadLocal could improve performance, but I think my bottlenecks are
> > elsewhere, like IndexInput.readVInt and inserting results in priority
> > queues.
> >
> > Peter
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message