Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: neutral (asf.osuosl.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=dk20050327; d=ix.netcom.com;
  b=d2EKyswUHTN3mRmfsZLDFs2d1QUBGLOBfcb6CwSvqxddeC6KG21eSKjQlDsm7zXm;
  h=Received:Reply-To:From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Priority:X-MSMail-Priority:X-Mailer:In-Reply-To:X-MimeOLE:Importance:X-ELNK-Trace:X-Originating-IP;
Reply-To: <rengels@ix.netcom.com>
From: "Robert Engels" <rengels@ix.netcom.com>
To: <java-dev@lucene.apache.org>
Subject: RE: FieldsReader synchronized access vs. ThreadLocal ?
Date: Fri, 19 May 2006 02:50:45 -0500
Message-ID: <LMENLAOACIBLMOIILNNNKEBPFOAA.rengels@ix.netcom.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
In-Reply-To: <e994873a0605181132k486a5e9ck6b8dce33b7789a73@mail.gmail.com>
Importance: Normal

Given that Lucene is generally VERY CPU bound, having stalled processors
implies that those threads (and more) are blocked, either by IO, or by a
synchronized block - as long as you have more threads than processors.

If the machine has a POOR disk subsystem in comparison to the CPU speed, and
the OS disk cache is too small, you can easily stall the threads.

-----Original Message-----
From: Peter Keegan [mailto:peterlkeegan@gmail.com]
Sent: Thursday, May 18, 2006 1:32 PM
To: java-dev@lucene.apache.org; rengels@ix.netcom.com
Subject: Re: FieldsReader synchronized access vs. ThreadLocal ?


I'm returning 20 results (about .5Kb each). In fact, I had to reduce that
from 50 because the network was becoming the bottleneck.

On the 16-cpu server, I ran tests using 8, 16 and 32 query threads, but
there was no improvement with more threads. I still believe the hardware was
to blame.

Peter

On 5/18/06, Robert Engels <rengels@ix.netcom.com> wrote:
>
> As someone else pointed out, the proposed mods will only affect queries
> the
> return a lot of Documents. If your test is only set up to return a few
> documents (or possible none at all), then you will see no difference.
>
> The fact that some of the CPUs were far less than 100%, and others were at
> 100% may be a good sign. How any query threads were you testing with?
>
> -----Original Message-----
> From: Peter Keegan [mailto:peterlkeegan@gmail.com]
> Sent: Thursday, May 18, 2006 1:01 PM
> To: java-dev@lucene.apache.org; rengels@ix.netcom.com
> Subject: Re: FieldsReader synchronized access vs. ThreadLocal ?
>
>
> Robert,
>
> Sorry I missed your questions.
>
> The test results seem hard to believe. Doubling the CPUs only increased
> > through put by 20%??? Seems rather low for primarily a "read only" test.
>
>
> I think this refers to the test I did on a 16 cpu (32 hyperthreaded)
> server.
> This system was actually two 8 cpu systems cabled together on their
> backplanes. I suspect that some tradeoffs were made in its design that
> allowed for this flexibility which resulted in the minimal improvement in
> the tests.
>
> Peter did not seem to answer many of the follow-up questions (at least I
> > could not find the answers) regarding whether or not the CPU usage was
> > 100%.
>
>
> On the 16-cpu system I noticed that load was not distributed very evenly -
> some were near 100%, others were less than 10%. On the AMD Opteron
> servers,
> the distribution was quite even and between 75-100%.
>
> I look forward to your thoughts, and others - hopefully someone can run
> the
> > test on a multiple CPU machine.
> >
> >
> I built Lucene with your mod's and ran my test on the 8 cpu AMD Linux
> server, but noticed no difference in query throughput. It would seem that
> ThreadLocal could improve performance, but I think my bottlenecks are
> elsewhere, like IndexInput.readVInt and inserting results in priority
> queues.
>
> Peter
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org