From java-user-return-19139-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org Thu Feb 23 19:48:47 2006 Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 54377 invoked from network); 23 Feb 2006 19:48:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 23 Feb 2006 19:48:46 -0000 Received: (qmail 39701 invoked by uid 500); 23 Feb 2006 19:48:39 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 39668 invoked by uid 500); 23 Feb 2006 19:48:39 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 39657 invoked by uid 99); 23 Feb 2006 19:48:39 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Feb 2006 11:48:39 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of clamprecht@gmail.com designates 64.233.182.197 as permitted sender) Received: from [64.233.182.197] (HELO nproxy.gmail.com) (64.233.182.197) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Feb 2006 11:48:38 -0800 Received: by nproxy.gmail.com with SMTP id x29so125675nfb for ; Thu, 23 Feb 2006 11:48:16 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=pEqxFxa4PTHmOOcbK4KfcDfjq+gfXk4j6+OCUtld5KasJiG1JNwOuGgpAvq4JxIEZOyS9S3Os3Ut8FwDYn8ywwhs2hHLmeAXf2WHt66IwowtX0eDzyBdUHSq1jqd8d4D5hFs2rtku0767bxg/sK/3CJxLAE3tF1Q+hdv9419plg= Received: by 10.49.2.4 with SMTP id e4mr159569nfi; Thu, 23 Feb 2006 11:48:16 -0800 (PST) Received: by 10.49.8.7 with HTTP; Thu, 23 Feb 2006 11:48:16 -0800 (PST) Message-ID: <88c6a6720602231148v520b3be6uab8cfd7fb48899a9@mail.gmail.com> Date: Thu, 23 Feb 2006 13:48:16 -0600 From: "Chris Lamprecht" To: java-user@lucene.apache.org Subject: Re: Throughput doesn't increase when using more concurrent threads In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <43D92935.1000809@apache.org> <43D92A28.6050501@apache.org> <43DD6821.8020302@nuix.com.au> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Peter, Have you given JRockit JVM a try? I've seen it help throughput compared to Sun's JVM on a dual xeon/linux machine, especially with concurrency (up to 6 concurrent searches happening). I'm curious to see if it makes a difference for you. -chris On 2/23/06, Peter Keegan wrote: > We discovered that the kernel was only using 8 CPUs. After recompiling fo= r > 16 (8+hyperthreads), it looks like the query rate will settle in around > 280-300 qps. Much better, although still quite a bit slower than the > opteron. > > Peter > > > > > On 2/22/06, Yonik Seeley wrote: > > > > Hmmm, not sure what that could be. > > You could try using the default FSDir instead of MMapDir to see if the > > differences are there. > > > > Some things that could be different: > > - thread scheduling (shouldn't make too much of a difference though) > > - synchronization workings > > - page replacement policy... how to figure out what pages to swap in > > and which to swap out, esp of the memory mapped files. > > > > You could also try a profiler on both platforms to try and see where > > the difference is. > > > > -Yonik > > > > On 2/22/06, Peter Keegan wrote: > > > I am doing a performance comparison of Lucene on Linux vs Windows. > > > > > > I have 2 identically configured servers (8-CPUs (real) x 3GHz Xeon > > > processors, 64GB RAM). One is running CentOS 4 Linux, the other is > > running > > > Windows server 2003 Enterprise Edition x64. Both have 64-bit JVMs fro= m > > Sun. > > > The Lucene server is using MMapDirectory. I'm running the jvm with > > > -Xmx16000M. Peak memory usage of the jvm on Linux is about 6GB and 7.= 8GBon > > > windows. > > > > > > I'm observing query rates of 330 queries/sec on the Wintel server, bu= t > > only > > > 200 qps on the Linux box. At first, I suspected a network bottleneck, > > but > > > when I 'short-circuited' Lucene, the query rates were identical. > > > > > > I suspect that there are some things to be tuned in Linux, but I'm no= t > > sure > > > what. Any advice would be appreciated. > > > > > > Peter > > > > > > > > > > > > On 1/30/06, Peter Keegan wrote: > > > > > > > > I cranked up the dial on my query tester and was able to get the ra= te > > up > > > > to 325 qps. Unfortunately, the machine died shortly thereafter (mem= ory > > > > errors :-( ) Hopefully, it was just a coincidence. I haven't measur= ed > > 64-bit > > > > indexing speed, yet. > > > > > > > > Peter > > > > > > > > On 1/29/06, Daniel Noll wrote: > > > > > > > > > > Peter Keegan wrote: > > > > > > I tried the AMD64-bit JVM from Sun and with MMapDirectory and I= 'm > > now > > > > > > getting 250 queries/sec and excellent cpu utilization (equal > > > > > concurrency on > > > > > > all cpus)!! Yonik, thanks for the pointer to the 64-bit jvm. I > > wasn't > > > > > aware > > > > > > of it. > > > > > > > > > > > Wow. That's fast. > > > > > > > > > > Out of interest, does indexing time speed up much on 64-bit > > hardware? > > > > > I'm particularly interested in this side of things because for ou= r > > own > > > > > application, any query response under half a second is good enoug= h, > > but > > > > > the indexing side could always be faster. :-) > > > > > > > > > > Daniel > > > > > > > > > > -- > > > > > Daniel Noll > > > > > > > > > > Nuix Australia Pty Ltd > > > > > Suite 79, 89 Jones St, Ultimo NSW 2007, Australia > > > > > Phone: (02) 9280 0699 > > > > > Fax: (02) 9212 6902 > > > > > > > > > > This message is intended only for the named recipient. If you are > > not > > > > > the intended recipient you are notified that disclosing, copying, > > > > > distributing or taking any action in reliance on the contents of > > this > > > > > message or attachment is strictly prohibited. > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > > > > > For additional commands, e-mail: java-user-help@lucene.apache.org > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > > For additional commands, e-mail: java-user-help@lucene.apache.org > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org