From java-dev-return-16510-apmail-lucene-java-dev-archive=lucene.apache.org@lucene.apache.org Mon Nov 13 20:50:58 2006 Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 82069 invoked from network); 13 Nov 2006 20:50:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 13 Nov 2006 20:50:56 -0000 Received: (qmail 85283 invoked by uid 500); 13 Nov 2006 20:51:04 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 85233 invoked by uid 500); 13 Nov 2006 20:51:04 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 85219 invoked by uid 99); 13 Nov 2006 20:51:03 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Nov 2006 12:51:03 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (herse.apache.org: local policy) Received: from [216.148.227.154] (HELO rwcrmhc14.comcast.net) (216.148.227.154) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Nov 2006 12:50:49 -0800 Received: from [192.168.168.15] (c-71-202-24-246.hsd1.ca.comcast.net[71.202.24.246]) by comcast.net (rwcrmhc14) with ESMTP id <20061113205028m14002m98je>; Mon, 13 Nov 2006 20:50:28 +0000 Message-ID: <4558DA94.6060000@apache.org> Date: Mon, 13 Nov 2006 12:50:28 -0800 From: Doug Cutting User-Agent: Thunderbird 1.5.0.7 (X11/20060922) MIME-Version: 1.0 To: java-dev@lucene.apache.org Subject: Re: ParallelMultiSearcher reimplementation References: <13F36D2926AD444697093282F20041B704F97C34@buf-ex01.cymfony.com> <454BB807.9080906@manawiz.com> <454BC01A.8080804@apache.org> <454E3331.9030009@manawiz.com> In-Reply-To: <454E3331.9030009@manawiz.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Chuck Williams wrote: > I followed this same logic in ParallelWriter and got burned. My first > implementation (still the version submitted as a patch in jira) used > dynamic threads to add the subdocuments to the parallel subindexes > simultaneously. This hit a problem with abnormal native heap OOM's in > the jvm. At first I thought it was simply a thread stack size / java > heap size configuration issue, but adjusting these did not resolve the > issue. This was on linux. ps -L showed large numbers of defunct > threads. jconsole showed enormous growing total-ever-allocated thread > counts. I switched to a thread pool and the issue went away with the > same config settings. Can you demonstrate the problem with a standalone program? Way back in the 90's I implemented a system at Excite that spawned one or more Java threads per request, and it ran for days on end, handling 20 or more requests per second. The thread spawning overhead was insignificant. That was JDK 1.2 on Solaris. Have things gotten that much worse in the interim? Today Hadoop's RPC allocates a thread per connection, and we see good performance. So I certainly have counterexamples. Doug --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org For additional commands, e-mail: java-dev-help@lucene.apache.org