Return-Path: Delivered-To: apmail-lucene-hadoop-user-archive@locus.apache.org Received: (qmail 26959 invoked from network); 31 Dec 2007 06:22:43 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 31 Dec 2007 06:22:43 -0000 Received: (qmail 14765 invoked by uid 500); 31 Dec 2007 06:22:30 -0000 Delivered-To: apmail-lucene-hadoop-user-archive@lucene.apache.org Received: (qmail 14741 invoked by uid 500); 31 Dec 2007 06:22:30 -0000 Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-user@lucene.apache.org Delivered-To: mailing list hadoop-user@lucene.apache.org Received: (qmail 14732 invoked by uid 99); 31 Dec 2007 06:22:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Dec 2007 22:22:30 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [63.203.238.117] (HELO dns.duboce.net) (63.203.238.117) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Dec 2007 06:22:16 +0000 Received: by dns.duboce.net (Postfix, from userid 1008) id 802B3C51D; Sun, 30 Dec 2007 20:50:13 -0800 (PST) X-Spam-Checker-Version: SpamAssassin 3.1.4 (2006-07-26) on dns.duboce.net X-Spam-Level: Received: from durruti.local (unknown [192.168.1.135]) by dns.duboce.net (Postfix) with ESMTP id 3F163C1CE for ; Sun, 30 Dec 2007 20:50:11 -0800 (PST) Message-ID: <47788A16.8080105@duboce.net> Date: Sun, 30 Dec 2007 22:20:06 -0800 From: stack User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: hadoop-user@lucene.apache.org Subject: Re: Powerset + Hadoop @ Rapleaf References: <30c6373b0712301911q46b0c94dqf204678920a345b5@mail.gmail.com> In-Reply-To: <30c6373b0712301911q46b0c94dqf204678920a345b5@mail.gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No, score=-4.1 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.4 "There are also core design flaws. For example, they use threaded IO...This just won�t scale." FYI, Kevin, hbase puts up non-blocking server sockets to field client and intra-server communications (It uses Hadoop RPC). Client's of Hadoop's DFS -- e.g. mapreduce jobs, hbase, etc. -- use blocking thread-per-socket for swapping big data blocks. Reportedly, the latter has been sufficient substrate supporting clusters of thousands of computers. My guess is that when synchronous socket I/O becomes a bottleneck or a good case -- rather than a "gut feeling" -- can be made that this model is overly consumptive, changing the HDFS servers to use async I/O will become a priority. St.Ack Kevin Burton wrote: > With all the activity over the holidays I forgot to post this to the list... > > > http://feedblog.org/2007/12/18/powerset-hadoop-rapleaf/ >