Return-Path: Delivered-To: apmail-lucene-hadoop-user-archive@locus.apache.org Received: (qmail 19914 invoked from network); 21 Dec 2007 02:47:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 21 Dec 2007 02:47:34 -0000 Received: (qmail 79987 invoked by uid 500); 21 Dec 2007 02:47:21 -0000 Delivered-To: apmail-lucene-hadoop-user-archive@lucene.apache.org Received: (qmail 79968 invoked by uid 500); 21 Dec 2007 02:47:21 -0000 Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-user@lucene.apache.org Delivered-To: mailing list hadoop-user@lucene.apache.org Received: (qmail 79959 invoked by uid 99); 21 Dec 2007 02:47:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Dec 2007 18:47:21 -0800 X-ASF-Spam-Status: No, hits=3.2 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.191.90.79] (HELO web806.biz.mail.mud.yahoo.com) (209.191.90.79) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 21 Dec 2007 02:47:09 +0000 Received: (qmail 75153 invoked by uid 60001); 21 Dec 2007 02:46:58 -0000 X-YMail-OSG: TPlXIMwVM1lhpnaDiozYGDNj6o4IN_yIEXzyTcU9tUV4Dlny8SaSZSjBqJxHiNL3Y4xs_YPEEg-- Received: from [64.170.221.123] by web806.biz.mail.mud.yahoo.com via HTTP; Thu, 20 Dec 2007 18:46:58 PST Date: Thu, 20 Dec 2007 18:46:58 -0800 (PST) From: Kirk True Subject: Appropriate use of Hadoop for non-map/reduce tasks? To: hadoop-user@lucene.apache.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-501118047-1198205218=:74734" Content-Transfer-Encoding: 8bit Message-ID: <375870.74734.qm@web806.biz.mail.mud.yahoo.com> X-Virus-Checked: Checked by ClamAV on apache.org --0-501118047-1198205218=:74734 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Hi all, A lot of the ideas I have for incorporating Hadoop into internal projects revolves around distributing long-running tasks over multiple machines. I've been able to get a quick prototype up in Hadoop for one of those projects and it seems to work pretty well. However, in this project and the others, I'm not processing a lot of text or mapping or reducing anything. I'm basically asynchronously processing a lot of work over many machines in a master/worker paradigm rather than map/reduce. I have shown that I can achieve what I'm looking for with Hadoop. I just can't get over the "feeling" that I'm shoe-horning it into a use it wasn't really meant to do. We've done a similar project with Gigaspaces, but Hadoop seems to alleviate a lot of the burden of what we're doing moving forward. Thoughts? Thanks! Kirk --0-501118047-1198205218=:74734--