Return-Path: Delivered-To: apmail-lucene-hadoop-user-archive@locus.apache.org Received: (qmail 42837 invoked from network); 23 Apr 2007 16:47:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 23 Apr 2007 16:47:15 -0000 Received: (qmail 91453 invoked by uid 500); 23 Apr 2007 16:47:20 -0000 Delivered-To: apmail-lucene-hadoop-user-archive@lucene.apache.org Received: (qmail 91415 invoked by uid 500); 23 Apr 2007 16:47:20 -0000 Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-user@lucene.apache.org Delivered-To: mailing list hadoop-user@lucene.apache.org Received: (qmail 91374 invoked by uid 99); 23 Apr 2007 16:47:20 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Apr 2007 09:47:20 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (herse.apache.org: local policy) Received: from [209.151.94.5] (HELO megs8.100mwh.com) (209.151.94.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Apr 2007 09:47:11 -0700 Received: from meetsawglad.corp.yahoo.com ([66.228.162.219]) by megs8.100mwh.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.63) (envelope-from ) id 1Hg1gi-00088U-98 for hadoop-user@lucene.apache.org; Mon, 23 Apr 2007 10:46:52 -0600 Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: <4d362c350704230739y75b507e5gc2c5f5e4bf98f5f3@mail.gmail.com> References: <4d362c350704230739y75b507e5gc2c5f5e4bf98f5f3@mail.gmail.com> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Owen O'Malley Subject: Re: Benchmarking question - how do you test a new cluster? Date: Mon, 23 Apr 2007 09:46:48 -0700 To: hadoop-user@lucene.apache.org X-Mailer: Apple Mail (2.752.2) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - megs8.100mwh.com X-AntiAbuse: Original Domain - lucene.apache.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - yahoo-inc.com X-Source: X-Source-Args: X-Source-Dir: X-Virus-Checked: Checked by ClamAV on apache.org On Apr 23, 2007, at 7:39 AM, Steve Schlosser wrote: > I've got a small hadoop cluster running (5 nodes today, going to 15+ > soon), and I'd like to do some benchmarking. My question to the group > is - what is the first benchmark you run on a new cluster? I usually use random-writer to generate some random data (it defaults to 10g/node) and then use sort to sort it. Sort provides a pretty decent simple testcase for moving a lot of data through map/reduce. -- Owen