Return-Path: Delivered-To: apmail-lucene-hadoop-user-archive@locus.apache.org Received: (qmail 72279 invoked from network); 23 Apr 2007 18:00:32 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 23 Apr 2007 18:00:32 -0000 Received: (qmail 32367 invoked by uid 500); 23 Apr 2007 18:00:33 -0000 Delivered-To: apmail-lucene-hadoop-user-archive@lucene.apache.org Received: (qmail 32348 invoked by uid 500); 23 Apr 2007 18:00:33 -0000 Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-user@lucene.apache.org Delivered-To: mailing list hadoop-user@lucene.apache.org Received: (qmail 32339 invoked by uid 99); 23 Apr 2007 18:00:32 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Apr 2007 11:00:32 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [63.133.162.107] (HELO linuxfly.dragonflymc.com) (63.133.162.107) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Apr 2007 11:00:25 -0700 Received: from [192.168.1.246] (bigfly.visvo.com [192.168.1.246]) by linuxfly.dragonflymc.com (Postfix) with ESMTP id AB8CCC20027 for ; Mon, 23 Apr 2007 13:00:04 -0500 (CDT) Message-ID: <462CF425.3050708@dragonflymc.com> Date: Mon, 23 Apr 2007 13:00:05 -0500 From: Dennis Kubes User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: hadoop-user@lucene.apache.org Subject: Re: Benchmarking question - how do you test a new cluster? References: <4d362c350704230739y75b507e5gc2c5f5e4bf98f5f3@mail.gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org You will also want to run hardware tests on the machines in your cluster to make sure memory, disk, and network is working properly. We use tools such as memtest86+ and Doug Ledfors Memory Test Script to do burn ins. Here is a link to different test programs for linux. http://linuxquality.sunsite.dk/articles/testsuites/ Dennis Kubes Owen O'Malley wrote: > > On Apr 23, 2007, at 7:39 AM, Steve Schlosser wrote: > >> I've got a small hadoop cluster running (5 nodes today, going to 15+ >> soon), and I'd like to do some benchmarking. My question to the group >> is - what is the first benchmark you run on a new cluster? > > I usually use random-writer to generate some random data (it defaults to > 10g/node) and then use sort to sort it. Sort provides a pretty decent > simple testcase for moving a lot of data through map/reduce. > > -- Owen