Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 84742 invoked from network); 19 May 2009 14:24:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 19 May 2009 14:24:04 -0000 Received: (qmail 79021 invoked by uid 500); 19 May 2009 14:24:02 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 78936 invoked by uid 500); 19 May 2009 14:24:02 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 78926 invoked by uid 99); 19 May 2009 14:24:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 May 2009 14:24:02 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 May 2009 14:23:52 +0000 Received: from localhost (localhost [127.0.0.1]) by tobor.hpl.hp.com (Postfix) with ESMTP id 7647CB7CAD for ; Tue, 19 May 2009 15:23:30 +0100 (BST) X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com Received: from tobor.hpl.hp.com ([127.0.0.1]) by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 9iNuRSl+ByPo for ; Tue, 19 May 2009 15:23:23 +0100 (BST) Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tobor.hpl.hp.com (Postfix) with ESMTPS id A13D4B7C93 for ; Tue, 19 May 2009 15:23:23 +0100 (BST) MailScanner-NULL-Check: 1243347793.14328@1JmHVdqi2clHOoMzYxEshA Received: from [16.25.171.118] (morzine.hpl.hp.com [16.25.171.118]) by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id n4JENC4K004027 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 19 May 2009 15:23:12 +0100 (BST) Message-ID: <4A12C0D0.20807@apache.org> Date: Tue, 19 May 2009 15:23:12 +0100 From: Steve Loughran User-Agent: Thunderbird 2.0.0.21 (X11/20090302) MIME-Version: 1.0 To: core-user@hadoop.apache.org Subject: Re: Suspend or scale back hadoop instance References: <4238036a0905190636q1c395a49n4dfdec4e1bd44d9d@mail.gmail.com> In-Reply-To: <4238036a0905190636q1c395a49n4dfdec4e1bd44d9d@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-HPL-MailScanner-Information: Please contact the ISP for more information X-MailScanner-ID: n4JENC4K004027 X-HPL-MailScanner: Found to be clean X-HPL-MailScanner-From: stevel@apache.org X-Virus-Checked: Checked by ClamAV on apache.org John Clarke wrote: > Hi, > > I am working on a project that is suited to Hadoop and so want to create a > small cluster (only 5 machines!) on our servers. The servers are however > used during the day and (mostly) idle at night. > > So, I want Hadoop to run at full throttle at night and either scale back or > suspend itself during certain times. You could add/remove new task trackers on idle systems, but * you don't want to take away datanodes, as there's a risk that data will become unavailable. * there's nothing in the scheduler to warn that machines will go away at a certain time If you only want to run the cluster at night, I'd just configure the entire cluster to go up and down