Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8B33E575F for ; Tue, 10 May 2011 17:00:10 +0000 (UTC) Received: (qmail 37218 invoked by uid 500); 10 May 2011 17:00:09 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 37118 invoked by uid 500); 10 May 2011 17:00:09 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 37110 invoked by uid 99); 10 May 2011 17:00:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 17:00:09 +0000 X-ASF-Spam-Status: No, hits=4.7 required=5.0 tests=FS_REPLICA,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 17:00:00 +0000 Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5]) by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4AGxPfw011932 for ; Tue, 10 May 2011 09:59:25 -0700 (PDT) Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.32]) by SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Tue, 10 May 2011 09:59:24 -0700 From: Matthew Foley To: "hdfs-user@hadoop.apache.org" CC: Matthew Foley Date: Tue, 10 May 2011 09:59:24 -0700 Subject: Re: more replicas on a single node Thread-Topic: more replicas on a single node Thread-Index: AcwPM52SSFNTCf8pQYmUHGOytlncLA== Message-ID: <53E12B43-4653-4280-936D-0C03B9FA2507@yahoo-inc.com> References: <4DC7CEF9.3020902@kalooga.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Hi Ferdy, I'm not aware of anyone running this way in production, but for test purpos= es it is often useful to run two DataNodes on a single physical server. It= works fine, you just need to give the two services different HADOOP_CONF_D= IR values with modified port numbers and storage directories. I've previou= sly posted recipes for doing those configurations, but the spam filter is b= ouncing messages from me containing the link, so just go to the Apache list= archive at http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/ and = browse to my email of Thu, 16 Sep 2010 00:45:23 GMT, for a discussion of th= e details. If you give the two DN control of separate subsets of disks, it will suppor= t your scenario below. --Matt On May 9, 2011, at 4:24 AM, Ferdy Galema wrote: Is it possible to enforce a replication of 2 for a single node, so that=20 replicas are spread out over disks? Currently with more replicas than=20 nodes this results in "under-replicated" blocks. I understand that=20 normally the best way to replicate is to span multiple machines for=20 availabilty purposes. However is there a way around this?