Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E8DF59D39 for ; Wed, 12 Nov 2014 16:37:42 +0000 (UTC) Received: (qmail 48163 invoked by uid 500); 12 Nov 2014 16:37:33 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 48070 invoked by uid 500); 12 Nov 2014 16:37:33 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 48060 invoked by uid 99); 12 Nov 2014 16:37:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Nov 2014 16:37:32 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of bhuffman@etinternational.com designates 65.222.140.81 as permitted sender) Received: from [65.222.140.81] (HELO mail02.etinternational.com) (65.222.140.81) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Nov 2014 16:37:27 +0000 X-Footer: ZXRpbnRlcm5hdGlvbmFsLmNvbQ== Received: from polaris.xmen.eti ([192.168.15.21]) (authenticated user bhuffman@etinternational.com) by mail02.etinternational.com (using TLSv1/SSLv3 with cipher AES128-SHA (128 bits)) for user@hadoop.apache.org; Wed, 12 Nov 2014 11:36:03 -0500 Message-ID: <54638C71.2050301@etinternational.com> Date: Wed, 12 Nov 2014 11:36:01 -0500 From: "Brian C. Huffman" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: user@hadoop.apache.org Subject: Datanode disk configuration Content-Type: multipart/alternative; boundary="------------050905060103000008020509" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------050905060103000008020509 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit All, I'm setting up a 4-node Hadoop 2.5.1 cluster. Each node has the following drives: 1 - 500GB drive (OS disk) 1 - 500GB drive 1 - 2 TB drive 1 - 3 TB drive. In past experience I've had lots of issues with non-uniform drive sizes for HDFS, but unfortunately it wasn't an option to get all 3TB or 2TB drives for this cluster. My thought is to set up the 2TB and 3TB drives as HDFS and the 500GB drive as intermediate data. Most our of jobs don't make large use of intermediate data, but at least this way, I get a good amount of space (2TB) per node before I run into issues. Then I may end up using the AvailableSpaceVolumeChoosingPolicy to help with balancing the blocks. If necessary I could put intermediate data on one of the OS partitions (/home). But this doesn't seem ideal. Anybody have any recommendations regarding the optimal use of storage in this scenario? Thanks, Brian --------------050905060103000008020509 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit All,

I'm setting up a 4-node Hadoop 2.5.1 cluster.  Each node has the following drives:
1 - 500GB drive (OS disk)
1 - 500GB drive
1 - 2 TB drive
1 - 3 TB drive.

In past experience I've had lots of issues with non-uniform drive sizes for HDFS, but unfortunately it wasn't an option to get all 3TB or 2TB drives for this cluster. 

My thought is to set up the 2TB and 3TB drives as HDFS and the 500GB drive as intermediate data.  Most our of jobs don't make large use of intermediate data, but at least this way, I get a good amount of space (2TB) per node before I run into issues.  Then I may end up using the AvailableSpaceVolumeChoosingPolicy to help with balancing the blocks.

If necessary I could put intermediate data on one of the OS partitions (/home).  But this doesn't seem ideal.

Anybody have any recommendations regarding the optimal use of storage in this scenario?

Thanks,
Brian
 --------------050905060103000008020509--