Return-Path: Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org Received: (qmail 61956 invoked from network); 22 Apr 2010 19:59:48 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 22 Apr 2010 19:59:48 -0000 Received: (qmail 58455 invoked by uid 500); 22 Apr 2010 19:59:47 -0000 Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org Received: (qmail 58420 invoked by uid 500); 22 Apr 2010 19:59:47 -0000 Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@hadoop.apache.org Delivered-To: mailing list general@hadoop.apache.org Received: (qmail 58411 invoked by uid 99); 22 Apr 2010 19:59:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 19:59:45 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=10.0 tests=AWL,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender) Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 19:59:40 +0000 DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns; h=X-IronPort-AV:Received:Received:Received:From:To:Subject: Thread-Topic:Thread-Index:Date:Message-ID:References: In-Reply-To:Accept-Language:Content-Language: X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type: Content-ID:Content-Transfer-Encoding:MIME-Version: Return-Path:X-OriginalArrivalTime; b=L0NGGDS0BmVQhMichuZuzpz0T5MQCmKSavEhp9D/vx7JPQGpwGNtbGvh DDXN+eFIgWiNVOW/QnLLuJVnjZjK+BDW9JTPCDFqbCEi7AXmc8FH2l/t+ eIWzm7lp98upEkk; DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt; s=proddkim; t=1271966380; x=1303502380; h=from:sender:reply-to:subject:date:message-id:to:cc: mime-version:content-transfer-encoding:content-id: content-description:resent-date:resent-from:resent-sender: resent-to:resent-cc:resent-message-id:in-reply-to: references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:list-owner:list-archive; z=From:=20Allen=20Wittenauer=20 |Subject:=20Re:=20dfs.data.dir|Date:=20Thu,=2022=20Apr=20 2010=2019:59:18=20+0000|Message-ID:=20<51FEB071-B163-4B01 -A08A-EB4DD9645923@linkedin.com>|To:=20""=20|MIME-Version: =201.0|Content-Transfer-Encoding:=20quoted-printable |Content-ID:=20<25c6fce2-8d3b-4ac2-8c2d-a17dca0af983> |In-Reply-To:=20<4BD04400.90800@apache.org>|References: =20=0D=0A=20=0D=0A=20<4BD04400.90800@apache .org>; bh=QKuYumno2DbWzE7An23Rd54R9ez8NI4Ylnnw27EwYW4=; b=WzpAmdES+gmoUY34Hi/V58F8OsrurDIinwgWgFJ6W4klN2K0obrnzPJW TnzY+33pNa22NC58KmEEMfJrO6740xP7ZdpPSFrJoI+WxdlHLjbFnH+mr 8V4gb28DsRUKPql; X-IronPort-AV: E=Sophos;i="4.52,258,1270450800"; d="scan'208";a="12124020" Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959); Thu, 22 Apr 2010 12:59:20 -0700 Received: from ESV4-CAS01.linkedin.biz (172.18.46.140) by esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id 14.0.682.1; Thu, 22 Apr 2010 12:59:20 -0700 Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Thu, 22 Apr 2010 12:59:19 -0700 From: Allen Wittenauer To: "" Subject: Re: dfs.data.dir Thread-Topic: dfs.data.dir Thread-Index: AQHK4UMhXQAV3PvzLEKrqdMha3hZoZIt54MAgAD9mQCAAHpLAA== Date: Thu, 22 Apr 2010 19:59:18 +0000 Message-ID: <51FEB071-B163-4B01-A08A-EB4DD9645923@linkedin.com> References: <4BD04400.90800@apache.org> In-Reply-To: <4BD04400.90800@apache.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: text/plain; charset="us-ascii" Content-ID: <25c6fce2-8d3b-4ac2-8c2d-a17dca0af983> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 22 Apr 2010 19:59:20.0905 (UTC) FILETIME=[4C52F390:01CAE256] On Apr 22, 2010, at 5:41 AM, Steve Loughran wrote: > that brings up a couple of issues I've been thinking about now that worke= rs can go to 6+ HDDs/node >=20 > * a way to measure the distribution across disks, rather than just nodes.= DfsClient doesn't provide enough info here yet. What should probably happen is that instead of throwing you to the file bro= wser, clicking on a host from the live nodes page should probably put you o= n a "stats about this node" page. > * a way to triger some rebalancing on a single node, to say "position stu= ff more fairly". You don't need to worry about network traffic, just local = disk load and CPU time, so it should be simpler. Yup. Working with 8 drives per node, it is interesting to see how unbalanc= ed the data gets after a while. [Luckily, we have MR tmp space segregated = off so I'm sure it would be a lot worse if we didn't!] Someone should file a jira. :)