Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 26B4510C31 for ; Wed, 5 Mar 2014 07:59:36 +0000 (UTC) Received: (qmail 4950 invoked by uid 500); 5 Mar 2014 07:59:29 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 4001 invoked by uid 500); 5 Mar 2014 07:59:27 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 3979 invoked by uid 99); 5 Mar 2014 07:59:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Mar 2014 07:59:25 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of azuryyyu@gmail.com designates 209.85.216.48 as permitted sender) Received: from [209.85.216.48] (HELO mail-qa0-f48.google.com) (209.85.216.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Mar 2014 07:59:19 +0000 Received: by mail-qa0-f48.google.com with SMTP id m5so648442qaj.35 for ; Tue, 04 Mar 2014 23:58:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=QzxCIh+d+0Nvgv7z9W2kS1RHGLsJUKGZmiHyzxFcCYc=; b=X+alX7vQBYFI7d8hk74Su8/U13b6Um/pXHlFFTD87X12O8vN/9BlUKh5oQhzf5RwSm B+8KfJL3xRLrtf/YUAd+5kkGg5PzBdXMl7VhXfs+hE7CF9x1xGy2LVITtZ+y305iHU4R IgcIYzl2gpmuPpbYl7CAjcBc8zo+Eq8NOmfrYzAz2RBc1c4bPBrmn4+mFZc8QthlgIO/ oc4yrFIDCVkmCf2Nm1YQJ8dxz9eXOoL5erOMc80gwooHsZMAbl9myIz5yNESRZboqodH frG4L5V85HX5LmadeyIQt7w+wF8xSkPxDYbVVEwq6N2LlRscXTOSKw6Paq6vS/YRvTrx nLzA== MIME-Version: 1.0 X-Received: by 10.140.81.244 with SMTP id f107mr418586qgd.104.1394006338439; Tue, 04 Mar 2014 23:58:58 -0800 (PST) Received: by 10.140.27.161 with HTTP; Tue, 4 Mar 2014 23:58:58 -0800 (PST) In-Reply-To: References: Date: Wed, 5 Mar 2014 15:58:58 +0800 Message-ID: Subject: Re: Question on DFS Balancing From: Azuryy Yu To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=001a11c136ca9e107004f3d7649f X-Virus-Checked: Checked by ClamAV on apache.org --001a11c136ca9e107004f3d7649f Content-Type: text/plain; charset=ISO-8859-1 Hi, That probably break something if you apply the patch from 2.x to 0.20.x, but it depends on. AFAIK, Balancer had a major refactor in HDFSv2, so you'd better fix it by yourself based on HDFS-1804. On Wed, Mar 5, 2014 at 3:47 PM, divye sheth wrote: > Thanks Harsh. The jira is fixed in version 2.1.0 whereas I am using Hadoop > 0.20.2 (we are in a process of upgrading) is there a workaround for the > short term to balance the disk utilization? The patch in the Jira, if > applied to the version that I am using, will it break anything? > > Thanks > Divye Sheth > > > On Wed, Mar 5, 2014 at 11:28 AM, Harsh J wrote: > >> You're probably looking for >> https://issues.apache.org/jira/browse/HDFS-1804 >> >> On Tue, Mar 4, 2014 at 5:54 AM, divye sheth wrote: >> > Hi, >> > >> > I am new to the mailing list. >> > >> > I am using Hadoop 0.20.2 with an append r1056497 version. The question I >> > have is related to balancing. I have a 5 datanode cluster and each node >> has >> > 2 disks attached to it. The second disk was added when the first disk >> was >> > reaching its capacity. >> > >> > Now the scenario that I am facing is, when the new disk was added hadoop >> > automatically moved over some data to the new disk. But over the time I >> > notice that data is no longer being written to the second disk. I have >> also >> > faced an issue on the datanode where the first disk had 100% >> utilization. >> > >> > How can I overcome such scenario, is it not hadoop's job to balance the >> disk >> > utilization between multiple disks on single datanode? >> > >> > Thanks >> > Divye Sheth >> >> >> >> -- >> Harsh J >> > > --001a11c136ca9e107004f3d7649f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,=A0
That probably break something if you apply the = patch from 2.x to 0.20.x, but it depends on.

AFAIK= , Balancer had a major refactor in HDFSv2, so you'd better fix it by yo= urself based on HDFS-1804.



On Wed, Mar 5, 2014 at 3:47 PM, divye sheth <<= a href=3D"mailto:divs.sheth@gmail.com" target=3D"_blank">divs.sheth@gmail.c= om> wrote:
Thanks Harsh. The jira is f= ixed in version 2.1.0 whereas I am using Hadoop 0.20.2 (we are in a process= of upgrading) is there a workaround for the short term to balance the disk= utilization? The patch in the Jira, if applied to the version that I am us= ing, will it break anything?

Thanks
<= div>Divye Sheth


On We= d, Mar 5, 2014 at 11:28 AM, Harsh J <harsh@cloudera.com> wr= ote:
You're probably looking for https://i= ssues.apache.org/jira/browse/HDFS-1804

On Tue, Mar 4, 2014 at 5:54 AM, divye sheth <divs.sheth@gmail.com> wrote:
> Hi,
>
> I am new to the mailing list.
>
> I am using Hadoop 0.20.2 with an append r1056497 version. The question= I
> have is related to balancing. I have a 5 datanode cluster and each nod= e has
> 2 disks attached to it. The second disk was added when the first disk = was
> reaching its capacity.
>
> Now the scenario that I am facing is, when the new disk was added hado= op
> automatically moved over some data to the new disk. But over the time = I
> notice that data is no longer being written to the second disk. I have= also
> faced an issue on the datanode where the first disk had 100% utilizati= on.
>
> How can I overcome such scenario, is it not hadoop's job to balanc= e the disk
> utilization between multiple disks on single datanode?
>
> Thanks
> Divye Sheth



--
Harsh J


--001a11c136ca9e107004f3d7649f--