Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of mathias.herberts@gmail.com
 designates 209.85.216.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=DOsfEcuv/6PN8x+eChIhSWxJ/mJor7NqXYDgfhrRWreOqIPIUecYcPaFN4YELVmFc5
         kghxpT/zUmllZe9j4onbI3jY+jJD37exRNIauVkdGFpALVTgJjqoUPm9PgVRtV6HO1fR
         +JxsX4wL7W8aJZCtbPUzyZAboZsJBrMT0Uy/4=
MIME-Version: 1.0
In-Reply-To: <4DC2987C.1010609@kalooga.com>
References: <4DC2987C.1010609@kalooga.com>
Date: Thu, 5 May 2011 14:57:26 +0200
Message-ID: <BANLkTin5781DC_Cscdzxp90X+QqGj-4oqg@mail.gmail.com>
Subject: Re: distcp performing much better for rebalancing than dedicated
 balancer
From: Mathias Herberts <mathias.herberts@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Did you explicitely start a balancer or did you decommission the nodes
using dfs.hosts.exclude and a dfsadmin -refreshNodes?

On Thu, May 5, 2011 at 14:30, Ferdy Galema <ferdy.galema@kalooga.com> wrote=
:
> Hi,
>
> On our 15node cluster (1GB ethernet and 4x1TB disk per node) I noticed th=
at
> distcp does a much better job at rebalancing than the dedicated balancer
> does. We needed to decommision 11 nodes, so that prior to rebalancing we =
had
> 4 used and 11 empty nodes. The 4 used nodes had about 25% usage each. Mos=
t
> of our files are of average size: We have about 500K files in 280K blocks
> and 800K blocks total (blocksize is 64MB).
>
> So I changed dfs.balance.bandwidthPerSec to 800100100 and restarted the
> cluster. Started the balancer tool and I noticed that the it moved about
> 200GB in 1 hour. (I grepped the balancer log for "Need to move").
>
> After stopping the balancer I started a distcp. =C2=A0This tool copied 90=
0GB in
> just 45 minutes, with an average replication of 2 so it's total throughpu=
t
> was around 2.4 TB/hour. Fair enough, it is not purely rebalancing because
> the 4 overused nodes also get new blocks, still it performs much better.
> Munin confirms the much higher disk/ethernet throughputs of the distcp.
>
> Are these characteristics to be expected? Either way, can the balancer be
> boosted even more? (Aside the dfs.balance.bandwidthPerSec property).
>
> Ferdy.
>