Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
Date: Sun, 28 Feb 2010 14:14:37 -0800
Message-ID: <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey Oded,

You don't need to format to to add disks to DNs. Just format them and
add the directories to dfs.data.dir in the config file, and restart
the DN. The data in an individual DN won't be automatically balanced
across disks when you restart. Rebalancing is not necessary as the DN
will round robin blocks over all disks and stop writing to a disk when
it fills. If you want the disks to be balanced you can do that
manually by copying the block files from the existing data directories
to the new ones--HDFS just checks for the blocks at startup, it
doesn't keep track of which directory they were stored in.

Thanks,
Eli

On Sun, Feb 28, 2010 at 1:45 PM, Oded Rosen <oded@legolas-media.com> wrote:
> We have an existing HDFS cluster with several datanodes, and we want to add
> each of the datanodes another hard-disk, as an addition to the existing
> ones.
> Is there a way of doing this without formatting the cluster? Our aim is to
> save all the data where it is, add and configure the new disks, perform a
> balance - with no format whatsoever.
> Is it possible? if so, how?
>
> Any kind of help will be welcomed.
> Thanks,
>
> --
> Oded
>