Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of adi@cloudera.com designates
 209.85.223.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAu13zGCLpJ-LuSBJSScK_ApKCzJqqB8FW1vwKimSbLLPWGrRw@mail.gmail.com>
References: 
 <CAAu13zFkgKZ3TR0ONeeP28_zvyORkkzQBYk-rWYizwrZ9yRq6g@mail.gmail.com>
	<CAO6W-2e5nNA2SMBY5g7SDocd=R98PBN=eHC_tMqZZXN7GMu_9Q@mail.gmail.com>
	<CAAu13zGCLpJ-LuSBJSScK_ApKCzJqqB8FW1vwKimSbLLPWGrRw@mail.gmail.com>
Date: Tue, 13 Nov 2012 13:53:33 -0800
Message-ID: 
 <CAE_FxDj6F=rTUrsWt6PYN019=5AOr=-53uFodw4ERjuy=Y2h-A@mail.gmail.com>
Subject: Re: Optimizing Disk I/O - does HDFS do anything ?
From: Andy Isaacson <adi@cloudera.com>
To: user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Nov 13, 2012 at 1:40 PM, Jay Vyas <jayunit100@gmail.com> wrote:
> 1) but I thought that this sort of thing (yes even on linux) becomes
> important when you have large amounts of data - because the way files are
> written can cause issues on highly packed drives.

If you're running any filesystem at 99% full with a workload that
creates or grows files, the filesystem will experience fragmentation.
Don't do that if you want good performance.

As long as there's a few dozen GB of free space to work with, ext4 on
a modern Linux kernel (2.6.38 or newer) will do a fine job of keeping
files sequential and shouldn't need defrag.

To answer the original question -- HDFS doesn't take any special
measures to enforce defragmentation, but HDFS does follow best
practices to avoid causing fragmentation.

-andy