Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of karl@conviva.com designates
 216.82.254.35 as permitted sender)
Subject: Re: problem with completion notification from block movement
From: Karl Kleinpaste <karl@conviva.com>
To: core-user@hadoop.apache.org
In-Reply-To: <314098690902022006j35f6e763nade324cb36429deb@mail.gmail.com>
References: <1233334799.5164.80.camel@awol.kleinpaste.org>
	 <314098690902011758u2ac19fbev8fd851757eb6bcff@mail.gmail.com>
	 <1233599004.16154.130.camel@awol.kleinpaste.org>
	 <314098690902022006j35f6e763nade324cb36429deb@mail.gmail.com>
Content-Type: multipart/mixed; boundary="=-rfZwOzvAf80T36AdJ4j4"
Date: Tue, 03 Feb 2009 08:15:41 -0500
Message-Id: <1233666941.16154.173.camel@awol.kleinpaste.org>
Mime-Version: 1.0

--=-rfZwOzvAf80T36AdJ4j4
Content-Type: text/plain
Content-Transfer-Encoding: 7bit

On Mon, 2009-02-02 at 20:06 -0800, jason hadoop wrote:
> This can be made significantly worse by your underlying host file
> system and the disks that support it.

Oh, yes, we know...  It was a late-realized mistake just yesterday that
we weren't using noatime on that cluster's slaves.

The attached graph is instructive.  We have our nightly-rotated logs for
DataNode all the way back to when this test cluster was created in
November.  This morning on one node, I sampled the first 10 BlockReport
scan lines from each day's log, up through the current hour today, and
handed it to gnuplot to graph.  The seriously erratic behavior that
begins around the 900K-1M point is very disturbing.

Immediate solutions for us include noatime, nodiratime, BIOS upgrade on
the discs, and eliminating enough small files (blocks) in DFS to get the
total count below 400K.

--=-rfZwOzvAf80T36AdJ4j4--