Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Brad Sarsfield <brad@bing.com>
To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Subject: RE: how to write hadoop client to test HDFS with fault injection
Thread-Topic: how to write hadoop client to test HDFS with fault injection
Thread-Index: AQHMrloe/r1FRI/B+UG/FA1pqOGSgZXEBCHA
Date: Tue, 29 Nov 2011 16:36:57 +0000
Message-ID: 
 <1153F8278FF25C49B2024B5CAAD1CAA3692AE0AD@TK5EX14MBXC135.redmond.corp.microsoft.com>
References: 
 <CAEWyy0-Vv7XnJEcLMZhf5Aa90VMx36SPbLHd0y4rxN5h=5+cMA@mail.gmail.com>
In-Reply-To: 
 <CAEWyy0-Vv7XnJEcLMZhf5Aa90VMx36SPbLHd0y4rxN5h=5+cMA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Hao,

You may want to setup a full Hadoop 'one box' install and start running the=
 TestDFSIO, Teragen and Terasort (with a small amount of data) scenarios us=
ing the injected jar to run hdfs.  That may be the fastest and best way to =
get up and running.

When it comes to fault injection the valuable issues that you find may be f=
ound using the end to end tests versus targeted api calls.  It's much more =
realistic. It's a much stronger case to get something fixed / fix something=
 when you can point to the end to end scenario, showing how it can fail.   =
That's where you find the good ones.

You may also want to consider what a data validation oracle would look like=
 that would allow you to do data validation to catch data corruption issues=
. If you do go further and create a scenario that, say generates a non-rand=
om / repeatable sequence that reads/writes to HDFS, you could use the fault=
 injection and the oracle would validate data correctness.=20

You'd get extra bonus points if you packaged things up and contributed your=
 work to Bigtop so others could benefit long term.

~Brad

-----Original Message-----
From: Hao Yang [mailto:hao.yang0614@gmail.com]=20
Sent: Monday, November 28, 2011 9:44 PM
To: common-dev@hadoop.apache.org
Subject: how to write hadoop client to test HDFS with fault injection

Hi, all:

I am a graduate student, now working on HDFS fault injection course project=
. I used FI framework to create aspects, and hdfs-fi.jar was generated. How=
 can I write Hadoop Client code to test the injection? Thank you.


Thank you very much for your time.


Best regards
Hao Yang