hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: how to write hadoop client to test HDFS with fault injection
Date Tue, 29 Nov 2011 16:59:06 GMT
I'd add that one thing I'd like is better low level hardware fault 
simulation, something that doesn't inject faults into the software, so 
much as stamp on the underlying data or kill bits of the stack entirely

That is
  -deliberately corrupting blocks or checksums
  -deleting blocks
  -dropping entire racks in the real/simulated cluster


Then follow Brad's suggestion and see how terasort copes (you can scale 
it down to GB for local tests first), before having a look at how the 
layers above -especially hbase- cope with problems.

On 29/11/11 16:36, Brad Sarsfield wrote:
> Hi Hao,
> You may want to setup a full Hadoop 'one box' install and start running the TestDFSIO,
Teragen and Terasort (with a small amount of data) scenarios using the injected jar to run
hdfs.  That may be the fastest and best way to get up and running.
> When it comes to fault injection the valuable issues that you find may be found using
the end to end tests versus targeted api calls.  It's much more realistic. It's a much stronger
case to get something fixed / fix something when you can point to the end to end scenario,
showing how it can fail.   That's where you find the good ones.
> You may also want to consider what a data validation oracle would look like that would
allow you to do data validation to catch data corruption issues. If you do go further and
create a scenario that, say generates a non-random / repeatable sequence that reads/writes
to HDFS, you could use the fault injection and the oracle would validate data correctness.
> You'd get extra bonus points if you packaged things up and contributed your work to Bigtop
so others could benefit long term.
> ~Brad

> -----Original Message-----
> From: Hao Yang [mailto:hao.yang0614@gmail.com]
> Sent: Monday, November 28, 2011 9:44 PM
> To: common-dev@hadoop.apache.org
> Subject: how to write hadoop client to test HDFS with fault injection
> Hi, all:
> I am a graduate student, now working on HDFS fault injection course project. I used FI
framework to create aspects, and hdfs-fi.jar was generated. How can I write Hadoop Client
code to test the injection? Thank you.
> Thank you very much for your time.
> Best regards
> Hao Yang

View raw message