hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hao Yang <hao.yang0...@gmail.com>
Subject Re: how to write hadoop client to test HDFS with fault injection
Date Wed, 30 Nov 2011 03:23:19 GMT
Thanks very much for the above. In 0.21 I found TestDFSIO in the
mapred-test.jar, but I didn't find Teragen and Terasort in the three test
jars (I found Teragen and Terasort in 0.20.04 test jars, but they cannot be
executed in 0.21). When I run TestDFSIO benchmark, I sometimes encounter
the situation below. I met the same problem when I test it in my laptop
with one datanode or two datanodes launched.

 java.io.IOException: File
/benchmarks/TestDFSIO/io_control/in_file_test_io_0 could only be replicated
to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1448)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:691)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344)


Thanks.

Hao Yang

2011/11/29 Konstantin Boudnik <cos@apache.org>

> Once you have an instrumented cluster you can run some end-to-end tests
> (TeraSort
> has been suggested and it would do fine). The tricky part is to assemble
> all
> moving parts of the Hadoop cluster together from three different projects
> (if
> you are on .21+ branch) and will be a way easier for 0.20.+ based versions.
>
> However, one fine point here is that your faults (implemented as aspects)
> should be triggered some how. You can do it logically (say every 3rd
> attempt
> to get new block) or via environment, or from the test application.
>
> The latter might be a most tricky part because you'll have to essentially
> communicate between different JVM processes perhaps located on different
> physical hosts if you're running in a real cluster environment.
>
> To address these issues we have project Herriot which is based on byte code
> injection (read aspectj) and allows for distributed fault injections in a
> real
> cluster. System test framework adds a few APIs to the Hadoop daemons and
> allow
> to call them via standard Hadoop RPC.
>
> You can find framework sources and tests code in src/aop/system.
> There's a separate set of build targets which will help you instrument
> everything you need (jar-test-system should do). More info is here
> http://is.gd/SBkjrC
>
> One last point: Herriot isn't fully integrated in trunk's maven build as
> of today.
>
> Hope it helps,
>  Cos
>
> >> From: Hao Yang [mailto:hao.yang0614@gmail.com]
> >> Sent: Monday, November 28, 2011 9:44 PM
> >> To: common-dev@hadoop.apache.org
> >> Subject: how to write hadoop client to test HDFS with fault injection
> >>
> >> Hi, all:
> >>
> >> I am a graduate student, now working on HDFS fault injection course
> >> project. I used FI framework to create aspects, and hdfs-fi.jar was
> >> generated. How can I write Hadoop Client code to test the injection?
> Thank
> >> you.
> >>
> >>
> >> Thank you very much for your time.
> >>
> >>
> >> Best regards
> >> Hao Yang
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message