hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@yahoo-inc.com>
Subject Re: How to run Fault injection in HDFS
Date Fri, 20 Nov 2009 21:21:04 GMT
Generally the idea was to provide everything needed for injection by what 
current build.xml is having in Common and Hdfs. Would you mind to share what 
extra changes you've needed and why?

Cos

On 11/20/09 12:32 , Thanh Do wrote:
> Thank you folks!
>
> Finally, I am able (really) to run FI with HADOOP. I added some aspects
> into the source code, changed the build.xml, and that's it.
>
> AspectJ is awesome!
>
> Have a nice weekend!
>
> On Fri, Nov 20, 2009 at 1:08 PM, Konstantin Boudnik <cos@yahoo-inc.com
> <mailto:cos@yahoo-inc.com>> wrote:
>
>     Hi Thanh.
>
>     hmm, it sounds like you have some issue with compilation of your code.
>
>     addDeprication() has been added to Configuration in 0.21, I believe.
>     And it is there no matter how do you compile your code (with FI or
>     without).
>
>     Cos
>
>
>     On 11/19/09 10:12 , Thanh Do wrote:
>
>         Sorry to dig this thread again!
>
>         I am expecting the release of 0.21 so that I don't have to
>         manually play
>         around with AspectJ FI any more.
>
>         I still have problem with running HDFS with instrumented code
>         (with aspect).
>
>         Here is what I did:
>
>         In the root directory of HDFS:
>         /$ ant injectfaults
>
>         $ ant jar-fault-inject
>         /At this point, i have a jar file containing hdfs classed, namely,
>         /hadoop-hdfs-0.22.0-dev-fi.jar/, located in /build-fi/ folder.
>
>         Now I go to the HADOOP folder (which contains running script in bin
>         directory), and do the following
>         /$ ant compile-core-classes/
>         ( now I need additional hdfs classes to be able to run
>         /start-dfs.sh/ <http://start-dfs.sh/>,
>         right)
>         What I did is copying
>         /$HDFS/build-fi/hadoop-hdfs-0.22.0-dev-fi.jar /to
>         /$HADOOP/hadoop-hdfs-fi-core.jar/ (I need to add suffix "core"
>         since the
>         script will include all hadoop-*-core.jar in classpath)
>
>         /$ bin/start-dfs.sh/ <http://start-dfs.sh/>
>         and got error message:
>
>         2009-11-19 11:52:57,479 ERROR
>         org.apache.hadoop.hdfs.server.namenode.NameNode:
>         java.lang.NoSuchMethodError:
>         org.apache.hadoop.conf.Configuration.addDeprecation(Ljava/lang/String;[Ljava/lang/String;)V
>                  at
>         org.apache.hadoop.hdfs.HdfsConfiguration.deprecate(HdfsConfiguration.java:44)
>                  at
>         org.apache.hadoop.hdfs.HdfsConfiguration.addDeprecatedKeys(HdfsConfiguration.java:48)
>                  at
>         org.apache.hadoop.hdfs.HdfsConfiguration.<clinit>(HdfsConfiguration.java:28)
>                  at
>         org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>                  at
>         org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1199)
>
>         2009-11-19 11:52:57,480 INFO
>         org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
>         Could any one tell me how to solve this problem?
>
>         Thank you so much.
>
>
>         On Thu, Oct 8, 2009 at 10:41 AM, Konstantin Boudnik
>         <cos@yahoo-inc.com <mailto:cos@yahoo-inc.com>
>         <mailto:cos@yahoo-inc.com <mailto:cos@yahoo-inc.com>>> wrote:
>
>             Thanks for looking into fault injection - it's very
>         interesting and
>             useful technique based on AspectJ.
>
>             Currently, it is fully integrated into HDFS only. There's a JIRA
>             (HADOOP-6204) which tracks the same effort for Common and
>         then all
>             Hadoop's components will have injection (as well as fault
>         injection)
>             in place. This JIRA should be committed in the matter of a
>         couple of
>             weeks.
>
>             For the immediate purpose you don't need to patch anything
>         or do any
>             tweaking of the code: the fault injection framework is in
>         already
>             and ready to work.
>
>             For your current needs: to be able to run HDFS with instrumented
>             code you need to run a special build. To do so:
>               - % ant injectfaults - similar to a 'normal' build, but does
>             instrument the code with aspects located under src/test/aop/**
>               - % ant jar-fault-inject - similar to a 'normal' jar
>         creation but
>             instrumented
>               - % ant jar-test-fault-inject - similar to a 'normal' jar-test
>             creation but instrumented
>
>             Now, if you have the rest of sub-projects built you need to
>         move the
>             instrumented jar files on top of the 'normal' files in your
>             installation directory. Please note that some renaming has to be
>             done: injected jar files have '-fi' suffix in their names
>         and normal
>             jar files don't have such. Thus currently you'll have to rename
>             those injected jars to pretend like they are normal, used by
>             configured's classpath.
>
>             At this point you all set: you have a production quality
>         Hadoop with
>             injected HDFS. As soon as the aforementioned JIRA is ready and
>             committed we'd be able to provide Hadoop-injected version by the
>             build's means rather than doing any renaming and manual
>         intervention.
>
>             Also, if you need to read more about fault injection (FI) in
>         HDFS
>             you can find FI-framework documentation in the current HDFS
>         trunk
>             (it isn't on the web yet for version 0.21 hasn't been
>         released yet).
>             Because building documentation requires some extra effort and
>             additional software to be installed, you can simply download and
>             read the PDF from this FI-framework JIRA
>
>         https://issues.apache.org/jira/secure/attachment/12414225/Fault+injection+development+guide+and+Framework+HowTo.pdf
>
>             Hope it helps,
>               Cos
>
>
>             On 10/8/09 8:10 AM, Thanh Do wrote:
>
>                 Thank you so much, Jakob.
>
>                 Could you please explain the fault injection running
>         procedure
>                 in details?
>
>                 My goal is running HDFS in a cluster (with a namenode
>         and several
>                 datanode), and see how fault injection techniques affect
>         HDFS
>                 behavior's. Also, I would like to define some new
>         aspects/fault
>                 to test
>                 the system.
>
>                 What I did was:
>                 1) I checked out the hadoop-common-trunk, but this
>         package doesn't
>                 contain HDFS classes. I finally noticed that FI framework is
>                 currently
>                 integrated with HDFS only.
>
>                 2) So, I checked out the hdfs-trunk. The build.xml contain
>                 injectfaults
>                 target and several other related things. I was able to
>         build those
>                 targets (injectfaults, run-test-hdfs-fault-inject, etc).
>         Up to this
>                 point, I stucked because I found no scripted that help me to
>                 start-dfs,
>                 stop-dfs...
>                 I copied the bin folder from common/core to HDFS project
>         folder
>                 and ran
>                 the  script:
>
>                 /bin/start-dfs.sh/ <http://start-dfs.sh/>
>         <http://start-dfs.sh/>
>
>
>                 but there is exception:
>
>                 /Exception in thread
>                 main"Java.lang.NoClassDefFoundError
>                 : org/apache/commons/logging/LogFactory
>                 /
>                 I guess the reason is I ran HDFS without any common
>         class. How I get
>                 around this?
>
>                 3) I also tried the third way, by download the hadoop
>         release
>                 (contain
>                 everything: core, hdfs, mapred), and used Eclipse to create
>                 project from
>                 existing code. I was able to build this project. The bin
>         scripts
>                 worked
>                 well but I found know FI related classes. What I did was
>         apply
>                 the patch
>                 (HADOOP-6003.patch) using Eclipse patch command (Team |
>         apply
>                 patch),
>                 but I failed the patching procedure.
>
>                 In summary, I would like to run a real HDFS with fault
>                 injection. I am
>                 not very familiar with ant. Could you please show me
>         some more
>                 details,
>                 so that I could get around this?
>
>                 On Thu, Oct 8, 2009 at 12:19 AM, Jakob Homan
>         <jhoman@yahoo-inc.com <mailto:jhoman@yahoo-inc.com>
>         <mailto:jhoman@yahoo-inc.com <mailto:jhoman@yahoo-inc.com>>
>         <mailto:jhoman@yahoo-inc.com <mailto:jhoman@yahoo-inc.com>
>         <mailto:jhoman@yahoo-inc.com <mailto:jhoman@yahoo-inc.com>>>>
wrote:
>
>                     Thanh-
>                     If you would like the run execute the tests that
>         have been
>                     instrumented to use the fault injection framework
>         the ant
>                 target is
>                     run-test-hdfs-fault-inject.  These were used
>         extensively in the
>                     recent append work and there are quite a few
>         append-related
>                 tests.
>                       Was there something more specific you were looking
>         for?
>
>                     Thanks,
>                     Jakob
>                     Hadoop at Yahoo!
>
>
>                     Thanh Do wrote:
>
>                         Hi everyone,
>
>                         Could any body so me how to run the fault injection
>                 framework
>                         mentioned in the following links?:
>
>         http://issues.apache.org/jira/browse/HDFS-435
>
>                         and
>
>         https://issues.apache.org/jira/browse/HDFS-436
>
>                         Thanks,
>                         Thanh
>
>
>
>
>
>
>                 --
>                 T
>
>
>             --
>             With best regards,
>                     Konstantin Boudnik (aka Cos)
>
>                     Yahoo! Grid Computing
>                     +1 (408) 349-4049
>
>             2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>             Attention! Streams of consciousness are disallowed
>
>
>
>
>         --
>         thanh
>
>
>
>
> --
> thanh

Mime
View raw message