hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Boudnik (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6332) Large-scale Automated Test Framework
Date Wed, 24 Feb 2010 19:33:28 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837968#action_12837968

Konstantin Boudnik commented on HADOOP-6332:

@Stephen: the main reason to use code injection is to completely hide testing handles from
any chance of misusing by a stranger. Apparently many of the contracts (interfaces, APIs)
we are interested in a course of testing either unveil internal states of key Hadoop components
or allow to perform 'undesirable' actions such as killing a job, a tasktracker, or a datanode
it'd be unwise to keep them in the A-grade production code. Therefore, code injection seems
to be the right technique for this. 

Next version of the patch is coming any minute now. It will be clear that all interfaces exposed
to test are defined statically. Their implementation is injected though, which shouldn't concern
anyone but framework developers.

Now, a particular implementation of injection doesn't really matter. We could've go with ASM
or BCEL for the purpose. It happens that we have readily available AspectJ providing high-level
language capabilities, Eclipse integration, etc. That explain the choice of the framework.

As for an extra burden for future contributors: instrumentation is used for internal framework
mechanics and shouldn't be exposed to the test developers. Thus, if one simply want to develop
a cluster test she/he can do it from a vanilla Eclipse without AJDT installed. Or from IDEA
(which I personally prefer and use all the time, except when I need to develop/fix some aspects).
Or from vim (not like I suggest to do it :-)

> Large-scale Automated Test Framework
> ------------------------------------
>                 Key: HADOOP-6332
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6332
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.21.0
>         Attachments: 6332.patch, 6332_v1.patch, 6332_v2.patch, HADOOP-6332-MR.patch,
HADOOP-6332-MR.patch, HADOOP-6332.patch, HADOOP-6332.patch
> Hadoop would benefit from having a large-scale, automated, test-framework. This jira
is meant to be a master-jira to track relevant work.
> ----
> The proposal is a junit-based, large-scale test framework which would run against _real_
> There are several pieces we need to achieve this goal:
> # A set of utilities we can use in junit-based tests to work with real, large-scale hadoop
clusters. E.g. utilities to bring up to deploy, start & stop clusters, bring down tasktrackers,
datanodes, entire racks of both etc.
> # Enhanced control-ability and inspect-ability of the various components in the system
e.g. daemons such as namenode, jobtracker should expose their data-structures for query/manipulation
etc. Tests would be much more relevant if we could for e.g. query for specific states of the
jobtracker, scheduler etc. Clearly these apis should _not_ be part of the production clusters
- hence the proposal is to use aspectj to weave these new apis to debug-deployments.
> ----
> Related note: we should break up our tests into at least 3 categories:
> # src/test/unit -> Real unit tests using mock objects (e.g. HDFS-669 & MAPREDUCE-1050).
> # src/test/integration -> Current junit tests with Mini* clusters etc.
> # src/test/system -> HADOOP-6332 and it's children

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message