hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6332) Large-scale Automated Test Framework
Date Tue, 01 Dec 2009 08:53:20 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Sharad Agarwal updated HADOOP-6332:

    Attachment: 6332_v1.patch

As mentioned above, the abstraction of cluster setup and teardown is well within the scope
of this JIRA. The attached patch tries to address this. Also it provides placeholders for
exposing additional APIs from daemon processes, and a client interface to talk to daemons.
About this patch:
* System test framework classes are in org.apache.hadoop.test.system
   ** DaemonClient provide the interface to manage a particular remote daemon process.
   ** DaemonProtocol is a RPC interface for a daemon. Note this needs to be woven in the server
side code via aspectj.
   ** MasterSlaveCluster interface provides access to master and slaves client handles.
* Abstraction for remote process management is org.apache.hadoop.test.system.process.ClusterProcessManager.
The default implementation being ShellProcessManager which will use hadoop bin scripts to
start/stop the daemon. Apart from process management, if later we want to push the tar balls
on cluster nodes etc, then this interface can be exploited.
* The implementation for mapreduce is in org.apache.hadoop.mapreduce. (Needs to be done in
MAPREDUCE-1154. Putting here for easy reference.)
 ** JTProtocol interface implementation needs to be woven in Jobtracker code. Similary TTProtocol
in TaskTracker code.
 ** JTClient and TTClient are the client classes. Note that JTClient composes org.apache.hadoop.mapreduce.Cluster
class. For maintainability, the intention is to do minimum weaving and if possible avoid it
on client side. The verification utilites which are generic and can be used for all system
test cases can be in JTClient/TTClient/MRCluster classes.
 ** Tests will create MRCluster via using MRClusterFactory class. A sample test class is TestCluster.
Perhaps we can have a test suite where cluster is setup and teardown once. The tests in a
particular suite are expected to be side effects free.

Thoughts ?

> Large-scale Automated Test Framework
> ------------------------------------
>                 Key: HADOOP-6332
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6332
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.21.0
>         Attachments: 6332_v1.patch
> Hadoop would benefit from having a large-scale, automated, test-framework. This jira
is meant to be a master-jira to track relevant work.
> ----
> The proposal is a junit-based, large-scale test framework which would run against _real_
> There are several pieces we need to achieve this goal:
> # A set of utilities we can use in junit-based tests to work with real, large-scale hadoop
clusters. E.g. utilities to bring up to deploy, start & stop clusters, bring down tasktrackers,
datanodes, entire racks of both etc.
> # Enhanced control-ability and inspect-ability of the various components in the system
e.g. daemons such as namenode, jobtracker should expose their data-structures for query/manipulation
etc. Tests would be much more relevant if we could for e.g. query for specific states of the
jobtracker, scheduler etc. Clearly these apis should _not_ be part of the production clusters
- hence the proposal is to use aspectj to weave these new apis to debug-deployments.
> ----
> Related note: we should break up our tests into at least 3 categories:
> # src/test/unit -> Real unit tests using mock objects (e.g. HDFS-669 & MAPREDUCE-1050).
> # src/test/integration -> Current junit tests with Mini* clusters etc.
> # src/test/system -> HADOOP-6332 and it's children

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message