incubator-bigtop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sujay Rau (JIRA)" <>
Subject [jira] [Updated] (BIGTOP-635) Implement a cluster-abstraction, discovery and manipulation framework for iTest
Date Fri, 20 Jul 2012 18:04:36 GMT


Sujay Rau updated BIGTOP-635:


Uploaded is a prototype of the BigtopClusterManager that has enough implemented to solve the
problems of BIGTOP-614. 

- Initializing the manager now creates an object for each host that contains its properties
such as rack assignment. All hosts are currently accessed via ssh commands.
- Daemons are also discovered through "ps" and stored so that the user can query the current
state of the daemons on each host object. These daemons are refreshed after a command is run
to keep the state up to date. The discovery code still needs to be updated as it doesn't properly
discover hbase daemons, but it works fine for killing of namenodes and issuing commands from
namenode hosts (which was needed in BIGTOP-614).
- Configuration values can be set and obtained through the bigtop configuration class.

Thanks for looking at the code.
- I have not updated the packages yet but it will definitely be done.
- I changed Shim to Adapter
- Nodes commissioning/decommissioning is not included in the scope currently, but I believe
it would be possible to add later on.

TestHAMRBCM stands for High Availability, Map Reduce, with BigtopClusterManager. I have tested
it on a 4 node cluster and it does what it is supposed to do.

> Implement a cluster-abstraction, discovery and manipulation framework for iTest
> -------------------------------------------------------------------------------
>                 Key: BIGTOP-635
>                 URL:
>             Project: Bigtop
>          Issue Type: New Feature
>          Components: Tests
>    Affects Versions: 0.4.0
>            Reporter: Roman Shaposhnik
>            Assignee: Sujay Rau
>             Fix For: 0.5.0
>         Attachments:,, ClusterManagerAPI.pdf
> We've come to a point where our tests need to have a uniform way of interfacing with
the cluster under test. It is no longer ok to assume that the test can be executed on a particular
node (and thus have access to services running on it). It is also less than ideal for tests
to assume a particular type of interaction with the services since it tends to break in different
deployment scenarios. 
> A framework that needs to be put in place has to be capable of (regardless of where a
test using it is executed on):
>   # representing the abstract configuration of the cluster
>   # representing the abstract topology of the entire cluster (services running on a cluster,
nodes hosting the daemons, racks, etc).
>   # giving tests an ability to query this topology
>   # giving tests an ability to affect the nodes in that topology in a particular way
(refreshing configuration, restarting services, etc.)
> Of course, the ideal solution here would be to give Bigtop tests a programmatic access
to a Hadoop cluster management framework such as Cloudera's CM or Apache Ambari. 
> As with any ideal solutions I don't think it is realistic though. Hence we have to cook
something up. At this point I'm really focused on getting the API right and I'm totally fine
with an implementation of that API to be something as silly as a bunch of ssh-based scripts
or something.
> This JIRA is primarily focused on coming up with such an API. Anybody who's willing to
help is welcome to.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message