whirr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (WHIRR-693) Control order of actions with waves of whirr.instance-templates
Date Sun, 03 Feb 2013 05:14:13 GMT

    [ https://issues.apache.org/jira/browse/WHIRR-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569704#comment-13569704

Steve Loughran commented on WHIRR-693:

# This could be good for more complex deployments, though there's a risk of workflow-related
feature creep (see below). 
# This could mark the time for moving beyond .properties files, as with multiple templates,
overridden attributes and now sequences, the requirements of more configuration management
languages crop up. Rather than propose a new one, have a look at JSON.
# There really needs to be probes in Whirr to trigger starting one phase on the observed state
of the previous set. That way rather than just saying "run HDFS", the HDFS phase is considered
to have successfully completed when a probe of the NN returned the filesystem was up (Hitting
URLS and getting 200 responses is the single most relevant probe here).
# Teardown gets trickier as you'd have to go back in order from what state you got to, worry
about timing etc. This is why a "say no to teardown" policy, relying on the PaaS infrastructure
to kill your VMs, is the only rational tactic here. 

Having probes to defined the barriers between phases -I'm using that term as its the traditional
one- makes it possible to define phase transition not as "all the previously started action
scripts finished", but "the cluster was put in the state we needed for the next actions".
URLs are the obvious choice

> Control order of actions with waves of whirr.instance-templates
> ---------------------------------------------------------------
>                 Key: WHIRR-693
>                 URL: https://issues.apache.org/jira/browse/WHIRR-693
>             Project: Whirr
>          Issue Type: Bug
>          Components: core
>            Reporter: Paul Baclace
>            Priority: Critical
> A cluster can be specified with "waves" by supporting an optional suffix on whirr.instance-templates
property in order to provide *running* base-level services (like ZK or a master-master mysql)
to another layer of services during the configuration phase. This enables storing cluster-wide
information in zookeeper or doing db creation beforehand. 
> An example of a two "wave" (avoiding "phase" which already has a meaning) cluster:
>   whirr.instance-templates.0=1 zookeeper, 2 zookeeper, 4 noop
> The .0 templates run like normal during "whirr launch-cluster" and then, during the same
Whirr run, the .1 template is applied as a modification of the same cluster so that new host
instances are not allocated:
>   whirr.instance-templates.1=1 my-master, 2 my-gateway, 4 my-worker
> In the second wave, instance provisioning is inhibited, the instance-templates must have
the same number of commas and same sequence of leading numbers, and a role is only allowed
to appear in one wave. 
> Here is another example:
>   whirr.instance-templates.0=1 mysql-master+zookeeper, 1 mysql-master+zookeeper,4 noop
>   whirr.instance-templates.1=1 hadoop-namenode+hadoop-jobtracker, 1 hbase-master+hadoop-secondarynn,4
> In the first wave, the two mysql-masters form a multi-master ensemble which keeps state
information about the setup in the Whirr process (much like ZooKeeperCluster.getHosts(cluster)
informs services/hbase of the quorum).
> In the second wave, nodes are not allocated (BootstrapClusterAction.doAction() is inhibited
if instances already exist), but all phases for LaunchClusterCommand, including beforeBootstrap()
and afterBootstrap() callbacks, are executed.  If other whirr.cli.command.*Command are run,
they would see a combined whirr.instance-templates that works like normal.
> Obviously, there would be no reason to limit this to 2 waves, but I do not expect more
than 10 waves to be useful, so the pattern could enable a suffix of "\.[0-9]". 
> Instead of WHIRR-221 which aims to specify a single global order for service startup,
the wave format has the advantage that it relies on the implicit synchronization barriers
at phases already supported by Whirr, the phases still run actions within phases in parallel,
and state generated by a wave is available to later waves.
> Conceptually, this merely splits the whirr.instance-templates into waves of (bootstrap,install,configure,start).
If no .[0-9] suffixes are present, then Whirr would behave just like normal.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message