ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yusaku Sako (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMBARI-2011) Add Hosts gets stuck at 33% (some hosts in the cluster were down)
Date Wed, 24 Apr 2013 03:29:17 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640041#comment-13640041
] 

Yusaku Sako commented on AMBARI-2011:
-------------------------------------

There were problems with the predicates for the API calls to install and start host components
on the newly added hosts.
The code was simply setting all host components' state from INIT to INSTALLED to install,
and setting all Ganglia Monitors, HBase RegionServers, DataNodes, and TaskTrackers that are
in INSTALLED state to STARTED across all hosts in the cluster.

Add Hosts got stuck at 33%, because the call to set host components' state from INSTALLED
to STARTED, HostRoles/state=INSTALLED was matching host components on other hosts whose live
state was STOP_FAILED.  Since STOP_FAILED->STARTED is not a valid transition, the entire
call was failing.

The fix is to do the following instead:
* Installing host components on the newly added hosts (both original call and install retry):
{noformat}
PUT /api/v1/clusters/cluster/host_components
{"RequestInfo":{"context":"Install Components","query":"HostRoles/host_name.in([comma-delimited
list of hostnames])"},"Body":{"HostRoles":{"state":"INSTALLED"}}}
{noformat}

* Starting host components on the newly added hosts
PUT /api/v1/clusters/cluster/host_components
{noformat}
{"RequestInfo":{"context":"Start Components","query":"HostRoles/component_name.in(GANGLIA_MONITOR,HBASE_REGIONSERVER,DATANODE,TASKTRACKER)&HostRoles/state=INSTALLED&HostRoles/host_name.in([comma-delimited
list of hostnames])"},"Body":{"HostRoles":{"state":"STARTED"}}}
{noformat}
                
> Add Hosts gets stuck at 33% (some hosts in the cluster were down)
> -----------------------------------------------------------------
>
>                 Key: AMBARI-2011
>                 URL: https://issues.apache.org/jira/browse/AMBARI-2011
>             Project: Ambari
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 1.2.0
>            Reporter: Yusaku Sako
>            Priority: Critical
>             Fix For: 1.3.0
>
>
> While adding new hosts, Add Host Wizard got stuck at 33%. After checking the "requests"
resource via the API, it seemed like the API to start the host components either never made
it to the server or the server errored out and did not properly finish to creating a request.
> When adding hosts, we noticed that there are STOP tasks for the TASKTRACKER on hosts
that were not even included as part of the add host wizard (seems like those belonged to hosts
that were either non-reachable or hosts that partially failed to be added).
> The new host was successfully added, but the DN, TT, and GM were never started. After
waiting for a while, refresh on the page showed a green progress bar at 100% for the overall
progress, and a blue 100% bar for the host added.
> We tried adding another set of hosts, and the wizard got stuck at 33% again.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message