ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Roberts <srobe...@hortonworks.com>
Subject Re: decommission multiple nodes issue
Date Mon, 02 Mar 2015 19:34:32 GMT
Greg - Same here on submitting JSON. Although they are JSON documents you have to submit them
as plain form. This is true across all of Ambari. I opened a bug for it a month back.


--
Hortonworks - We do Hadoop

Sean Roberts
Partner Solutions Engineer - EMEA
@seano

From: Greg Hill <greg.hill@rackspace.com><mailto:greg.hill@rackspace.com>
Date: March 2, 2015 at 19:32:34
To: Sean Roberts <sroberts@hortonworks.com>><mailto:sroberts@hortonworks.com>,
user@ambari.apache.org <user@ambari.apache.org>><mailto:user@ambari.apache.org>
Subject:  Re: decommission multiple nodes issue

That causes a server error.  I’ve yet to see any part of the API that accepts JSON arrays
like that as input; it’s almost always, if not always, a comma-separated string like I posted.
 Many methods even return double-encoded JSON values (i.e. “key”: “[\”value1\”,\”value2\”]").
 It’s kind of annoying and inconsistent, honestly, and not documented anywhere.  You just
have to have your client code choke on it and then go add another data[key] = json.loads(data[key])
in the client to account for it.

I am starting to think it’s because I set the nodes into maintenance mode first, as doing
the decommission command manually from the client works fine when the nodes aren’t in maintenance
mode.  I’ll keep digging, I guess, but it is weird that the exact same command worked this
time (the commandArgs are identical to the one that did nothing).

Greg

From: Sean Roberts <sroberts@hortonworks.com<mailto:sroberts@hortonworks.com>>
Date: Monday, March 2, 2015 at 1:22 PM
To: Greg <greg.hill@rackspace.com<mailto:greg.hill@rackspace.com>>, "user@ambari.apache.org<mailto:user@ambari.apache.org>"
<user@ambari.apache.org<mailto:user@ambari.apache.org>>
Subject: Re: decommission multiple nodes issue


Racker Greg - I’m not familiar with the decommissioning API, but if it’s consistent with
the rest of Ambari, you’ll need to change from this:

"excluded_hosts": “slave-1.local,slave-2.local"

To this:

"excluded_hosts" : [ "slave-1.local","slave-2.local" ]


--
Hortonworks - We do Hadoop

Sean Roberts
Partner Solutions Engineer - EMEA
@seano

From: Greg Hill <greg.hill@rackspace.com><mailto:greg.hill@rackspace.com>
Reply: user@ambari.apache.org<mailto:user@ambari.apache.org> <user@ambari.apache.org>><mailto:user@ambari.apache.org>
Date: March 2, 2015 at 19:08:13
To: user@ambari.apache.org<mailto:user@ambari.apache.org> <user@ambari.apache.org>><mailto:user@ambari.apache.org>
Subject:  decommission multiple nodes issue

I have some code for decommissioning datanodes prior to removal.  It seems to work fine with
a single node, but with multiple nodes it fails.  When passing multiple hosts, I am putting
the names in a comma-separated string, as seems to be the custom with other Ambari API commands.
 I attempted to send it as a JSON array, but the server complained about that.  Let me know
if that is the wrong format.  The decommission request completes successfully, it just never
writes the excludes file so no nodes are decommissioned.

This fails for mutiple nodes:

"RequestInfo": {
                "command": "DECOMMISSION",
                "context": "Decommission DataNode”),
                "parameters": {"slave_type": “DATANODE", "excluded_hosts": “slave-1.local,slave-2.local"},
                "operation_level": {
“level”: “CLUSTER”,
“cluster_name”: cluster_name
},
            },
            "Requests/resource_filters": [{
                "service_name": “HDFS",
                "component_name": “NAMENODE",
            }],

But this works for a single node:

"RequestInfo": {
                "command": "DECOMMISSION",
                "context": "Decommission DataNode”),
                "parameters": {"slave_type": “DATANODE", "excluded_hosts": “slave-1.local"},
                "operation_level": {
“level”: “HOST_COMPONENT”,
“cluster_name”: cluster_name,
“host_name”: “slave-1.local”,
“service_name”: “HDFS”
},
            },
            "Requests/resource_filters": [{
                "service_name": “HDFS",
                "component_name": “NAMENODE",
            }],

Looking on the actual node, it’s obvious that the file isn’t being written by the command
output:

(multiple hosts, notice there is no ‘Writing File’ line)
File['/etc/hadoop/conf/dfs.exclude'] {'owner': 'hdfs', 'content': Template('exclude_hosts_list.j2'),
'group': 'hadoop'}
Execute[''] {'user': 'hdfs'}
ExecuteHadoop['dfsadmin -refreshNodes'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin',
'conf_dir': '/etc/hadoop/conf', 'kinit_override': True, 'user': 'hdfs'}
Execute['hadoop --config /etc/hadoop/conf dfsadmin -refreshNodes'] {'logoutput': False, 'path':
['/usr/hdp/current/hadoop-client/bin'], 'tries': 1, 'user': 'hdfs', 'try_sleep': 0}

(single host, it writes the exclude file)
File['/etc/hadoop/conf/dfs.exclude'] {'owner': 'hdfs', 'content': Template('exclude_hosts_list.j2'),
'group': 'hadoop'}
Writing File['/etc/hadoop/conf/dfs.exclude'] because contents don't match
Execute[''] {'user': 'hdfs'}
ExecuteHadoop['dfsadmin -refreshNodes'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin',
'conf_dir': '/etc/hadoop/conf', 'kinit_override': True, 'user': 'hdfs'}
Execute['hadoop --config /etc/hadoop/conf dfsadmin -refreshNodes'] {'logoutput': False, 'path':
['/usr/hdp/current/hadoop-client/bin'], 'tries': 1, 'user': 'hdfs', 'try_sleep': 0}

The only notable difference in the command.json is the commandParams/excluded_hosts param,
so it’s not like the request is passing the information along incorrectly.  I’m going
to play around with the format I use to pass it in and take some wild guesses like it’s
expecting double-encoded JSON as I’ve seen that in other places, but if someone knows the
answer offhand and can help out, that would be appreciated.  If it turns out to be a bug in
Ambari, I’ll open a JIRA and rewrite our code to issue the decommission call independently
for each host.

Greg
Mime
View raw message