Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Fri, 30 Mar 2012 17:14:30 +0000 (UTC)
From: "Steve Loughran (Commented) (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: 
 <1725328690.38757.1333127670068.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1006003888.38571.1333125391641.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-8232) Provide a command line entry point
 to view/test topology options
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242553#comment-13242553 ] 

Steve Loughran commented on HADOOP-8232:
----------------------------------------

Testing with the {{echo}} script.
{code}
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/valid-script org.apache.hadoop.net.TopologyTool testfile /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/hosts.txt
net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping
Caching wrapper class = class org.apache.hadoop.net.ScriptBasedMapping
Mapping script filename= "echo"
Mapping script path = "/Users/slo/Java/Hadoop/github/trunk/echo"
2012-03-30 18:03:37,291 WARN  net.TopologyTool (TopologyTool.java:run(91)) - Script file not found -the script must be in the execution path
Instance information: script-based mapping with script "echo"
Topology is not  considered single-switch
Resolving host1
Hostname "host1" resolved to "host1"  in 24.4 milliseconds
Resolving host2
Hostname "host2" resolved to "host2"  in 12.681 milliseconds
Resolving host3
Hostname "host3" resolved to "host3"  in 9.753 milliseconds
Resolving host4
Hostname "host4" resolved to "host4"  in 10.084 milliseconds
Resolving host5
Hostname "host5" resolved to "host5"  in 9.431 milliseconds
Resolving host5
Hostname "host5" resolved to "host5"  in 0.128 milliseconds
Resolving host3
Hostname "host3" resolved to "host3"  in 0.056 milliseconds
Resolving host1.hadoop.apache.org
Hostname "host1.hadoop.apache.org" resolved to "host1.hadoop.apache.org"  in 8.377 milliseconds

Final topology:

Mapping: script-based mapping with script "echo"
Known mappings:
  host1.hadoop.apache.org -> host1.hadoop.apache.org
  host5 -> host5
  host4 -> host4
  host3 -> host3
  host2 -> host2
  host1 -> host1
Nodes: 6
Switches: 6

{code}

# the entry point first lists the mapping class and creates it.
# the statement {{Caching wrapper class = class org.apache.hadoop.net.ScriptBasedMapping}} means that the script has not been wrapped by a CachingDNS mapper, as it was not deemed necessary (maybe explain that better in the output?)
# the script filename is extracted from the configuration and printed
# the absolute path to the script is resolved -to avoid working directory confusion.
# because there is no file at that point the program warns that the file must be on the path (maybe: not use log.warn)
# the {{mapping.toString()}} operation is called to see what the string has to say about itself, I've been slowly making those useful while still robust.
# the topology is asked whether or not it considers itself single switch. As there is a non-null script, it is assumed to be multi-switch.
# the hosts are resolved with their times.
# when a cached hostname is resolved execution time drops significantly, showing caching works
# finally, the topology is dumped, which for the caching DNS mapper  means a dump of its (hostname, switch) map. 
# from the topology dump the number of hosts and switches is counted.

                
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
>                 Key: HADOOP-8232
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8232
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.23.1
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup. 
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience: 
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ... 
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
>  This version is intended for the ops team who have a list of hostnames, IP addresses. 
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira