hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8232) Provide a command line entry point to view/test topology options
Date Fri, 30 Mar 2012 17:34:29 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242574#comment-13242574

Steve Loughran commented on HADOOP-8232:

test with a table that can be found; the table is
host1 /rack1
host2 /rack1
host3 /rack2/switch1
host4 /rack2/switch1

the output is
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table
org.apache.hadoop.net.TopologyTool testfile /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/hosts.txt
Caching wrapper class = class org.apache.hadoop.net.TableMapping
Instance information: cached switch mapping relaying to TableMapping with table "/Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt"
Path: /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt
Map size: 0
Topology is not  considered single-switch
Resolving host1
Hostname "host1" resolved to "/rack1"  in 17.476 milliseconds
Resolving host2
Hostname "host2" resolved to "/rack1"  in 0.878 milliseconds
Resolving host3
Hostname "host3" resolved to "/rack2/switch1"  in 0.428 milliseconds
Resolving host4
Hostname "host4" resolved to "/rack2/switch1"  in 0.34 milliseconds
Resolving host5
Hostname "host5" resolved to "/default-rack"  in 0.578 milliseconds
Resolving host5
Hostname "host5" resolved to "/default-rack"  in 0.095 milliseconds
Resolving host3
Hostname "host3" resolved to "/rack2/switch1"  in 0.023 milliseconds
Resolving host1.hadoop.apache.org
Hostname "host1.hadoop.apache.org" resolved to "/default-rack"  in 0.555 milliseconds

Final topology:

Mapping: cached switch mapping relaying to TableMapping with table "/Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt"
Path: /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt
Map size: 4
Known mappings:
  host1.hadoop.apache.org -> /default-rack
  host5 -> /default-rack
  host4 -> /rack2/switch1
  host3 -> /rack2/switch1
  host2 -> /rack1
  host1 -> /rack1
Nodes: 6
Switches: 3

The output at the end is less than ideal. The "Map Size" value is from the {{TableMapping.toString()}}
method, which, in the patch, lists it. The name could be changed, but what it represents can't:
it is the size of the hostname->rack values provided by the table file.

The map that is then printed below it is the map known by the CachedDNSSwitchMapping, which
implements the method needed to get a copy of its mapping for diagnostics purposes. 

Ideally, both should be shown, which could be done by having the inner {{RawTableMapping}}
extend {{AbstractDNSToSwitchMapping}} and implement the {{getSwitchMap()}} method by returning
its static map. That still needs to be made visible somehow through {{CachedDNSMapping}} 

An alternative would just be for {{TableMapping.getSwitchMap()} to merge its cached values
with those provided by the original table, so displaying everything that came in the table,
and any cached values for unknown hosts now mapped to /default-rack.

> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>                 Key: HADOOP-8232
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8232
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.23.1
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-8232.patch
> Add a new command line entry point "topo" with commands for preflight checking of a clusters
topology setup. 
> The initial operations would be to list the implementation class of the mapper, and attempt
to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution
> Target audience: 
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ... 
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
>  This version is intended for the ops team who have a list of hostnames, IP addresses.

> * Rather than just list them, the ops team may want to mandate that there were no /default-rack
mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message