hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8232) Provide a command line entry point to view/test topology options
Date Fri, 30 Mar 2012 17:34:29 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242574#comment-13242574
] 

Steve Loughran commented on HADOOP-8232:
----------------------------------------

test with a table that can be found; the table is
{code}
host1 /rack1
host2 /rack1
host3 /rack2/switch1
host4 /rack2/switch1
{code}

the output is
{code}
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table
org.apache.hadoop.net.TopologyTool testfile /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/hosts.txt
net.topology.node.switch.mapping.impl=org.apache.hadoop.net.TableMapping
Caching wrapper class = class org.apache.hadoop.net.TableMapping
Instance information: cached switch mapping relaying to TableMapping with table "/Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt"
Path: /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt
Map size: 0
Topology is not  considered single-switch
Resolving host1
Hostname "host1" resolved to "/rack1"  in 17.476 milliseconds
Resolving host2
Hostname "host2" resolved to "/rack1"  in 0.878 milliseconds
Resolving host3
Hostname "host3" resolved to "/rack2/switch1"  in 0.428 milliseconds
Resolving host4
Hostname "host4" resolved to "/rack2/switch1"  in 0.34 milliseconds
Resolving host5
Hostname "host5" resolved to "/default-rack"  in 0.578 milliseconds
Resolving host5
Hostname "host5" resolved to "/default-rack"  in 0.095 milliseconds
Resolving host3
Hostname "host3" resolved to "/rack2/switch1"  in 0.023 milliseconds
Resolving host1.hadoop.apache.org
Hostname "host1.hadoop.apache.org" resolved to "/default-rack"  in 0.555 milliseconds

Final topology:

Mapping: cached switch mapping relaying to TableMapping with table "/Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt"
Path: /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt
Map size: 4
Known mappings:
  host1.hadoop.apache.org -> /default-rack
  host5 -> /default-rack
  host4 -> /rack2/switch1
  host3 -> /rack2/switch1
  host2 -> /rack1
  host1 -> /rack1
Nodes: 6
Switches: 3
{code}

The output at the end is less than ideal. The "Map Size" value is from the {{TableMapping.toString()}}
method, which, in the patch, lists it. The name could be changed, but what it represents can't:
it is the size of the hostname->rack values provided by the table file.

The map that is then printed below it is the map known by the CachedDNSSwitchMapping, which
implements the method needed to get a copy of its mapping for diagnostics purposes. 

Ideally, both should be shown, which could be done by having the inner {{RawTableMapping}}
extend {{AbstractDNSToSwitchMapping}} and implement the {{getSwitchMap()}} method by returning
its static map. That still needs to be made visible somehow through {{CachedDNSMapping}} 

An alternative would just be for {{TableMapping.getSwitchMap()} to merge its cached values
with those provided by the original table, so displaying everything that came in the table,
and any cached values for unknown hosts now mapped to /default-rack.

                
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
>                 Key: HADOOP-8232
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8232
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.23.1
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters
topology setup. 
> The initial operations would be to list the implementation class of the mapper, and attempt
to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution
process.
> Target audience: 
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ... 
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
>  This version is intended for the ops team who have a list of hostnames, IP addresses.

> * Rather than just list them, the ops team may want to mandate that there were no /default-rack
mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message