hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8232) Provide a command line entry point to view/test topology options
Date Fri, 30 Mar 2012 17:24:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242566#comment-13242566

Steve Loughran commented on HADOOP-8232:

Test with table mapping pointing to a table that does not exist relative to the directory
where the hadoop command is being invoked from:
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/missing-table
org.apache.hadoop.net.TopologyTool test 4 home extra 
Caching wrapper class = class org.apache.hadoop.net.TableMapping
Instance information: cached switch mapping relaying to TableMapping with table "table.txt"
Path: /Users/slo/Java/Hadoop/github/trunk/table.txt
Map size: 0
Topology is not  considered single-switch
Resolving 4
2012-03-30 18:22:03,831 WARN  net.TableMapping (TableMapping.java:load(113)) - table.txt cannot
be read. /default-rack will be returned.
java.io.FileNotFoundException: table.txt (No such file or directory)
	at java.io.FileInputStream.open(Native Method)
	at java.io.FileInputStream.<init>(FileInputStream.java:120)
	at java.io.FileInputStream.<init>(FileInputStream.java:79)
	at java.io.FileReader.<init>(FileReader.java:41)
	at org.apache.hadoop.net.TableMapping$RawTableMapping.load(TableMapping.java:98)
	at org.apache.hadoop.net.TableMapping$RawTableMapping.resolve(TableMapping.java:132)
	at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
	at org.apache.hadoop.net.TopologyTool.resolveHostnames(TopologyTool.java:189)
	at org.apache.hadoop.net.TopologyTool.run(TopologyTool.java:103)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	at org.apache.hadoop.net.TopologyTool.main(TopologyTool.java:228)
Hostname "4" resolved to "/default-rack"  in 15.548 milliseconds
Resolving home
Hostname "home" resolved to "/default-rack"  in 6.812 milliseconds
Resolving extra
Hostname "extra" resolved to "/default-rack"  in 0.702 milliseconds

Final topology:

Mapping: cached switch mapping relaying to TableMapping with table "table.txt"
Path: /Users/slo/Java/Hadoop/github/trunk/table.txt
Map size: 0
Known mappings:
  home -> /default-rack
  extra -> /default-rack
  4 -> /default-rack
Nodes: 3
Switches: 1

The error message is printed inside the TableMapping, and is not visible to the caller. It
could be improved by listing the absolute path to the table.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>                 Key: HADOOP-8232
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8232
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.23.1
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-8232.patch
> Add a new command line entry point "topo" with commands for preflight checking of a clusters
topology setup. 
> The initial operations would be to list the implementation class of the mapper, and attempt
to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution
> Target audience: 
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ... 
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
>  This version is intended for the ops team who have a list of hostnames, IP addresses.

> * Rather than just list them, the ops team may want to mandate that there were no /default-rack
mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message