hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3694) if MiniDFS startup time could be improved, testing time would be reduced
Date Wed, 09 Jul 2008 13:19:31 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612083#action_12612083

Steve Loughran commented on HADOOP-3694:

Turning off IPv6 doesnt help, just changes the stack trace:

    [junit] BEA JRockit(R) R27.4.0-90-89592-1.6.0_02-20070928-1715-linux-x86_64
    [junit] "Main Thread" id=1 idx=0x4 tid=11333 prio=5 alive, in native
    [junit]     at java/net/Inet4AddressImpl.getHostByAddr([B)Ljava/lang/String;(Native Method)
    [junit]     at java/net/InetAddress$1.getHostByAddr(InetAddress.java:853)
    [junit]     at java/net/InetAddress.getHostFromNameService(InetAddress.java:533)
    [junit]     at java/net/InetAddress.getHostName(InetAddress.java:476)
    [junit]     at java/net/InetAddress.getHostName(InetAddress.java:448)
    [junit]     at java/net/InetSocketAddress.getHostName(InetSocketAddress.java:210)
    [junit]     at org/apache/hadoop/hdfs/server/namenode/FSNamesystem.initialize(FSNamesystem.java:316)
    [junit]     at org/apache/hadoop/hdfs/server/namenode/FSNamesystem.<init>(FSNamesystem.java:263)
    [junit]     at org/apache/hadoop/hdfs/server/namenode/NameNode.initialize(NameNode.java:159)
    [junit]     at org/apache/hadoop/hdfs/server/namenode/NameNode.start(NameNode.java:224)
    [junit]     ^-- Holding lock: org/apache/hadoop/hdfs/server/namenode/NameNode@0x1132d80[thin
    [junit]     at org/apache/hadoop/util/Service.deploy(Service.java:261)
    [junit]     at org/apache/hadoop/hdfs/server/namenode/NameNode.createNameNode(NameNode.java:893)
    [junit]     at org/apache/hadoop/hdfs/MiniDFSCluster.<init>(MiniDFSCluster.java:274)
    [junit]     at org/apache/hadoop/hdfs/MiniDFSCluster.<init>(MiniDFSCluster.java:123)
    [junit]     at org/apache/hadoop/fs/TestCopyFiles.testCopyFromLocalToDfs(TestCopyFiles.java:271)
    [junit]     at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

Its the determining of the local hostname from the ipaddress that the DFS is listening on
that usually appears in the stack trace in a startup. Its not that is is particularly slow,
a few tens of seconds, but that being run on every testcase, it brings the system to its knees.
its also why the CPU load during testing is slow: they are not CPU bound, but driven by network

> if MiniDFS startup time could be improved, testing time would be reduced
> ------------------------------------------------------------------------
>                 Key: HADOOP-3694
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3694
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.19.0
>            Reporter: Steve Loughran
> Its taking me 140 minutes to run a test build; looking into the test results its the
20s startup delay of every MiniDFS cluster that is slowing things down. If we could find out
why it is taking so long and cut it down, every test case that relied on a cluster would be
speeded up. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message