hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10743) MiniDFSCluster test runtimes can be drastically reduce
Date Wed, 10 Aug 2016 16:47:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415565#comment-15415565

Daryn Sharp commented on HDFS-10743:

Don't have the bandwidth to handle ensuring all test cases continue to work if the default
is dropped for tests - some tests are surely predicated on assumptions of the heartbeat interval.
 This is an excellent opportunity for someone else to be a hero.

> MiniDFSCluster test runtimes can be drastically reduce
> ------------------------------------------------------
>                 Key: HDFS-10743
>                 URL: https://issues.apache.org/jira/browse/HDFS-10743
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>    Affects Versions: 2.0.0-alpha
>            Reporter: Daryn Sharp
> {{MiniDFSCluster}} tests have excessive runtimes.  The main problem appears to be the
heartbeat interval.  The NN may have to wait up to 3s (default value) for all DNs to heartbeat,
triggering registration, so NN can go active.  Tests that repeatedly restart the NN are severely
> Example for varying heartbeat intervals for {{TestFSImageWithAcl}}:
> * 3s = ~70s -- (disgusting, why I investigated)
> * 1s = ~27s
> * 500ms = ~17s -- (had to hack DNConf for millisecond precision)
> That a 4x improvement in runtime.
> 17s is still excessively long for what the test does.  Further areas to explore when
running tests:
> * Reduce numerous sleeps intervals in DN's {{BPServiceActor}}.
> * Ensure heartbeats and initial BR are sent immediately upon (re)registration.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message