hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kuhu Shukla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10743) MiniDFSCluster test runtimes can be drastically reduce
Date Mon, 15 Aug 2016 03:49:22 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420595#comment-15420595
] 

Kuhu Shukla commented on HDFS-10743:
------------------------------------

[~linyiqun], Thanks for the patch! I have been testing this change( with only 1 second heartbeat
interval and no changes to cachereport msec) to look for test failures caused by this change.
On example is TestDataNodeVolumeFailure. I have the patch almost ready but need a clean run
without the change for my testing. Is it ok with you if I continue working on this? Appreciate
your valuable feedback.

> MiniDFSCluster test runtimes can be drastically reduce
> ------------------------------------------------------
>
>                 Key: HDFS-10743
>                 URL: https://issues.apache.org/jira/browse/HDFS-10743
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>    Affects Versions: 2.0.0-alpha
>            Reporter: Daryn Sharp
>            Assignee: Kuhu Shukla
>         Attachments: HDFS-10743.001.patch
>
>
> {{MiniDFSCluster}} tests have excessive runtimes.  The main problem appears to be the
heartbeat interval.  The NN may have to wait up to 3s (default value) for all DNs to heartbeat,
triggering registration, so NN can go active.  Tests that repeatedly restart the NN are severely
affected.
> Example for varying heartbeat intervals for {{TestFSImageWithAcl}}:
> * 3s = ~70s -- (disgusting, why I investigated)
> * 1s = ~27s
> * 500ms = ~17s -- (had to hack DNConf for millisecond precision)
> That a 4x improvement in runtime.
> 17s is still excessively long for what the test does.  Further areas to explore when
running tests:
> * Reduce numerous sleeps intervals in DN's {{BPServiceActor}}.
> * Ensure heartbeats and initial BR are sent immediately upon (re)registration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message