hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manoj Govindassamy (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-11353) Speed the unit tests relevant to DataNode volume failure testing
Date Fri, 20 Jan 2017 23:07:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832575#comment-15832575
] 

Manoj Govindassamy edited comment on HDFS-11353 at 1/20/17 11:06 PM:
---------------------------------------------------------------------

Thanks for working on this [~linyiqun]. 

* In {{testSuccessiveVolumeFailures}}, by placing {{DataNodeTestUtils.checkDiskErrorSync()}}
we are making it to synchronously wait for the error check to complete which was otherwise
not there before. 

* {noformat}
    DataNodeTestUtils.injectDataDirFailure(dn1Vol1, dn2Vol1);
    Path file1 = new Path("/test1");
    DFSTestUtil.createFile(fs, file1, 1024, (short)3, 1L);
    DFSTestUtil.waitReplication(fs, file1, (short)3);
{noformat} 
All these {{injectDataDirFailure}} are followed by createFile. In my test create file on failed
volume is immediately caught by {{handleDiskError}} which is what would {{DataNodeTestUtils.checkDiskErrorSync()}}
also do, right ? So, createFile in a way simulating the needed checkDiskErrorSync() ?


was (Author: manojg):
Thanks for working on this [~linyiqun]. 

* In {{testSuccessiveVolumeFailures}}, by placing {{DataNodeTestUtils.checkDiskErrorSync()}}
we are making it to synchronously wait for the error check to complete which was otherwise
not there before. 

* {noformat}
    DataNodeTestUtils.injectDataDirFailure(dn1Vol1, dn2Vol1);
    Path file1 = new Path("/test1");
    DFSTestUtil.createFile(fs, file1, 1024, (short)3, 1L);
    DFSTestUtil.waitReplication(fs, file1, (short)3);
{noformat} 
* All these {{injectDataDirFailure}} are followed by createFile. In my test create file on
failed volume is immediately caught by {{handleDiskError}} which is what would {{DataNodeTestUtils.checkDiskErrorSync()}}
also do, right ? So, createFile in a way simulating the needed checkDiskErrorSync() ?

> Speed the unit tests relevant to DataNode volume failure testing
> ----------------------------------------------------------------
>
>                 Key: HDFS-11353
>                 URL: https://issues.apache.org/jira/browse/HDFS-11353
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Yiqun Lin
>            Assignee: Yiqun Lin
>         Attachments: HDFS-11353.001.patch
>
>
> Currently there are many tests which start with {{TestDataNodeVolumeFailure*}} frequently
run timedout or failed. I found one failure test in recent Jenkins building. The stack info:
> {code}
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.testSuccessiveVolumeFailures
> java.util.concurrent.TimeoutException: Timed out waiting for DN to die
> 	at org.apache.hadoop.hdfs.DFSTestUtil.waitForDatanodeDeath(DFSTestUtil.java:702)
> 	at org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.testSuccessiveVolumeFailures(TestDataNodeVolumeFailureReporting.java:208)
> {code}
> The related codes:
> {code}
>     /*
>      * Now fail the 2nd volume on the 3rd datanode. All its volumes
>      * are now failed and so it should report two volume failures
>      * and that it's no longer up. Only wait for two replicas since
>      * we'll never get a third.
>      */
>     DataNodeTestUtils.injectDataDirFailure(dn3Vol2);
>     Path file3 = new Path("/test3");
>     DFSTestUtil.createFile(fs, file3, 1024, (short)3, 1L);
>     DFSTestUtil.waitReplication(fs, file3, (short)2);
>     // The DN should consider itself dead
>     DFSTestUtil.waitForDatanodeDeath(dns.get(2));
> {code}
> Here the code waits for the datanode failed all the volume and then become dead. But
it timed out. We can do an additional operation {{DataNodeTestUtils.checkDiskErrorSync}} to
speed the error check for here. And this has been done in many similar places after doing
{{DataNodeTestUtils.injectDataDirFailure}} in test {{TestDataNodeVolumeFailure}}.
> I suppose that recent {{TestDataNodeVolumeFailure*}} failure test can also be improved
by this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message