hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gera Shegalov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
Date Fri, 08 Jun 2018 17:48:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506322#comment-16506322
] 

Gera Shegalov commented on MAPREDUCE-7108:
------------------------------------------

Thanks for the patch.

Instead of the manual a loop of closing Readers you can use {{org.apache.hadoop.io.IOUtils#closeStreams}}}.

Since you are already changing this block, I think you can remove the catch block
{code:java}
} catch (Exception e) {
  fail("Fail to read from MapFileOutputFormat: " + e);
}{code}
because it does not look useful. And validation should be moved inside try as well whereas
fullyDelete be in finally. Would this make sense?
{code:java}
// Ensure getReaders call works and also ignores
// hidden filenames (_ or . prefixes)
MapFile.Reader[] readers = {};
try {
  readers = getReaders(outDir, conf);
  // validate output
  validateMapFileOutputContent(FileSystem.get(job.getConfiguration()), outDir);
} finally {
  IOUtils.closeStreams(r);
  FileUtil.fullyDelete(new File(outDir.toString()));
}{code}
 

 

 

> TestFileOutputCommitter fails on Windows
> ----------------------------------------
>
>                 Key: MAPREDUCE-7108
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 3.1.0, 2.9.1
>            Reporter: Zuoming Zhang
>            Priority: Minor
>              Labels: Windows
>             Fix For: 3.1.0, 2.9.1
>
>         Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, MAPREDUCE-7108.000.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> TestFileOutputCommitter.java fails on Windows.
> Reason:
> This is because in testMapFileOutputCommitterInternal, after executing MapFileOutputFormat.getReaders(outDir,
conf), the readers are not properly closed. So other apis that attempt to delete the existing
file fails due to the file handler already being used.
>  
> Solution:
> Close all readers after call MapFileOutputFormat.getReaders(outDir, conf);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message