hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6339) SequenceFile writer does not properly flush stream with external DataOutputStream
Date Wed, 28 Oct 2009 19:56:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771083#action_12771083
] 

Aaron Kimball commented on HADOOP-6339:
---------------------------------------

I agree that the app should close the underlying OutputStream. But after the SequenceFile.Writer
is closed, its data should be available in the file. SequenceFile.Writer.close() in fact calls
flush() on the underlying DataOutputStream. It is bizarre that this data is not available
to other readers. I still consider this a bug.

at SequenceFile.java, line 985:
{code}
        // Close the underlying stream iff we own it...
        if (ownOutputStream) {
          out.close();
        } else {
          out.flush(); // <-- This does not seem to work right.
        }
        out = null;
{code}



> SequenceFile writer does not properly flush stream with external DataOutputStream
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6339
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1
>            Reporter: Jonathan Hsieh
>
> When using the SequenceFile.createWriter(..,FSDataOutputStream, ...) method to create
a Writer, data is not flushed when the encapsulating SequenceFile is closed.
> Example test case skeleton:
> {code}
> public void testWhyFail() throws IOException {
>     // There a was a failure case using :
>     Configuration conf = ... ;
>     Path path = new Path("file:///tmp/testfile");
>     FileSystem hdfs = path.getFileSystem(conf);
>     // writing
>     FSDataOutputStream dos = hdfs.create(path);
>     hdfs.deleteOnExit(path);
>     // it is specifically with this writer.
>     Writer writer = SequenceFile.createWriter(conf, dos,
>         WriteableEventKey.class, WriteableEvent.class,
>         SequenceFile.CompressionType.NONE, new DefaultCodec());
>     Writable value = ...;
>     Writable key = ...;
>     writer.append(key, value);
>     writer.sync();
>     writer.close();
>     // Test fails unless I close the underlying FSDataOutputStream handle with the line
below.
>     //    dos.close(); 
>     
>     // WTF: nothing written by this writer!
>     FileStatus stats = hdfs.getFileStatus(path);
>     assertTrue(stats.getLen() > 0);
>     // it should have written something but it failed.
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message