hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-6339) SequenceFile writer does not properly flush stream with external DataOutputStream
Date Wed, 28 Oct 2009 00:23:59 GMT
SequenceFile writer does not properly flush stream with external DataOutputStream
---------------------------------------------------------------------------------

                 Key: HADOOP-6339
                 URL: https://issues.apache.org/jira/browse/HADOOP-6339
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.20.1
            Reporter: Jonathan Hsieh


When using the SequenceFile.createWriter(..,FSDataOutputStream, ...) method to create a Writer,
data is not flushed when the encapsulating SequenceFile is closed.

Example test case skeleton:
{code}
public void testWhyFail() throws IOException {

    // There a was a failure case using :
    Configuration conf = ... ;
    Path path = new Path("file:///tmp/testfile");
    FileSystem hdfs = path.getFileSystem(conf);

    // writing
    FSDataOutputStream dos = hdfs.create(path);
    hdfs.deleteOnExit(path);

    // it is specifically with this writer.
    Writer writer = SequenceFile.createWriter(conf, dos,
        WriteableEventKey.class, WriteableEvent.class,
        SequenceFile.CompressionType.NONE, new DefaultCodec());

    Writable value = ...;
    Writable key = ...;

    writer.append(key, value);
    writer.sync();
    writer.close();

    // Test fails unless I close the underlying FSDataOutputStream handle with the line below.
    //    dos.close(); 
    
    // WTF: nothing written by this writer!
    FileStatus stats = hdfs.getFileStatus(path);
    assertTrue(stats.getLen() > 0);
    // it should have written something but it failed.
  }
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message