hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jothi Padmanabhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4673) IFile.Writer close() uses compressor after returning it to CodecPool.
Date Tue, 18 Nov 2008 04:01:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648483#action_12648483
] 

Jothi Padmanabhan commented on HADOOP-4673:
-------------------------------------------

Yes, that needs to be fixed. The reason for that is because of IFileOutputStream.close just
does the checksum calculation, but does not call close of the underlying output stream. 
This possibly should be fixed by treating  it the same way as compressed output stream --
have the functionality of checksum calculation in finish() and have close call finish and
the underlying stream's close.

{code}

public void finish() {
  (if finished) {
    return;
  }
  else {
    finished = true;
    // calcualte checksum and write to
    // underlying stream
  }
}

 public void close() throws IOException {
    finish();
    out.close();
  }
{code}

>  IFile.Writer close() uses compressor after returning it to CodecPool.
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4673
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4673
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.18.1, 0.18.2
>            Reporter: Hong Tang
>            Assignee: Arun C Murthy
>
> The problem is of the same nature as HADOOP-4195.
> The compression codec is returned to the CodecPool, and later is finished in "out.close()".
> {code:title=IFile.java|borderStyle=solid}
>     public void close() throws IOException {
>       // Close the serializers
>       keySerializer.close();
>       valueSerializer.close();
>       // Write EOF_MARKER for key/value length
>       WritableUtils.writeVInt(out, EOF_MARKER);
>       WritableUtils.writeVInt(out, EOF_MARKER);
>       decompressedBytesWritten += 2 * WritableUtils.getVIntSize(EOF_MARKER);
>       
>       if (compressOutput) {
>         // Flush data from buffers into the compressor
>         out.flush();
>         
>         // Flush & return the compressor
>         compressedOut.finish();
>         compressedOut.resetState();
>         CodecPool.returnCompressor(compressor);
>         compressor = null;
>       }
>       // Close the stream
>       rawOut.flush();
>       compressedBytesWritten = rawOut.getPos() - start;
>       // Close the underlying stream iff we own it...
>       if (ownOutputStream) {
>         out.close();
>       }
>       out = null;
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message