hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Broberg (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8615) EOFException in DecompressorStream.java needs to be more verbose
Date Wed, 25 Jul 2012 06:41:35 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422037#comment-13422037

Tim Broberg commented on HADOOP-8615:

Harsh, I'm under some pretty Draconian schedule pressure right now. I did take a look at this
none the less, and it doesn't look like a quick job. Some issues to consider:

 - Will all the codecs throw EOFException in this case, or will some throw IOException?
 - What if a decompressor gets a consistency check? Should it hit this same case? Certainly
it should throw IOException rather than EOFException.
 - Clients include SequenceFile, TFile, offlineImageViewer, FSImageCompression, a few classes
in mapred(uce), rumen, gridmix and various tests - no small patch.
 - Soon we should address compression stream usage in general to address reducing copies.
Lots of places we buffer the stream when the CompressionInputStream is already buffering,
resulting in still more copies. If we need to pick through these one by one, perhaps that
would be a good time to touch up error handling as well?

Sorry to wimp out on you, but it seems like a little much right now.
> EOFException in DecompressorStream.java needs to be more verbose
> ----------------------------------------------------------------
>                 Key: HADOOP-8615
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8615
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Jeff Lord
> In ./src/core/org/apache/hadoop/io/compress/DecompressorStream.java
> The following exception should at least pass back the file that it encounters this error
in relation to:
>   protected void getCompressedData() throws IOException {
>     checkStream();
>     int n = in.read(buffer, 0, buffer.length);
>     if (n == -1) {
>       throw new EOFException("Unexpected end of input stream");
>     }
> This would help greatly to debug bad/corrupt files.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message