Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 9 Nov 2017 18:16:00 +0000 (UTC)
From: "John Zhuge (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13102679.1505507836000.198578.1510251360374@Atlassian.JIRA>
In-Reply-To: <JIRA.13102679.1505507836000@Atlassian.JIRA>
References: <JIRA.13102679.1505507836000@Atlassian.JIRA> <JIRA.13102679.1505507836241@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-14872) CryptoInputStream should
 implement unbuffer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 09 Nov 2017 18:16:06 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246199#comment-16246199 ] 

John Zhuge commented on HADOOP-14872:
-------------------------------------

Ran "test-patch" locally and got all +1s except these expected javac deprecation warnings. Committing.
{noformat}
[WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:[42,46] [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:[251,30] [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:[252,33] [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java:[32,46] [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:[40,46] [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java:[201,16] [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java:[204,16] [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:[362,25] [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:[364,25] [deprecation] StreamCapability in StreamCapabilities has been deprecated
{noformat}


> CryptoInputStream should implement unbuffer
> -------------------------------------------
>
>                 Key: HADOOP-14872
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14872
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, security
>    Affects Versions: 2.6.4
>            Reporter: John Zhuge
>            Assignee: John Zhuge
>         Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
>     DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, FSDataInputStream#unbuffer will be called:
> {code:java}
>     try {
>       ((CanUnbuffer)in).unbuffer();
>     } catch (ClassCastException e) {
>       throw new UnsupportedOperationException("this stream does not " +
>           "support unbuffering.");
>     }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very well.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org