hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14872) CryptoInputStream should implement unbuffer
Date Thu, 28 Sep 2017 12:42:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184098#comment-16184098

Steve Loughran commented on HADOOP-14872:

LGTM +1, 
can you cut the extra blank line added to {{TestUnbuffer.java}} when you commit it?

Related to this JIRA, I've been instrumenting an s3a stream (and the FS itself), using a switch

I can't use the enums, as to get the string value I have to go StreamCapability.HSYNC.getValue(),
and you can only use string constants in enums. Nor can i use the enums in a switch as I'm
adding my own values. I've ended up just using "hsync" and "hflush" as string constants.
  public boolean hasCapability(String capability) {
    switch (capability.toLowerCase(Locale.ENGLISH)) {

      // does the output stream have delayed visibility
      return putTracker.isDelayedVisibility();

      // The flush/sync options are absolutely not supported
    case "hflush":
    case "hsync":
      return false;

        return false;
I think we will need to supplement the {{StreamCapability}} enums with a string constant for
every value

> CryptoInputStream should implement unbuffer
> -------------------------------------------
>                 Key: HADOOP-14872
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14872
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 2.6.4
>            Reporter: John Zhuge
>            Assignee: John Zhuge
>         Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, HADOOP-14872.003.patch,
HADOOP-14872.004.patch, HADOOP-14872.005.patch, HADOOP-14872.006.patch
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
>     DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, FSDataInputStream#unbuffer
will be called:
> {code:java}
>     try {
>       ((CanUnbuffer)in).unbuffer();
>     } catch (ClassCastException e) {
>       throw new UnsupportedOperationException("this stream does not " +
>           "support unbuffering.");
>     }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If the application
is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very well.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message