flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Turcsanyi (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (FLUME-3146) Use public API HdfsDataOutputStream#getCurrentBlockReplication where applicable
Date Thu, 03 Jan 2019 09:45:00 GMT

     [ https://issues.apache.org/jira/browse/FLUME-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Peter Turcsanyi reassigned FLUME-3146:

    Assignee: Peter Turcsanyi  (was: Endre Major)

> Use public API HdfsDataOutputStream#getCurrentBlockReplication where applicable
> -------------------------------------------------------------------------------
>                 Key: FLUME-3146
>                 URL: https://issues.apache.org/jira/browse/FLUME-3146
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Wei-Chiu Chuang
>            Assignee: Peter Turcsanyi
>            Priority: Major
>         Attachments: FLUME-3146.001.patch
> AbstractHDFSWrite#reflectGetNumCurrentReplicas uses a private HDFS API, which causes
it to emit an confusing error message:
> {noformat}
> 2017-08-08 22:01:10,743 INFO org.apache.flume.sink.hdfs.AbstractHDFSWriter: FileSystem's
output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.crypto.CryptoOutputStream;
err=java.lang.NoSuchMethodException: org.apache.hadoop.crypto.CryptoOutputStream.getNumCurrentReplicas()
> 2017-08-08 22:02:11,086 ERROR org.apache.flume.sink.hdfs.AbstractHDFSWriter: Error while
trying to hflushOrSync!
> {noformat}
> This error message is emitted even with the most recent version of HDFS that has HDFS-826
for sure.
> Turns out that the relevant code is not valid with HDFS encryption. Instead of using
the current private (and deprecated) getNumCurrentReplicas#getNumCurrentReplicas() API, it
should use HdfsDataOutputStream#getCurrentBlockReplication, which is a public API and which
considers the case of HDFS encryption.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org

View raw message