flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Endre Major (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (FLUME-3146) Use public API HdfsDataOutputStream#getCurrentBlockReplication where applicable
Date Thu, 18 Oct 2018 08:23:00 GMT

     [ https://issues.apache.org/jira/browse/FLUME-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Endre Major reassigned FLUME-3146:
----------------------------------

    Assignee: Endre Major

> Use public API HdfsDataOutputStream#getCurrentBlockReplication where applicable
> -------------------------------------------------------------------------------
>
>                 Key: FLUME-3146
>                 URL: https://issues.apache.org/jira/browse/FLUME-3146
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Wei-Chiu Chuang
>            Assignee: Endre Major
>            Priority: Major
>         Attachments: FLUME-3146.001.patch
>
>
> AbstractHDFSWrite#reflectGetNumCurrentReplicas uses a private HDFS API, which causes
it to emit an confusing error message:
> {noformat}
> 2017-08-08 22:01:10,743 INFO org.apache.flume.sink.hdfs.AbstractHDFSWriter: FileSystem's
output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.crypto.CryptoOutputStream;
err=java.lang.NoSuchMethodException: org.apache.hadoop.crypto.CryptoOutputStream.getNumCurrentReplicas()
> 2017-08-08 22:02:11,086 ERROR org.apache.flume.sink.hdfs.AbstractHDFSWriter: Error while
trying to hflushOrSync!
> {noformat}
> This error message is emitted even with the most recent version of HDFS that has HDFS-826
for sure.
> Turns out that the relevant code is not valid with HDFS encryption. Instead of using
the current private (and deprecated) getNumCurrentReplicas#getNumCurrentReplicas() API, it
should use HdfsDataOutputStream#getCurrentBlockReplication, which is a public API and which
considers the case of HDFS encryption.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org


Mime
View raw message