hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
Date Mon, 19 Jun 2017 13:54:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054033#comment-16054033

Kai Zheng commented on HDFS-11943:

Thanks [~liaoyuxiangqin] & [~Sammi] for the discussions!

bq. Do you have any idea that NativeXORRawEncoder should support direct Buffer or not? I think
the answer is yes. Correct me if it's not the case.
This is correct. Should all native coders indicate they prefer direct bytebuffer for better

bq. so i think the condition not neccessary to add
This is correct. Agree.

bq. So as you guess, the NativeXORRawEncoder doesn't indicate itself support the direct buffer.
A good reasoning! Actually you got it, that's the cause. Ensuring to have the following block
in the base native class {{AbstractNativeRawEncoder/AbstractNativeRawDecoder}} should assure
all native coders will behave as the thought.
  public boolean preferDirectBuffer() {
    return true;
[~liaoyuxiangqin] would you update your patch to fix this? Thanks!

> Warn log frequently print to screen in doEncode function on  AbstractNativeRawEncoder
> -------------------------------------------------------------------------------------------
>                 Key: HDFS-11943
>                 URL: https://issues.apache.org/jira/browse/HDFS-11943
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: erasure-coding, native
>    Affects Versions: 3.0.0-alpha4
>         Environment: cluster: 3 nodes
> os:(Red Hat,  Red Hat 3.10.0-514.6.1.el7.x86_64, Ubuntu4.4.0-31-generic)
> hadoop version: hadoop-3.0.0-alpha4
> erasure coding: XOR-2-1-64k and enabled Intel ISA-L
> hadoop fs -put file /
>            Reporter: liaoyuxiangqin
>            Assignee: liaoyuxiangqin
>            Priority: Minor
>         Attachments: HDFS-11943.002.patch, HDFS-11943.patch
>   Original Estimate: 0.05h
>  Remaining Estimate: 0.05h
>  when i write file to hdfs on above environment,  the hdfs client  frequently print warn
log of use direct ByteBuffer inputs/outputs in doEncode function to screen, detail information
as follows:
> 2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: convertToByteBufferState
is invoked, not efficiently. Please use direct ByteBuffer inputs/outputs

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message