hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liaoyuxiangqin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
Date Tue, 13 Jun 2017 03:37:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16047379#comment-16047379
] 

liaoyuxiangqin edited comment on HDFS-11943 at 6/13/17 3:36 AM:
----------------------------------------------------------------

Thanks [~Sammi] for review on this. The RawErasureEncoder.encode call relation as  follows:

{code:title=RawErasureEncoder.java|borderStyle=solid}
 if (usingDirectBuffer) {
      doEncode(bbeState);
    } else {
      ByteArrayEncodingState baeState = bbeState.convertToByteArrayState();
      doEncode(baeState);  //AbstractNativeRawEncoder.doEncode
    }
{code}

After the test, i find hdfs default use heap buffer, so the usingDirectBuffer is false,
and call AbstractNativeRawEncoder.doEncode print log frequent. In addition, after i modify
default value of buffer type, usingDirectBuffer  change to true and the call stack is change
too and frequent log disappeared.

So as you guess, the NativeXORRawEncoder doesn't indicate itself support the direct buffer.


was (Author: liaoyuxiangqin):
Thanks [~Sammi] for review on this. The RawErasureEncoder.encode call relation as  follows:

{code:title=RawErasureEncoder.java|borderStyle=solid}
 if (usingDirectBuffer) {
      doEncode(bbeState);
    } else {
      ByteArrayEncodingState baeState = bbeState.convertToByteArrayState();
      doEncode(baeState);  //AbstractNativeRawEncoder.doEncode
    }
{code}

After the test, i find hdfs default use heap buffer, so the usingDirectBuffer is false,
and call AbstractNativeRawEncoder.doEncode print log frequent. In addition, after i modify
default value of buffer type, usingDirectBuffer  change to true and the call stack is change
too and frequent log disappeared.

So as you guess, the NativeXORRawEncoder doesn't indicate itself support the direct buffe.

> Warn log frequently print to screen in doEncode function on  AbstractNativeRawEncoder
class
> -------------------------------------------------------------------------------------------
>
>                 Key: HDFS-11943
>                 URL: https://issues.apache.org/jira/browse/HDFS-11943
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: erasure-coding, native
>    Affects Versions: 3.0.0-alpha4
>         Environment: cluster: 3 nodes
> os:(Red Hat 2.6.33.20,  Red Hat 3.10.0-514.6.1.el7.x86_64, Ubuntu4.4.0-31-generic)
> hadoop version: hadoop-3.0.0-alpha4
> erasure coding: XOR-2-1-64k and enabled Intel ISA-L
> hadoop fs -put file /
>            Reporter: liaoyuxiangqin
>            Assignee: liaoyuxiangqin
>            Priority: Minor
>         Attachments: HDFS-11943.002.patch, HDFS-11943.patch
>
>   Original Estimate: 0.05h
>  Remaining Estimate: 0.05h
>
>  when i write file to hdfs on above environment,  the hdfs client  frequently print warn
log of use direct ByteBuffer inputs/outputs in doEncode function to screen, detail information
as follows:
> 2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: convertToByteBufferState
is invoked, not efficiently. Please use direct ByteBuffer inputs/outputs



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message