hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liaoyuxiangqin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
Date Tue, 13 Jun 2017 03:43:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16047349#comment-16047349
] 

liaoyuxiangqin edited comment on HDFS-11943 at 6/13/17 3:42 AM:
----------------------------------------------------------------

Thanks [~andrew.wang] for review, i have get the call stack to  respond for Kai's question,
because hdfs use heap buffer default, so RawErasureEncoder.encode will call AbstractNativeRawEncoder.doEncode,
detial encoding stack information as  follows:

WARN rawcoder.AbstractNativeRawEncoder: convertToByteBufferState is invoked, not efficiently.
Please use direct ByteBuffer inputs/outputs
java.lang.Exception: this is ec write log
at org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawEncoder.doEncode(AbstractNativeRawEncoder.java:69)
at org.apache.hadoop.io.erasurecode.rawcoder.RawErasureEncoder.encode(RawErasureEncoder.java:87)
at org.apache.hadoop.hdfs.DFSStripedOutputStream.encode(DFSStripedOutputStream.java:367)
at org.apache.hadoop.hdfs.DFSStripedOutputStream.writeParityCells(DFSStripedOutputStream.java:909)
at org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:995)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:829)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:67)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:126)
at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:485)
at 

h4. In addition, the create Encoder stack as follows:

at org.apache.hadoop.io.erasurecode.ErasureCodeNative.loadLibrary(Native Method)
at org.apache.hadoop.io.erasurecode.ErasureCodeNative.<clinit>(ErasureCodeNative.java:46)
at org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawEncoder.<clinit>(NativeXORRawEncoder.java:33)
at org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory.createEncoder(NativeXORRawErasureCoderFactory.java:32)
at org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoderWithFallback(CodecUtil.java:206)
at org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoder(CodecUtil.java:152)
at org.apache.hadoop.hdfs.DFSStripedOutputStream.<init>(DFSStripedOutputStream.java:303)
at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:305)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1214)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1193)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1131)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:447)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:444)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:458)



was (Author: liaoyuxiangqin):
Thanks [~andrew.wang] for review, i have get the call stack to  respond for Kai's question,
because hdfs use heap buffer default, so RawErasureEncoder.encode will call AbstractNativeRawEncoder.doEncode,
detial stack information as  follows:

WARN rawcoder.AbstractNativeRawEncoder: convertToByteBufferState is invoked, not efficiently.
Please use direct ByteBuffer inputs/outputs
java.lang.Exception: this is ec write log
at org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawEncoder.doEncode(AbstractNativeRawEncoder.java:69)
at org.apache.hadoop.io.erasurecode.rawcoder.RawErasureEncoder.encode(RawErasureEncoder.java:87)
at org.apache.hadoop.hdfs.DFSStripedOutputStream.encode(DFSStripedOutputStream.java:367)
at org.apache.hadoop.hdfs.DFSStripedOutputStream.writeParityCells(DFSStripedOutputStream.java:909)
at org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:995)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:829)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:67)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:126)
at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:485)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:407)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228)
at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:286)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119)
at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:326)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)


> Warn log frequently print to screen in doEncode function on  AbstractNativeRawEncoder
class
> -------------------------------------------------------------------------------------------
>
>                 Key: HDFS-11943
>                 URL: https://issues.apache.org/jira/browse/HDFS-11943
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: erasure-coding, native
>    Affects Versions: 3.0.0-alpha4
>         Environment: cluster: 3 nodes
> os:(Red Hat 2.6.33.20,  Red Hat 3.10.0-514.6.1.el7.x86_64, Ubuntu4.4.0-31-generic)
> hadoop version: hadoop-3.0.0-alpha4
> erasure coding: XOR-2-1-64k and enabled Intel ISA-L
> hadoop fs -put file /
>            Reporter: liaoyuxiangqin
>            Assignee: liaoyuxiangqin
>            Priority: Minor
>         Attachments: HDFS-11943.002.patch, HDFS-11943.patch
>
>   Original Estimate: 0.05h
>  Remaining Estimate: 0.05h
>
>  when i write file to hdfs on above environment,  the hdfs client  frequently print warn
log of use direct ByteBuffer inputs/outputs in doEncode function to screen, detail information
as follows:
> 2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: convertToByteBufferState
is invoked, not efficiently. Please use direct ByteBuffer inputs/outputs



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message