Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 11 Apr 2017 18:02:42 +0000 (UTC)
From: "Xiaobing Zhou (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13060745.1490981840000.258627.1491933762049@Atlassian.JIRA>
In-Reply-To: <JIRA.13060745.1490981840000@Atlassian.JIRA>
References: <JIRA.13060745.1490981840000@Atlassian.JIRA> <JIRA.13060745.1490981840790@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HDFS-11608) HDFS write crashed with block size
 greater than 2 GB
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 11 Apr 2017 18:02:47 -0000


    [ https://issues.apache.org/jira/browse/HDFS-11608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15964731#comment-15964731 ] 

Xiaobing Zhou commented on HDFS-11608:
--------------------------------------

Posted 2.7 patch. Thanks [~xyao] for committing it and all for reviews.

> HDFS write crashed with block size greater than 2 GB
> ----------------------------------------------------
>
>                 Key: HDFS-11608
>                 URL: https://issues.apache.org/jira/browse/HDFS-11608
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.8.0
>            Reporter: Xiaobing Zhou
>            Assignee: Xiaobing Zhou
>            Priority: Critical
>             Fix For: 2.9.0, 3.0.0-alpha3, 2.8.1
>
>         Attachments: HDFS-11608.000.patch, HDFS-11608.001.patch, HDFS-11608.002.patch, HDFS-11608.003.patch, HDFS-11608-branch-2.7.003.patch
>
>
> We've seen HDFS write crashes in the case of huge block size. For example, writing a 3 GB file using block size > 2 GB (e.g., 3 GB), HDFS client throws out of memory exception. DataNode gives out IOException. After changing heap size limit,  DFSOutputStream ResponseProcessor exception is seen followed by Broken pipe and pipeline recovery.
> Give below:
> DN exception,
> {noformat}
> 2017-03-30 16:34:33,828 ERROR datanode.DataNode (DataXceiver.java:run(278)) - c6401.ambari.apache.org:50010:DataXceiver error processing WRITE_BLOCK operation  src: /192.168.64.101:47167 dst: /192.168.64.101:50010
> java.io.IOException: Incorrect value for packet payload size: 2147483128
>         at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:159)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
>         at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:502)
>         at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:898)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:806)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org