hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10312) Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.
Date Wed, 20 Apr 2016 00:06:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15248984#comment-15248984
] 

Chris Nauroth commented on HDFS-10312:
--------------------------------------

[~arpitagarwal], I like your suggestion for speeding up the test.  Unfortunately, I think
this doesn't quite give us the same test coverage.  To demonstrate this, apply patch v004,
then revert the src/main changes, and then run the test.  It will fail on a protobuf decoding
exception.  That's exactly the condition we want to test, and the src/main changes make the
test pass.  After applying the delta, that's no longer true.  The test passes with or without
the src/main changes.  That's because with the smaller block report sizes, we don't hit the
internal protobuf default of 64 MB maximum.  Using a block report size of 6000000, we definitely
push over 64 MB for the RPC message size, so we definitely trigger the right condition.

> Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length
restriction.
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10312
>                 URL: https://issues.apache.org/jira/browse/HDFS-10312
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HDFS-10312.001.patch, HDFS-10312.002.patch, HDFS-10312.003.patch,
HDFS-10312.004.patch
>
>
> Our RPC server caps the maximum size of incoming messages at 64 MB by default.  For exceptional
circumstances, this can be uptuned using {{ipc.maximum.data.length}}.  However, for block
reports, there is still an internal maximum length restriction of 64 MB enforced by protobuf.
 (Sample stack trace to follow in comments.)  This issue proposes to apply the same override
to our block list decoding, so that large block reports can proceed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message