Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 16 Oct 2017 15:38:00 +0000 (UTC)
From: "Shashikant Banerjee (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13107228.1507210353000.7193.1508168280648@Atlassian.JIRA>
In-Reply-To: <JIRA.13107228.1507210353000@Atlassian.JIRA>
References: <JIRA.13107228.1507210353000@Atlassian.JIRA> <JIRA.13107228.1507210353782@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if
 the snapshotDiff report exceeds the RPC response limit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 16 Oct 2017 15:38:06 -0000


     [ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shashikant Banerjee updated HDFS-12594:
---------------------------------------
    Attachment: SnapshotDiff_Improvemnets .pdf

Replacing the earlier uploaded doc with the updated doc.

> SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
> -------------------------------------------------------------------------------------------
>
>                 Key: HDFS-12594
>                 URL: https://issues.apache.org/jira/browse/HDFS-12594
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>            Reporter: Shashikant Banerjee
>            Assignee: Shashikant Banerjee
>         Attachments: HDFS-12594.001.patch, SnapshotDiff_Improvemnets .pdf
>
>
> The snapshotDiff command fails if the snapshotDiff report size is larger than the configuration value of ipc.maximum.response.length which is by default 128 MB. 
> Worst case, with all Renames ops in sanpshots each with source and target name equal to MAX_PATH_LEN which is 8k characters, this would result in at 8192 renames.
>  
> SnapshotDiff is currently used by distcp to optimize copy operations and in case of the the diff report exceeding the limit , it fails with the below exception:
> Test set: org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport)  Time elapsed: 111.906 sec  <<< ERROR!
> java.io.IOException: Failed on local exception: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; Host Details : local host is: "hw15685.local/10.200.5.230"; destination host is: "localhost":59808;
> Attached is the proposal for the changes required.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org