hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shashikant Banerjee (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
Date Wed, 25 Oct 2017 09:39:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218301#comment-16218301
] 

Shashikant Banerjee edited comment on HDFS-12594 at 10/25/17 9:38 AM:
----------------------------------------------------------------------

[~ehiggs], Thanks for the review comments. I will upload a patch addressing the review comments
soon.

[~szetszwo], thanks for the review comments. 
Implementing RemoteIterator class so that rpc calls are made on demand while consuming the
diff may not be possible here. We need the entire diff  which is collection of modify list
,createList and DeleteList generated at the namenode with each RPC, so as to exactly figure
out the Renames across directories within the Snapshottable Root Directory at the client itself.
All the Renames within the Snapshottable directory can be figured out after the complete diff
processing of the snapshottable directory tree.


was (Author: shashikant):
[~ehiggs], Thanks for the review comments. I will upload a patch with review comments soon.

[~szetszwo], thanks for the review comments. 
Implementing RemoteIterator class so that rpc calls are made on demand while consuming the
diff may not be possible here. We need the entire diff  which is collection of modify list
,createList and DeleteList generated at the namenode with each RPC, so as to exactly figure
out the Renames across directories within the Snapshottable Root Directory at the client itself.
All the Renames within the Snapshottable directory can be figured out after the complete diff
processing of the snapshottable directory tree.

> SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response
limit
> -------------------------------------------------------------------------------------------
>
>                 Key: HDFS-12594
>                 URL: https://issues.apache.org/jira/browse/HDFS-12594
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>            Reporter: Shashikant Banerjee
>            Assignee: Shashikant Banerjee
>         Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, HDFS-12594.003.patch,
SnapshotDiff_Improvemnets .pdf
>
>
> The snapshotDiff command fails if the snapshotDiff report size is larger than the configuration
value of ipc.maximum.response.length which is by default 128 MB. 
> Worst case, with all Renames ops in sanpshots each with source and target name equal
to MAX_PATH_LEN which is 8k characters, this would result in at 8192 renames.
>  
> SnapshotDiff is currently used by distcp to optimize copy operations and in case of the
the diff report exceeding the limit , it fails with the below exception:
> Test set: org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport)
 Time elapsed: 111.906 sec  <<< ERROR!
> java.io.IOException: Failed on local exception: org.apache.hadoop.ipc.RpcException: RPC
response exceeds maximum data length; Host Details : local host is: "hw15685.local/10.200.5.230";
destination host is: "localhost":59808;
> Attached is the proposal for the changes required.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message