hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shashikant Banerjee (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
Date Thu, 16 Nov 2017 15:15:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255468#comment-16255468
] 

Shashikant Banerjee edited comment on HDFS-12594 at 11/16/17 3:14 PM:
----------------------------------------------------------------------

Thanks [~szetszwo] , for the review comments. 
patch v8 addresses the same.

>>DFSUtilClient.bytes2byteArray and DFSUtil.bytes2byteArray are very similar but there
is a small difference when len == 0:
DFSUtilClient returns new byte[0][] and
DFSUtil returns new byte[][]{null}.
Is it a bug?

<<The mapping seems to be incorrect here:
EmptyString("")->{}(byte[])->byte[][]{null};
Reverse Mapping:
byte[][]{null}->byte[]{(byte) ("/") }->String("/")
I have addressed the problems in conversion of byte[][] to byte[] . Please have a look. 


was (Author: shashikant):
Thanks [~szetszwo] , for the review comments. 
patch v8 addresses the same.

>>DFSUtilClient.bytes2byteArray and DFSUtil.bytes2byteArray are very similar but there
is a small difference when len == 0:
DFSUtilClient returns new byte[0][] and
DFSUtil returns new byte[][]{null}.
Is it a bug?

<<The mapping seems to be incorrect here:
EmptyString("") -> {}(byte[]) -> byte[][]{null};
Reverse Mapping:
byte[][]{null} -> byte[]{(byte) ("/") } ->String("/")
I have addressed the problems in conversion of byte[][] to byte[] . Please have a look. 

> SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response
limit
> -------------------------------------------------------------------------------------------
>
>                 Key: HDFS-12594
>                 URL: https://issues.apache.org/jira/browse/HDFS-12594
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>            Reporter: Shashikant Banerjee
>            Assignee: Shashikant Banerjee
>         Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, HDFS-12594.003.patch,
HDFS-12594.004.patch, HDFS-12594.005.patch, HDFS-12594.006.patch, HDFS-12594.007.patch, HDFS-12594.008.patch,
SnapshotDiff_Improvemnets .pdf
>
>
> The snapshotDiff command fails if the snapshotDiff report size is larger than the configuration
value of ipc.maximum.response.length which is by default 128 MB. 
> Worst case, with all Renames ops in sanpshots each with source and target name equal
to MAX_PATH_LEN which is 8k characters, this would result in at 8192 renames.
>  
> SnapshotDiff is currently used by distcp to optimize copy operations and in case of the
the diff report exceeding the limit , it fails with the below exception:
> Test set: org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport)
 Time elapsed: 111.906 sec  <<< ERROR!
> java.io.IOException: Failed on local exception: org.apache.hadoop.ipc.RpcException: RPC
response exceeds maximum data length; Host Details : local host is: "hw15685.local/10.200.5.230";
destination host is: "localhost":59808;
> Attached is the proposal for the changes required.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message