hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Chiu Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13877) HttpFS: Implement GETSNAPSHOTDIFF
Date Mon, 01 Oct 2018 14:03:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634063#comment-16634063
] 

Wei-Chiu Chuang commented on HDFS-13877:
----------------------------------------

Thanks [~smeng] for the patch. I should have suggested you to implement DistributedFileSystem#snapshotDiffReportListingRemoteIterator.

Quoting my comments in HDFS-13052:
{quote}I'm late to review this (coming from HDFS-13877). While this Jira provides a handy
snapshotdiff api, in practice this is not usable in production.

See: HDFS-12594 and HDFS-12165. In extreme cases I've seen getSnapshotDiffReport RPC sending
2GB protobuf message and failed. Even in not-so-extreme cases, since webhdfs server side runs
in NameNode process, NN heap usage change like that can easily fail SLA or even lead to fail
over.

Instead, we should implement something equivalent to DistributedFileSystem#snapshotDiffReportListingRemoteIterator.
Use the interface implemented in HDFS-12594 and return an iterator.
{quote}

> HttpFS: Implement GETSNAPSHOTDIFF
> ---------------------------------
>
>                 Key: HDFS-13877
>                 URL: https://issues.apache.org/jira/browse/HDFS-13877
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: httpfs
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>         Attachments: HDFS-13877.001.patch, HDFS-13877.001.patch
>
>
> Implement GETSNAPSHOTDIFF (from HDFS-13052) in HttpFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message