hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9055) WebHDFS REST v2
Date Mon, 14 Sep 2015 23:40:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744506#comment-14744506

Colin Patrick McCabe commented on HDFS-9055:

A big part of the goal of webhdfs is to support distCp between clusters on different major
versions of Hadoop.  We never guaranteed that with our RPC protocol (no, not even RPCv9),
so we need a protocol that does guarantee it.  Bumping the major version of webhdfs does nothing
to advance that goal, and in fact it makes it harder by creating a different, incompatible
"flavor" of webhdfs.  We retired the trusty old HFTP protocol because webhdfs was the do-all,
be-all solution for cross-version compatibility.  I think it's not unfair to ask webhdfs to
actually provide that compatibility!

Some of the JIRAs here seem to be moving processing from the client to the NameNode.  For
example, HDFS-9058 proposes that we implement "find" on the NameNode and send the results
back to the client.  HDFS-9051 proposes that the NameNode support recursive listing of directories
(probably similar to getContent etc.).  I am concerned that this will lead to extra complexity
on the NameNode, and-- at least if implemented in the obvious way-- longer latencies since
the FSN lock will be held for longer for the "do it all" operation.  I am not convinced that
the benefits outweigh the extra complexity and potential scalability issues.

As [~cnauroth] commented, all of the JIRAs up here besides HDFS-7822  are just new features
(like adding snapshot or truncate support) that can easily be done without bumping the major
version of libwebhdfs.  And even for that JIRA, [~kihwal]'s original proposal for HDFS-7822
was a compatible one.

Unless there is something I am missing, we should simply add the features that make sense
to WebHDFS v1 rather than creating an incompatible version.

> ---------------
>                 Key: HDFS-9055
>                 URL: https://issues.apache.org/jira/browse/HDFS-9055
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: webhdfs
>    Affects Versions: 3.0.0
>            Reporter: Allen Wittenauer
> There's starting to be enough changes to fix and add missing functionality to webhdfs
that we should probably update to REST v2.  This also gives us an opportunity to deal with
some incompatible issues.

This message was sent by Atlassian JIRA

View raw message