hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9055) WebHDFS REST v2
Date Fri, 11 Sep 2015 18:42:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741337#comment-14741337

Haohui Mai commented on HDFS-9055:

bq. Adding a third protocol which nothing really supports yet doesn't fix REST. The ability
to use curl and wget is a feature, not a bug.

1. GRPC is over HTTP / HTTPS. Both curl and wget talks HTTP / HTTPS. You do that for the same
2. Experiences of WebHDFS development shows the burdens of maintaining REST APIs is quite
high. This suggests to some extent the approach of adding REST calls one by one is broken.
My point is that if the goal of webhdfs v2 is  fixing these problems, exposing these calls
automatically is a high priority from a development point of view.

bq. I'm hesitant to make the client do this work in the WebHDFS case because it's likely going
to be extremely expensive network-wise, especially over high latency networks. Worse, I can
easily see someone want to get the speed back by multi-threading the connections and effectively
DDoSing the NN.

Can you show some numbers on what are the latencies on data-center networks? If the NN takes
the read / write lock to perform the recursive requests, shouldn't it be considered as DDoS
as well?

> ---------------
>                 Key: HDFS-9055
>                 URL: https://issues.apache.org/jira/browse/HDFS-9055
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: webhdfs
>    Affects Versions: 3.0.0
>            Reporter: Allen Wittenauer
> There's starting to be enough changes to fix and add missing functionality to webhdfs
that we should probably update to REST v2.  This also gives us an opportunity to deal with
some incompatible issues.

This message was sent by Atlassian JIRA

View raw message