hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
Date Thu, 14 Sep 2017 15:29:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166451#comment-16166451
] 

Daryn Sharp commented on HDFS-11156:
------------------------------------

-1.  Let's file another jira to fix immediately or revert this one.  The NN *cannot* make
RPC calls to itself.  Other than completely unnecessary, it can cause all the jetty threads
to be exhausted if they get into a retry loop with NN because it's too busy, in safemode,
standby, etc.   Notice that other webhdfs calls do not do this.

There are no compatibility issues and no api needs to be changed.  On the server-side, call
the existing method and encode to same json format as today.  On the client-side, just decode
straight into a BlockLocation[].

> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
> ----------------------------------------------------
>
>                 Key: HDFS-11156
>                 URL: https://issues.apache.org/jira/browse/HDFS-11156
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>    Affects Versions: 2.7.3
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>             Fix For: 2.9.0, 3.0.0-alpha2
>
>         Attachments: BlockLocationProperties_JSON_Schema.jpg, BlockLocations_JSON_Schema.jpg,
FileStatuses_JSON_Schema.jpg, HDFS-11156.01.patch, HDFS-11156.02.patch, HDFS-11156.03.patch,
HDFS-11156.04.patch, HDFS-11156.05.patch, HDFS-11156.06.patch, HDFS-11156.07.patch, HDFS-11156.08.patch,
HDFS-11156.09.patch, HDFS-11156.10.patch, HDFS-11156.11.patch, HDFS-11156.12.patch, HDFS-11156.13.patch,
HDFS-11156.14.patch, HDFS-11156.15.patch, HDFS-11156.16.patch, HDFS-11156-branch-2.01.patch,
Output_JSON_format_v10.jpg, SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
>   "LocatedBlocks" : {
>     "fileLength" : 1073741824,
>     "isLastBlockComplete" : true,
>     "isUnderConstruction" : false,
>     "lastLocatedBlock" : { ... },
>     "locatedBlocks" : [ {...} ]
>   }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to *FileSystem* API,

> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be fixed. Marked
as Incompatible change as this will change the output of the GET_BLOCK_LOCATIONS API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message