hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
Date Wed, 13 Sep 2017 22:24:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165387#comment-16165387
] 

Rushabh S Shah commented on HDFS-11156:
---------------------------------------

I have one concern in this patch.
{code:title=NamenodeWebHdfsMethods.java|borderStyle=solid}
// Some comments here
public String get()
{
    case GETFILEBLOCKLOCATIONS:
    {
      final long offsetValue = offset.getValue();
      final Long lengthValue = length.getValue();

      FileSystem fs = FileSystem.get(conf != null ?
          conf : new Configuration());
      BlockLocation[] locations = fs.getFileBlockLocations(
          new org.apache.hadoop.fs.Path(fullpath),
          offsetValue,
          lengthValue != null? lengthValue: Long.MAX_VALUE);
      final String js = JsonUtil.toJsonString("BlockLocations",
          JsonUtil.toJsonMap(locations));
      return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
    }
}
{code}
In the above code snippet, why are we creating dfsClient object which will again create an
rpc to itself.
We are already in namenode. We just have to call relevant NamenodeRpcServer method.
Correct me if I am missing something.
Cc [~cheersyang] [~andrew.wang] [~liuml07] 

Also this fix introduced a bug. See more details here: HDFS-12442


> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
> ----------------------------------------------------
>
>                 Key: HDFS-11156
>                 URL: https://issues.apache.org/jira/browse/HDFS-11156
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>    Affects Versions: 2.7.3
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>             Fix For: 2.9.0, 3.0.0-alpha2
>
>         Attachments: BlockLocationProperties_JSON_Schema.jpg, BlockLocations_JSON_Schema.jpg,
FileStatuses_JSON_Schema.jpg, HDFS-11156.01.patch, HDFS-11156.02.patch, HDFS-11156.03.patch,
HDFS-11156.04.patch, HDFS-11156.05.patch, HDFS-11156.06.patch, HDFS-11156.07.patch, HDFS-11156.08.patch,
HDFS-11156.09.patch, HDFS-11156.10.patch, HDFS-11156.11.patch, HDFS-11156.12.patch, HDFS-11156.13.patch,
HDFS-11156.14.patch, HDFS-11156.15.patch, HDFS-11156.16.patch, HDFS-11156-branch-2.01.patch,
Output_JSON_format_v10.jpg, SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
>   "LocatedBlocks" : {
>     "fileLength" : 1073741824,
>     "isLastBlockComplete" : true,
>     "isUnderConstruction" : false,
>     "lastLocatedBlock" : { ... },
>     "locatedBlocks" : [ {...} ]
>   }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to *FileSystem* API,

> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be fixed. Marked
as Incompatible change as this will change the output of the GET_BLOCK_LOCATIONS API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message