hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
Date Wed, 07 Dec 2016 14:38:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728894#comment-15728894
] 

Weiwei Yang edited comment on HDFS-11156 at 12/7/16 2:38 PM:
-------------------------------------------------------------

Hello [~andrew.wang]

Thanks for your comments. I just uploaded v8 patch with following changes

bq.  unit test for the fallback behavior

Added an unit test for the fallback code. Mocked an old version of server, tested 3 scenarios,
1) Server doesn't recognize GETFILEBLOCKLOCATIONS and send back an error. 2) Server handles
GET_BLOCK_LOCATIONS and returns a sample of data. 3) Server sends out a random exception.

bq.  separate functions

I am still using single function to handle that, but I added more java doc ahead. The reason
is 1) Separate them would have majority of code similar 2) I checked the times of this function
gets called while doing the fallback UT, this is easier with single method.

bq. The example curl and response are indented by 9 instead of 8

I am following the indents format with rest of the document. They are also 9. Anyhow I checked
the indents again in the doc I added, it should be OK now.

bq. provide an example response for a file with more than one block

Done

bq. I don't think this schema works

I verified the schema again with this [online tool|http://www.jsoneditoronline.org/], and
compared with FileStatuses/FileStatus schemas (they are similar). See [Sample response|^SampleResponse_JSON.jpg],
[BlockLocations JSON schema|^BlockLocations_JSON_Schema.jpg] which looks just like [FileStatuses
JSON schema|^FileStatuses_JSON_Schema.jpg], and [BlockLocation Properties Schema |^BlockLocationProperties_JSON_Schema.jpg]It
looks good to me now.  

Thank you and let me know if you have any further comments.


was (Author: cheersyang):
Hello [~andrew.wang]

Thanks for your comments. I just uploaded v8 patch with following changes

bq.  unit test for the fallback behavior

Added an unit test for the fallback code. Mocked an old version of server, tested 3 scenarios,
1) Server doesn't recognize GETFILEBLOCKLOCATIONS and send back an error. 2) Server handles
GET_BLOCK_LOCATIONS and returns a sample of data. 3) Server sends out a random exception.

bq.  separate functions

I am still using single function to handle that, but I added more java doc ahead. The reason
is 1) Separate them would have majority of code similar 2) I checked the times of this function
gets called while doing the fallback UT, this is easier with single method.

bq. The example curl and response are indented by 9 instead of 8

I am following the indents format with rest of the document. They are also 9. Anyhow I checked
the indents again in the doc I added, it should be OK now.

bq. provide an example response for a file with more than one block

Done

bq. I don't think this schema works

I verified the schema again with this [online tool|http://www.jsoneditoronline.org/], and
compared with FileStatuses/FileStatus schemas (they are similar). It looks good to me now.
 

Thank you and let me know if you have any further comments.

> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
> ----------------------------------------------------
>
>                 Key: HDFS-11156
>                 URL: https://issues.apache.org/jira/browse/HDFS-11156
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>    Affects Versions: 2.7.3
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>         Attachments: BlockLocationProperties_JSON_Schema.jpg, BlockLocations_JSON_Schema.jpg,
FileStatuses_JSON_Schema.jpg, HDFS-11156.01.patch, HDFS-11156.02.patch, HDFS-11156.03.patch,
HDFS-11156.04.patch, HDFS-11156.05.patch, HDFS-11156.06.patch, HDFS-11156.07.patch, HDFS-11156.08.patch,
SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
>   "LocatedBlocks" : {
>     "fileLength" : 1073741824,
>     "isLastBlockComplete" : true,
>     "isUnderConstruction" : false,
>     "lastLocatedBlock" : { ... },
>     "locatedBlocks" : [ {...} ]
>   }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to *FileSystem* API,

> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be fixed. Marked
as Incompatible change as this will change the output of the GET_BLOCK_LOCATIONS API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message