hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lucas Lustosa Madureira (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10176) WebHdfs LISTSTATUS does not offer any sorting
Date Tue, 22 Nov 2016 00:27:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15685217#comment-15685217

Lucas Lustosa Madureira commented on HDFS-10176:


I gave it a try on resolving his issue.
As I understand it, the WebHDFS is supposed to expose the methods of [org.apache.hadoop.fs.FileSystem|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html].
As it stands, this class does not have a version of listStatus that supports ordering. Ideally,
to solving this issue involves adding such method to this class. Adding this method was easy
enough (take a look at [this patch|http://pastebin.com/raw/R4G5FjQq]).
The next step should be to modify the WebHDFS url to understand a new parameter, for instance:

curl -i  "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=LISTSTATUS[&orderBy=<ORDERBY>]"

Then it would be necessary to check if the parameter is set and call the appropriate version
of listStatus.
Unfortunately I could not figure this part. I could verify that the request to the URL above
is (at least partially) fulfilled by the class [org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java].
Looking at the class [org.apache.hadoop.hdfs.web.WebHdfsFileSystem|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java],
I got the impression that this class somehow causes the code on NamenodeWebHdfsMethods to
run. If that is what happens, I think I need to know which part of the code calls WebHdfsFileSystem.listStatus,
and apply there the changes proposed on the previous paragraph.
Unfortunatelly I could not figure this out, specially as there is no documentation of the
inner workings of the hadoop project.
To add to my confusion, there are also the classes at [org.apache.hadoop.http|https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http],
which I am not sure are related to WebHdfs or not (they also deal with http requests and hdfs).
Should I make no progress, at least this comment is a starting point for anyone who tries
to address this issue.

PS. How do you guys package a new hadoop source? It takes me almost half an hour to package
it with maven, even though I only change a few lines of a file.

> WebHdfs LISTSTATUS does not offer any sorting
> ---------------------------------------------
>                 Key: HDFS-10176
>                 URL: https://issues.apache.org/jira/browse/HDFS-10176
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: webhdfs
>    Affects Versions: 2.7.0
>            Reporter: Romain Rigaux
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#List_a_Directory
> Times, names, sizes would allow a Web client to offer a richer experience to its user:
> {code}
> {
>         "accessTime"      : 1320171722771,
>         "blockSize"       : 33554432,
>         "group"           : "supergroup",
>         "length"          : 24930,
>         "modificationTime": 1320171722771,
>         "owner"           : "webuser",
>         "pathSuffix"      : "a.patch",
>         "permission"      : "644",
>         "replication"     : 1,
>         "type"            : "FILE"
>       },
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message