hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
Date Mon, 30 Oct 2017 13:28:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16224959#comment-16224959
] 

Steve Loughran commented on HADOOP-14993:
-----------------------------------------

Looks nearly ready to go in. Optimisting listing can make a big diff in performance of object
stores; this will be a good improvement

* ignore the license issues; not your files
* do fix the checkstyles though.

L233: leave at IOE so when you need to add more exceptions, there's no incompatibilites with
exist ing code

L425 L456 : with SLF4J you don't need toString && String.format: these are expensive,
especially for debug-level ops.
Go {{LOG.debug("{} is a file", qualifiedPath)}}


{{AbstractContractGetFileStatusTest}} is the core of the listing tests, make sure they are
happy. Looks like you aren't skipping any which is good. 
A fun one is to iterate through the list until a NoSuchElementException is raised, that is

{code}
try {
  do {
   n = fs.next()
  }
} catch (NoSuchElementException ignored) {}

{code}

I think your code handles it, but do review it carefully just to make sure.

> AliyunOSS: Override listFiles and listLocatedStatus 
> ----------------------------------------------------
>
>                 Key: HADOOP-14993
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14993
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/oss
>    Affects Versions: 3.0.0-beta1
>            Reporter: Genmao Yu
>            Assignee: Genmao Yu
>         Attachments: HADOOP-14993.001.patch
>
>
> Do a bulk listing off all entries under a path in one single operation, there is no need
to recursively walk the directory tree.
> Updates:
> - override listFiles and listLocatedStatus by using bulk listing
> - some minor updates in hadoop-aliyun index.md



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message