hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-12994) Specify PositionedReadable, add contract tests, fix problems
Date Sun, 03 Apr 2016 21:23:25 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated HADOOP-12994:
    Attachment: HADOOP-12994-001.patch

patch 001, including 

# fs specification doc to define readFully properly
# tests in Seek contract test (the operations go with seek; the default implementation uses
# argument validation across hadoop-common filesystems
# argument validation for DFS client
# s3a to downgrade an EOF exception in a read() to a -1. That is what HDFS does, and allows
PositionedReadable.read() to work

I also unrolled s3a's readFully to avoid so many expensive seek calls; we may want to think
about pulling that into a separate patch as its an optimisation and not needed.

tested on: local, hdfs, s3a against amazon ireland.
not tested on: swift, azure

> Specify PositionedReadable, add contract tests, fix problems
> ------------------------------------------------------------
>                 Key: HADOOP-12994
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12994
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-12994-001.patch
> Some work on S3a has shown up that there aren't tests catching regressions in readFully,
reviewing the documentation shows that its specification could be improved.
> # review the spec
> # review the implementations
> # add tests (proposed: to the seek contract; streams which support seek should support
positioned readable)
> # fix code, where it differs significantly from HDFS or LocalFS

This message was sent by Atlassian JIRA

View raw message