hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
Date Wed, 19 Apr 2017 11:25:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974467#comment-15974467

Steve Loughran commented on HADOOP-14305:

I'm trying to think of what would make some good extra tests. There's already one for rename,
but it's for a file.

How about

* recursive rename of a 2-level directory tree, verify ls of directories find dirs, loading
files still works. listFiles(path, recursive=true) should also be used for listing, on account
of it being a different list mechanism.
* open , seek to end + read to -1, then seek to (0) and read there. Why? Helps verify that
filelengths match data, and that the re-opens done by seek() stay valid. 

Maybe we could also subclass a Huge test and add encryption, again using SSE-C as the worst-case
example. That will test multipart operations and large file copies. With that there'd be coverage
of the set of S3 verbs: PUT, mulipart POST, COPY, LIST, GET, HEAD.

IT might also be fun to do a getFileStatus() and open() on the stock landsat CSV, expecting
failures. We'll be able to make an assert about the error type/text. verify anything I do
w.r.t diagnostics text in HADOOP-14291 doesn't break the exception class type. (and provides
details on encryption type)

Note that I've added something to the release notes of HADOOP-13075, made clear this must
be considered experimental for now. I'm confident you think it's ready, but everything related
to S3 usually takes one iteration to stabilise. The phase II stuff all went through the HDCloud
4-6 week dev cycle before shipping in longer-lived releases, and yes, we found things. I'll
expect to see things here too, now that you are shipping it. If they go to you internally,
don't be shy of matching them with things related to the ASF code, test and docs.

> S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
> ---------------------------------------------------------------------------
>                 Key: HADOOP-14305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14305
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, test
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Moist
>            Priority: Minor
>         Attachments: HADOOP-14305-001.patch
> The S3a encryption tests all run in parallel (they were interfering with each other,
apparently). This adds ~1 min to the test runs.
> They should run in serial. That they fail when this is attempted due to Bad Auth problems
must be considered a serious problem, as is indicates issues related to working with SSE encryption
from hadoop

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message