hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13139) Branch-2: S3a to use thread pool that blocks clients
Date Mon, 13 Jun 2016 16:39:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327713#comment-15327713

Chris Nauroth commented on HADOOP-13139:

mvn test -Dtest=TestS3A*  -Pparallel-tests -DtestsThreadCount=2

[~stevel@apache.org], when you pass a {{-Dtest=}} argument, you are overriding the {{<includes>}}
list passed to Surefire during the build.  Passing {{TestS3A*}} matches all the S3A test suites,
including the ones I listed in {{<excludes>}} to prevent parallel execution.  I just
tested this, and it looks like if Surefire finds the same test matching both the whitelist
and the blacklist, then the whitelist wins.  Therefore, tests that can't handle parallel execution
end up running in the parallel phase.

Unfortunately, I think this means you can't reliably do both {{-Pparallel-tests}} and {{-Dtest=TestS3A*}}
in the same {{mvn}} run.  I had flagged all multi-part upload tests to run outside the parallel
phase because of the abort-existing-uploads-on-init logic, so I wouldn't expect them to work
well in the parallel phase.

It would be nice to be able to run only S3A tests in parallel for more rapid iterations on
patches that touch only S3A code.  I think we'd need to use some other naming convention on
all non-parallel-safe test suite classes to support that, so that you could specify a wildcard
to {{-Dtest=}} that wouldn't match those suites.

> Branch-2: S3a to use thread pool that blocks clients
> ----------------------------------------------------
>                 Key: HADOOP-13139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13139
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: fs/s3
>    Affects Versions: 2.8.0
>            Reporter: Pieter Reuse
>            Assignee: Pieter Reuse
>         Attachments: HADOOP-13139-001.patch, HADOOP-13139-branch-2-003.patch, HADOOP-13139-branch-2.001.patch,
> HADOOP-11684 is accepted into trunk, but was not applied to branch-2. I will attach a
patch applicable to branch-2.
> It should be noted in CHANGES-2.8.0.txt that the config parameter 'fs.s3a.threads.core'
has been been removed and the behavior of the ThreadPool for s3a has been changed.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message