hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13139) Branch-2: S3a to use thread pool that blocks clients
Date Tue, 22 Aug 2017 14:12:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136843#comment-16136843

Jason Lowe commented on HADOOP-13139:

Right.  If >=2.8.x configs had a fs.s3a.threads.core=10 setting in core-default then this
would have "worked" for <=2.7.x.  I say that in quotes since the old code would be running
with a default of core=10 and max=10 which is a bit less than the 15 core threads and far
less than the 256 max threads it defaulted to before.  I'm not familiar with S3AFileSystem
and don't know if that's reasonable, hence my asking what others thought should be done, if
anything, for this scenario.  At first glance it's a bit weird to be mixing Hadoop versions
between the job client and job which is required to hit this problem, but running via Oozie
makes this more likely to happen in practice.  In hindsight it would have been a smoother
transition to abandon the old config names and use new configs.

> Branch-2: S3a to use thread pool that blocks clients
> ----------------------------------------------------
>                 Key: HADOOP-13139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13139
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: fs/s3
>    Affects Versions: 2.8.0
>            Reporter: Pieter Reuse
>            Assignee: Pieter Reuse
>             Fix For: 2.8.0
>         Attachments: HADOOP-13139-001.patch, HADOOP-13139-branch-2.001.patch, HADOOP-13139-branch-2.002.patch,
HADOOP-13139-branch-2-003.patch, HADOOP-13139-branch-2-004.patch, HADOOP-13139-branch-2-005.patch,
> HADOOP-11684 is accepted into trunk, but was not applied to branch-2. I will attach a
patch applicable to branch-2.
> It should be noted in CHANGES-2.8.0.txt that the config parameter 'fs.s3a.threads.core'
has been been removed and the behavior of the ThreadPool for s3a has been changed.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message