hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-13106) if fs.s3a.block.size option == 0, use partition size option for blocksize
Date Fri, 06 May 2016 11:05:13 GMT
Steve Loughran created HADOOP-13106:
---------------------------------------

             Summary: if fs.s3a.block.size option == 0, use partition size option for blocksize
                 Key: HADOOP-13106
                 URL: https://issues.apache.org/jira/browse/HADOOP-13106
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 2.7.2
            Reporter: Steve Loughran
            Assignee: Steve Loughran
            Priority: Minor


Input perf is best if blocksize matches the partition size used to upload the data.

There's currently no way of knowing this —but S3A does know the partition size to use when
uploading data. 

I propose picking that up as the blocksize if the configured value of {{fs.s3a.block.size}}
is zero. This value is utterly illegal today (more precisely, it will break any app using
it to calculate splits), even when there is no check for its range



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message