hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13680) fs.s3a.readahead.range to use getLongBytes
Date Mon, 24 Oct 2016 10:14:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601540#comment-15601540

Steve Loughran commented on HADOOP-13680:

Ooh I'd forgotten about this; I'd one a quick PoC patch and then stashed it off, while I was
doing HADOOP-13560. I've applied that to branch-2 and stuck it up [https://github.com/steveloughran/hadoop/tree/s3/HADOOP-13680-get-long-bytes].
I don't want to attach the patch or link it as a PR here as it will only cause confusion.

Here it is [https://github.com/apache/hadoop/pull/145]

Comparing the two: 
* I added  {{longBytesOption}} to {{S3AUtils}} for broader use; by using it in {{getMultipartSizeProperty}}.
then {{S3ABlockOutputStream}} gets it for its sizes too
* we both patched {{S3AFileSystem}}
* you added a test, which makes it vastly superior.
* neither of us documented this.

How about we combine them: my changes to s3a Test utils, your changes on s3a FS and the test,
and the docs get updated to indicated that byte values can be declared (we can do this in
the examples and core-default for completeness); 

...if you do that, I could probably stay untainted enough to be the reviewer....

> fs.s3a.readahead.range to use getLongBytes
> ------------------------------------------
>                 Key: HADOOP-13680
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13680
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
> The {{fs.s3a.readahead.range}} value is measured in bytes, but can be hundreds of KB.
Easier to use getLongBytes and set to things like "300k"
> This will be backwards compatible with the existing settings if anyone is using them,
because the no-prefix default will still be bytes

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message