Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 23 Oct 2017 13:23:00 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13110619.1508406949000.34110.1508764980657@Atlassian.JIRA>
In-Reply-To: <JIRA.13110619.1508406949000@Atlassian.JIRA>
References: <JIRA.13110619.1508406949000@Atlassian.JIRA> <JIRA.13110619.1508406949932@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-14965) s3a input stream "normal" fadvise
 mode to be adaptive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 23 Oct 2017 13:23:09 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16215123#comment-16215123 ] 

Steve Loughran commented on HADOOP-14965:
-----------------------------------------

It won't be very adaptive: I'm thinking: sequential & random do as expected. "normal" will behave as sequential until the first backwards seek, at which point it -> random IO, on the basis that it is clearly not doing sequential access. Once in random mode, you stay there.

I'm also thinking we need to make setting policies something which can be passed all the way down from, say, spark queries down to the FS client, which is not impossible given that there is the ability to set options on a read and have them passed down to the reader factory. That just needs to (somehow) be passed down to the FS. Presumably we'd need to add an FSDataInputStreamBuilder alongside the new output stream builder, define a standard set of key values for seek policy (hints) & implement in those store clients where cost of seek is high (the objects stores). Not on my immediate TODO this there.

> s3a input stream "normal" fadvise mode to be adaptive
> -----------------------------------------------------
>
>                 Key: HADOOP-14965
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14965
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Steve Loughran
>
> HADOOP-14535 added seek optimisation to wasb, but rather than require the caller to declare sequential vs random, it works out for itself.
> # defaults to sequential, lazy seek
> # if the caller ever seeks backwards, switches to random IO.
> This means that on the use pattern of columnar stores: of go to end of file, read summary, then go to columns and work forwards, will switch to random IO after that first seek back (cost: one aborted HTTP connection)/.
> Where this should benefit the most is in downstream apps where you are working with different data sources in the same object store/running of the same app config, but have different read patterns. I'm seeing exactly this in some of my spark tests, where it's near impossible to set things up so that .gz files are read sequentially, but ORC data is read in random IO
> I propose the "normal" fadvise => adaptive, sequential==sequential always, random => random from the outset.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org