Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Sat, 9 Apr 2016 09:31:25 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12957431.1460193588000.183557.1460194285495@Atlassian.JIRA>
In-Reply-To: <JIRA.12957431.1460193588000@Atlassian.JIRA>
References: <JIRA.12957431.1460193588000@Atlassian.JIRA>
 <JIRA.12957431.1460193588270@arcas>
Subject: [jira] [Updated] (HADOOP-13009) add option for lazy open() on s3a
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HADOOP-13009?page=3Dcom.atlass=
ian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-13009:
------------------------------------
    Description:=20
After lazy-seek, I want to add a =E2=80=94very much non-default =E2=80=94la=
zy-open option.

If you look at a trace of what goes on with object store access, there's us=
ually a GET at offset 0 (the {{open()}} command, followed by a {{seek()}}.=
=20

If there was a lazy option option, then {{open()}} would set up the instanc=
e for reading, but not actually talk to the object store =E2=80=94it'd be t=
he first seek or read which would hit the service. You'd eliminate one HTTP=
 operation from a read sequence, for a faster startup time, especially long=
-haul.

That's a big break in the normal assumption: if a file isn't there, {{open(=
)}} fails, so it'd only work with apps which did open+read, open+seek, or o=
pened+positioned readable action back to back. By making it an option peopl=
e can experiment to see what happens =E2=80=94though full testing would nee=
d to do some fault injection on the first seek/read to see how code handled=
 late failure.

  was:
After lazy-seekI want to add a =E2=80=94very much non-default =E2=80=94lazy=
-open option.

If you look at a trace of what goes on with object store access, there's us=
ually a GET at offset 0 (the {{open()}} command, followed by a {{seek()}}.=
=20

If there was a lazy option option, then {{open()}} would set up the instanc=
e for reading, but not actually talk to the object store =E2=80=94it'd be t=
he first seek or read which would hit the service. You'd eliminate one HTTP=
 operation from a read sequence, for a faster startup time, especially long=
-haul.

That's a big break in the normal assumption: if a file isn't there, {{open(=
)}} fails, so it'd only work with apps which did open+read, open+seek, or o=
pened+positioned readable action back to back. By making it an option peopl=
e can experiment to see what happens =E2=80=94though full testing would nee=
d to do some fault injection on the first seek/read to see how code handled=
 late failure.


> add option for lazy open() on s3a
> ---------------------------------
>
>                 Key: HADOOP-13009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13009
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>
> After lazy-seek, I want to add a =E2=80=94very much non-default =E2=80=94=
lazy-open option.
> If you look at a trace of what goes on with object store access, there's =
usually a GET at offset 0 (the {{open()}} command, followed by a {{seek()}}=
.=20
> If there was a lazy option option, then {{open()}} would set up the insta=
nce for reading, but not actually talk to the object store =E2=80=94it'd be=
 the first seek or read which would hit the service. You'd eliminate one HT=
TP operation from a read sequence, for a faster startup time, especially lo=
ng-haul.
> That's a big break in the normal assumption: if a file isn't there, {{ope=
n()}} fails, so it'd only work with apps which did open+read, open+seek, or=
 opened+positioned readable action back to back. By making it an option peo=
ple can experiment to see what happens =E2=80=94though full testing would n=
eed to do some fault injection on the first seek/read to see how code handl=
ed late failure.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)