hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ben Roling (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15625) S3A input stream to use etags to detect changed source files
Date Fri, 01 Mar 2019 17:47:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781915#comment-16781915
] 

Ben Roling commented on HADOOP-15625:
-------------------------------------

On another tangent, something I noticed is that it doesn't look like [S3 Select|https://aws.amazon.com/blogs/aws/s3-glacier-select/]
supports ETag or versionId qualification (either client or server side).  That doesn't seem
directly applicable to this particular JIRA since SelectInputStream already doesn't support
seek() backwards or any other reason for re-open, but it would mean something for HADOOP-16085.
 S3Guard will not be able to avoid read-after-overwrite inconsistency with S3 Select.  I think
that will just need to be a highlighted caveat in the documentation.  It's possible that eventually
S3 Select will support such qualifications and the implementation could be updated at that
point.

Steve - do you have any comments as to how the patch looks now otherwise?  Are there any items
I should/could still be following up on before this can be merged?

> S3A input stream to use etags to detect changed source files
> ------------------------------------------------------------
>
>                 Key: HADOOP-15625
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15625
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2.0
>            Reporter: Brahma Reddy Battula
>            Assignee: Brahma Reddy Battula
>            Priority: Major
>         Attachments: HADOOP--15625-006.patch, HADOOP-15625-001.patch, HADOOP-15625-002.patch,
HADOOP-15625-003.patch, HADOOP-15625-004.patch, HADOOP-15625-005.patch, HADOOP-15625-006.patch,
HADOOP-15625-007.patch, HADOOP-15625-008.patch, HADOOP-15625-009.patch, HADOOP-15625-010.patch,
HADOOP-15625-011.patch, HADOOP-15625-012.patch
>
>
> S3A input stream doesn't handle changing source files any better than the other cloud
store connectors. Specifically: it doesn't noticed it has changed, caches the length from
startup, and whenever a seek triggers a new GET, you may get one of: old data, new data, and
even perhaps go from new data to old data due to eventual consistency.
> We can't do anything to stop this, but we could detect changes by
> # caching the etag of the first HEAD/GET (we don't get that HEAD on open with S3Guard,
BTW)
> # on future GET requests, verify the etag of the response
> # raise an IOE if the remote file changed during the read.
> It's a more dramatic failure, but it stops changes silently corrupting things.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message