beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-1190) FileBasedSource should ignore files that matched the glob but don't exist
Date Wed, 21 Dec 2016 18:44:58 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767812#comment-15767812
] 

Daniel Halperin commented on BEAM-1190:
---------------------------------------

Here are my general concerns:

* How do we distinguish this event from files getting deleted between expand and read time,
which is obviously an error.
* How do we ensure consistency? Aka, suppose I produce 3 shards of a file returned by a list
on an eventually-list-consistent filesystem, and when we try to open them at runtime the file
does not exist. How can we distinguish whether the file was deleted before expand-glob, after
expand-glob but before any reading, after reading some shards but not others, etc?
* If the filesystem (or the part of the filesystem used by the user) is not eventually consistent,
this is clearly a bad thing to do. So this would somehow have to be filesystem+glob aware.

So I would prefer a solution like giving [BEAM-60] which will give the user code they can
execute at expand-the-glob time. That's where the user can put in the "stat the file and make
sure it's there, or remove it" code, if they want such.

For your proposal:

* Will it inspect the properties of the Filesystem to ensure it is eventually consistent for
the expanded glob?
* Do we plan to add a mandatory stat at glob-expand time? That seems like the only correct
place to add this logic -- otherwise, we can't tell from files that legitimately did exist
at expand time, but were deleted in error at runtime.
* How do we plan to allow the user to opt into this behavior?

I think putting the user explicitly in the loop is the right thing to do, either through one
of the two JIRA issues I linked earlier. I am not yet persuaded that changing default behavior
is the right answer.

> FileBasedSource should ignore files that matched the glob but don't exist
> -------------------------------------------------------------------------
>
>                 Key: BEAM-1190
>                 URL: https://issues.apache.org/jira/browse/BEAM-1190
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Eugene Kirpichov
>            Assignee: Eugene Kirpichov
>
> See user issue:
> http://stackoverflow.com/questions/41251741/coping-with-eventual-consistency-of-gcs-bucket-listing
> We should, after globbing the files in FileBasedSource, individually stat every file
and remove those that don't exist, to account for the possibility that glob yielded non-existing
files due to eventual consistency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message