beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-66) Unbounded source for directory/file scanning
Date Thu, 25 Feb 2016 16:30:18 GMT
Daniel Halperin created BEAM-66:
-----------------------------------

             Summary: Unbounded source for directory/file scanning
                 Key: BEAM-66
                 URL: https://issues.apache.org/jira/browse/BEAM-66
             Project: Beam
          Issue Type: New Feature
          Components: sdk-java-core
            Reporter: Daniel Halperin
            Assignee: Davor Bonaci
            Priority: Minor


(Blocked on SplittableDoFn, https://issues.apache.org/jira/browse/BEAM-65.)

This is a streaming glob watcher, for instance a directory tailer -- poll a directory read
the data from new files as they show up. To get this right in a streaming pipeline requires
careful management of timestamps and watermarks -- e.g., a filename or other metadata should
reflect a timestamp earlier than any records in it, so that we are able to provide accurate
lower bounds on output timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message