beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <>
Subject [jira] [Updated] (BEAM-66) Unbounded source for directory/file scanning
Date Thu, 25 Feb 2016 16:31:18 GMT


Daniel Halperin updated BEAM-66:
    Assignee:     (was: Daniel Halperin)

> Unbounded source for directory/file scanning
> --------------------------------------------
>                 Key: BEAM-66
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Daniel Halperin
>            Priority: Minor
> (Blocked on SplittableDoFn,
> This is a streaming glob watcher, for instance a directory tailer -- poll a directory
read the data from new files as they show up. To get this right in a streaming pipeline requires
careful management of timestamps and watermarks -- e.g., a filename or other metadata should
reflect a timestamp earlier than any records in it, so that we are able to provide accurate
lower bounds on output timestamp.

This message was sent by Atlassian JIRA

View raw message