spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zsxwing <>
Subject [GitHub] spark pull request #16251: [SPARK-18826][SS]Add 'newestFirst' option to File...
Date Mon, 12 Dec 2016 05:19:40 GMT
GitHub user zsxwing opened a pull request:

    [SPARK-18826][SS]Add 'newestFirst' option to FileStreamSource

    ## What changes were proposed in this pull request?
    When starting a stream with a lot of backfill and maxFilesPerTrigger, the user could often
want to start with most recent files first. This would let you keep low latency for recent
data and slowly backfill historical data.
    This PR adds a new option `newestFirst` to control this behavior. When it's true, `FileStreamSource`
will sort the files by the modified time from newest to oldest, and take the first `maxFilesPerTrigger`
files as a new batch.
    ## How was this patch tested?
    The added test.

You can merge this pull request into a Git repository by running:

    $ git pull newest-first

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #16251
commit 58a57d4004c45ff2290b95ec8c70ef95828d379b
Author: Shixiong Zhu <>
Date:   2016-12-12T05:15:10Z

    Add 'newestFirst' option to FileStreamSource


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message