beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-3093) add an option 'FirstPollOffsetStrategy' to KafkaIO
Date Tue, 28 Nov 2017 05:53:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268168#comment-16268168
] 

Raghu Angadi edited comment on BEAM-3093 at 11/28/17 5:52 AM:
--------------------------------------------------------------

[~mingmxu], assigning this to you. Let me know {{withStartReadTime()}} does not do what you
are looking for.


was (Author: rangadi):
[~mingmxu], assigning this to you. Let me know `withStartReadTime()` does not do what you
are looking for.

> add an option 'FirstPollOffsetStrategy' to KafkaIO
> --------------------------------------------------
>
>                 Key: BEAM-3093
>                 URL: https://issues.apache.org/jira/browse/BEAM-3093
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-core
>            Reporter: Xu Mingmin
>            Assignee: Xu Mingmin
>
> This is a feature borrowed from Storm KafkaSpout.
> *What's the issue?*
> In KafkaIO, when offset is stored either in checkpoint or auto_committed, it cannot be
changed in application, to force to read from earliest/latest. --This feature is important
to reset the start offset when relaunching a job.
> *Proposed solution:*
> By borrowing the FirstPollOffsetStrategy concept, users can have more options:
> 1). *{{EARLIEST}}*: always start_from_beginning no matter of what's in checkpoint/auto_commit;
> 2). *{{LATEST}}*: always start_from_latest no matter of what's in checkpoint/auto_commit;
> 3). *{{UNCOMMITTED_EARLIEST}}*: if no offset in checkpoint/auto_commit then start_from_beginning
if, otherwise start_from_previous_offset;
> 4). *{{UNCOMMITTED_LATEST}}*: if no offset in checkpoint/auto_commit then start_from_latest,
otherwise start_from_previous_offset;
> [~rangadi], any comments?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message