beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xu Mingmin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-1775) fix issue of start_from_previous_offset in KafkaIO
Date Tue, 21 Mar 2017 15:55:41 GMT
Xu Mingmin created BEAM-1775:
--------------------------------

             Summary: fix issue of start_from_previous_offset in KafkaIO
                 Key: BEAM-1775
                 URL: https://issues.apache.org/jira/browse/BEAM-1775
             Project: Beam
          Issue Type: Improvement
          Components: sdk-java-extensions
            Reporter: Xu Mingmin
            Assignee: Xu Mingmin


Jins George jins.george@aeris.net via aermail.onmicrosoft.com 
	
5:50 PM (15 hours ago)
	
to user
Hello,

I am writing a Beam pipeline(streaming) with Flink runner to consume data from Kafka and apply
some transformations and persist to Hbase.

If I restart the application ( due to failure/manual restart), consumer does not resume from
the offset where it was prior to restart. It always resume from the latest offset.

If I enable Flink checkpionting with hdfs state back-end, system appears to be resuming from
the earliest offset

Is there a recommended way to resume from the offset where it was stopped ?

Thanks,
Jins George



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message