spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jerryshao <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-6051][Streaming] Add ZooKeeper offest p...
Date Tue, 10 Mar 2015 02:28:45 GMT
Github user jerryshao commented on the pull request:

    https://github.com/apache/spark/pull/4805#issuecomment-77985091
  
    Yes, that's my original purpose of PR, user can monitor and reuse these offsets, also
we can offer user the similar functionality as old Kafka stream to get data from previous
offset (if not out-of-range and no checkpointing). My concern is that this might break the
exact-once semantics. 
    
    So basically let me rephrase what I think about it:
    
    1. If checkpointing is not enabled, do we need to offer user the ability to get data from
previous offset or always from earliest or latest offsets? If we want to offer the similar
functionality as previous Kafka stream, then I think putting offsets to Kafka is required
and keep basic no data loss is necessary.
    2. If checkpointing is enabled, these offsets is only used for monitoring, otherwise we
will break the exact-once semantics when using these offsets again. So if checkpointing is
enabled, we can update the offset at any time because we don't use these offsets to make sure
no data loss.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message