spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-28738) Add ability to include metadata in CanCommitOffsets API
Date Fri, 16 Aug 2019 04:47:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-28738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908704#comment-16908704
] 

Hyukjin Kwon commented on SPARK-28738:
--------------------------------------

[~jrciii], please clarify the usecase, examples and reproducer to show why this API should
be exposed.

> Add ability to include metadata in CanCommitOffsets API
> -------------------------------------------------------
>
>                 Key: SPARK-28738
>                 URL: https://issues.apache.org/jira/browse/SPARK-28738
>             Project: Spark
>          Issue Type: New Feature
>          Components: DStreams
>    Affects Versions: 2.4.4
>            Reporter: Joseph Cooper
>            Priority: Major
>
> It is possible to commit metadata with an offset to Kafka. Currently, the CanCommitOffsets
API does not expose this functionality. See [https://github.com/apache/spark/blob/017919b636fd3ce43ccf5ec57f1c1842aa2130db/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala#L300]
>  
> We could add a commitSync function which commits an offset right away and accepts metadata.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message