spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Quentin Ambard (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-24720) kafka transaction creates Non-consecutive Offsets (due to transaction offset) making streaming fail when failOnDataLoss=true
Date Mon, 02 Jul 2018 17:56:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-24720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Quentin Ambard updated SPARK-24720:
-----------------------------------
    External issue ID:   (was: SPARK-17147)

> kafka transaction creates Non-consecutive Offsets (due to transaction offset) making
streaming fail when failOnDataLoss=true
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-24720
>                 URL: https://issues.apache.org/jira/browse/SPARK-24720
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.3.1
>            Reporter: Quentin Ambard
>            Priority: Major
>
> When kafka transactions are used, sending 1 message to kafka will result to 1 offset
for the data + 1 offset to mark the transaction.
> When kafka read a topic with non-consecutive offset, it leads to a failure. SPARK-17147
fixed this issue for compacted topics.
> However, SPARK-17147 doesn't fix this issue: if 1 message + 1 transaction commit are in
a partition, spark will try to read offsets  [0 2[. offset 0 (containing the message) will
be read, but offset 1 won't return a value and buffer.hasNext() will be false even after
a poll since no data are present for offset 1 (it's the transaction commit)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message