spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-5718) Add native offset management for ReliableKafkaReceiver
Date Mon, 17 Oct 2016 12:08:58 GMT

     [ https://issues.apache.org/jira/browse/SPARK-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen updated SPARK-5718:
-----------------------------
    Assignee: Cody Koeninger

> Add native offset management for ReliableKafkaReceiver
> ------------------------------------------------------
>
>                 Key: SPARK-5718
>                 URL: https://issues.apache.org/jira/browse/SPARK-5718
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.3.0
>            Reporter: Saisai Shao
>            Assignee: Cody Koeninger
>             Fix For: 2.0.0
>
>
> Kafka 0.8.2 supports native offsets management instead of ZK, this will get better performance,
for now in ReliableKafkaReceiver, we rely on ZK to manage the offsets, this potentially will
be a bottleneck if the injection rate is high (once per 200ms by default), so here in order
to get better performance as well as keeping consistent with Kafka, add native offset management
for ReliableKafkaReceiver.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message