spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mario Briggs (JIRA)" <>
Subject [jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.9 Consumer API
Date Tue, 29 Dec 2015 18:12:49 GMT


Mario Briggs commented on SPARK-12177:

with regards to review comment 'C' (2 comments above), the kafka folks have clarified that
the timeout parameter on the poll() method is the time to spend waiting till the client side
gets the data (from the server) and not time to spend waiting for data to become available
on the server. This means that one might have to call poll() more than once, till we get to
the consumerRecord of 'untilOffset' 

> Update KafkaDStreams to new Kafka 0.9 Consumer API
> --------------------------------------------------
>                 Key: SPARK-12177
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.6.0
>            Reporter: Nikita Tarasenko
>              Labels: consumer, kafka
> Kafka 0.9 already released and it introduce new consumer API that not compatible with
old one. So, I added new consumer api. I made separate classes in package org.apache.spark.streaming.kafka.v09
with changed API. I didn't remove old classes for more backward compatibility. User will not
need to change his old spark applications when he uprgade to new Spark version.
> Please rewiew my changes

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message