crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-609) KafkaSource could skip data with slow broker response
Date Thu, 30 Jun 2016 15:00:13 GMT


Micah Whitacre updated CRUNCH-609:
    Attachment: CRUNCH-609c.patch

One more patch revision that should be good.

> KafkaSource could skip data with slow broker response
> -----------------------------------------------------
>                 Key: CRUNCH-609
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: IO
>            Reporter: Micah Whitacre
>            Assignee: Micah Whitacre
>         Attachments: CRUNCH-609.patch, CRUNCH-609b.patch, CRUNCH-609c.patch
> In the case of the consumer.poll timing out, the KafkaRecordReader could return with
no values in the records iterator which then causes the KafkaRecordReader to exit prematurely[1].
 To fix this we need to probably better track the start/end offset and that progress vs the
null value.
> Additionally if we wanted to be smarter we could do some "backoff" for the timeout or
just have consumer specify a larger value.
> [1] -

This message was sent by Atlassian JIRA

View raw message