kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <david.frank...@bt.com>
Subject RE: KafkaConnect SinkTask::put
Date Fri, 06 Jan 2017 10:19:13 GMT
Hi Shikhar - thankyou very much for that :)

Best wishes,
David

-----Original Message-----
From: Shikhar Bhushan [mailto:shikhar@confluent.io] 
Sent: 05 January 2017 18:12
To: dev@kafka.apache.org
Subject: Re: KafkaConnect SinkTask::put

Hi David,

You can override the underlying consumer's `max.poll.records` setting for this. E.g.
    consumer.max.poll.records=500

Best,

Shikhar

On Thu, Jan 5, 2017 at 3:59 AM <david.franklin@bt.com> wrote:

> Is there any way of limiting the number of events that are passed into 
> the call to the put(Collection<SinkRecord>) method?
>
> I'm writing a set of events to Kafka via a source Connector/Task and 
> reading these from a sink Connector/Task.
> If I generate of the order of 10k events the number of SinkRecords 
> passed to the put method starts off very low but quickly rises in 
> large increments such that 9k events are passed to a later invocation of the put method.
>
> Furthermore, processing a large number of events in a single call (I'm 
> writing to Elasticsearch) appears to cause the source task poll() 
> method to timeout, raising a CommitFailedException which, 
> incidentally, I can't see how to catch.
>
> Thanks for any help you can provide,
> David
>
Mime
View raw message