have you tried increasing max spout pending parameter for the spout?
builder.setSpout("kafka",
new KafkaSpout(spoutConfig),
TOPOLOGY_NUM_TASKS_KAFKA_SPOUT)
.setNumTasks(TOPOLOGY_NUM_TASKS_KAFKA_SPOUT)
//the maximum parallelism you can have on a KafkaSpout is the
number of partitions
.setMaxSpoutPending(*TOPOLOGY_MAX_SPOUT_PENDING*);
----------
Andrey Yegorov
On Tue, Feb 3, 2015 at 4:03 AM, clay teahouse <clayteahouse@gmail.com>
wrote:
> Hi all,
>
> In my topology, kafka spout is responsible for over 85% of the latency. I
> have tried different spout max pending and played with the buffer size and
> fetch size, still no luck. Any hint on how to optimize the spout? The issue
> doesn't seem to be with the kafka side, as I see high throughput with the
> simple kafka consumer.
>
> thank you for your feedback
> Clay
>
>
|