kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krishna Raj <reach.krishna...@gmail.com>
Subject Requesting advice about Producer & Consumer Design
Date Wed, 19 Mar 2014 09:33:37 GMT
Hello Experts & Kafka Team,

Its existing to learn and work on using Kafka. I have been going through
lot of pages and Q&A.

We are building an infra & topology using Kafka for events processing in
our application.

We need some advice about designing the Producer and Consumer.

*Please find attached file/below picture* of our current setup that we are
thinking of.

[image: Inline image 1]

*1) Producer:*

I understand that from 0.8.1, the message balancing is done in a fashion
that the broker will choose a partition after every meta refresh(the
default of which is 10 mins)

Questions are:

a.* Is there any other mechanism other than changing meta refresh ?* (I
understand that the logic implementation using custom class is no longer
supported in 0.8.1)

b. We ultimately want the message to be evenly distributed across
partitions so that consumer's load is also evenly distributed thus paving
scalability & reduce lag and will help us scale easily as we can just add
partition with a corresponding consumer node attached to it. Is this
advised ? And to achieve this, *what is the optimized meta refresh time
without affecting performance ?*

*2) Consumer*

a.  I was in a though that SimpleConsumer has more flexibillity and
features. But after reading Neha's below JavaDoc, I am liking KafkaConsumer
features and the less need to handle at granular level. *What is the
adviced Consumer, SimpleCosumer or KafkaConsumer ?*

Neha's KafkaCosumer JavaDoc:

b. For keeping track of the Offset at each Consumer node, I am thinking to
manually control the Offset commit(to ensure that processing a message is
neither missed nor duplicated). On failure or exception, I would also log
the current Offset in a file or something before exiting so that when I
start my consumer again I can start from the Offset where I left. *Is this
a good design ?*

Thanks for the time and really appreciate the effort for making Kafka
amazing :)


  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message