kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: application scenerio and suggested kafka setup
Date Sun, 27 Jan 2013 23:54:21 GMT
Partition is useful for increasing the degree of parallelism of consumers
and to a certain degree, producers too. You can have multiple consumer
groups consuming the same topic/partition.

Thanks,

Jun

On Sun, Jan 27, 2013 at 1:06 PM, S Ahmed <sahmed1020@gmail.com> wrote:

> Say I create web application/service where customers signup, and they place
> some javascript on their website which will then send over http a message
> to my servers every time someone clicks on a link on their website.
>
> Each customer will send to their own custom subdomain like:
>
> customer1.example.com/api/put?linkId=1&......
>
> Say I have 100,000 customers.
>
> 1. If all events are of the same type, what are the potential means I could
> partition my topics?  Or does it not make sense to?  I'm confused as to
> what I am reading, is a given kafka topic + paritition combination ONLY
> allowed to be consumed by a single consumer group?  If so, why is that?
>  the kafka server can only handle a single thread connecting to it??
>
> 2. I will have a java servlet that will contain my producer (each front end
> server will have the same servlet that will contain a producer).  I want to
> batch every x messages.  From what I understand, my producer is something I
> will create using a singleton correct?
>
> 3. I want my consumers to by dynamic in size, so during peak hours I want
> to fire up more nodes to  keep up with traffic, is there a production
> worthy consumer daemon that I can use (or learn from) that is open sourced
> somewhere?
>
> Much appreciated!
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message