kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tauzell, Dave" <Dave.Tauz...@surescripts.com>
Subject RE: Handling 2 to 3 Million Events before Kafka
Date Wed, 21 Jun 2017 18:23:15 GMT
I’m not really familiar with Netty so I won’t be of much help.   Maybe try posting on a
Netty forum to see what they think?
-Dave

From: SenthilKumar K [mailto:senthilec566@gmail.com]
Sent: Wednesday, June 21, 2017 10:28 AM
To: Tauzell, Dave
Cc: users@kafka.apache.org; senthilec566@apache.org; dev@kafka.apache.org
Subject: Re: Handling 2 to 3 Million Events before Kafka

So netty would work for this case ?  I do have netty server and seems to be i'm not getting
the expected results .. here is the git https://github.com/senthilec566/netty4-server , is
this right implementation ?

Cheers,
Senthil

On Wed, Jun 21, 2017 at 7:45 PM, Tauzell, Dave <Dave.Tauzell@surescripts.com<mailto:Dave.Tauzell@surescripts.com>>
wrote:
I see.

1.       You don’t want the 100k machines sending directly to kafka.

2.       You can only have a small number of web servers

People certainly have web-servers handling over 100k concurrent connections.  See this for
some examples:  https://github.com/smallnest/C1000K-Servers .

It seems possible with the right sort of kafka producer tuning.

-Dave

From: SenthilKumar K [mailto:senthilec566@gmail.com<mailto:senthilec566@gmail.com>]
Sent: Wednesday, June 21, 2017 8:55 AM
To: Tauzell, Dave
Cc: users@kafka.apache.org<mailto:users@kafka.apache.org>; senthilec566@apache.org<mailto:senthilec566@apache.org>;
dev@kafka.apache.org<mailto:dev@kafka.apache.org>; Senthil kumar
Subject: Re: Handling 2 to 3 Million Events before Kafka

Thanks Jeyhun. Yes http server would be problematic here w.r.t network , memory ..

Hi Dave ,  The problem is not with Kafka , it's all about how do you handle huge data before
kafka.  I did a simple test with 5 node Kafka Cluster which gives good result ( ~950 MB/s
) ..So Kafka side i dont see a scaling issue ...

All we are trying is before kafka how do we handle messages from different servers ...  Webservers
can send fast to kafka but still i can handle only 50k events per second which is less for
my use case.. also i can't deploy 20 webservers to handle this load. I'm looking for an option
what could be the best candidate before kafka , it should be super fast in getting all and
send it to kafka producer ..


--Senthil

On Wed, Jun 21, 2017 at 6:53 PM, Tauzell, Dave <Dave.Tauzell@surescripts.com<mailto:Dave.Tauzell@surescripts.com>>
wrote:
What are your configurations?

- production
- brokers
- consumers

Is the problem that web servers cannot send to Kafka fast enough or your consumers cannot
process messages off of kafka fast enough?
What is the average size of these messages?

-Dave

-----Original Message-----
From: SenthilKumar K [mailto:senthilec566@gmail.com<mailto:senthilec566@gmail.com>]
Sent: Wednesday, June 21, 2017 7:58 AM
To: users@kafka.apache.org<mailto:users@kafka.apache.org>
Cc: senthilec566@apache.org<mailto:senthilec566@apache.org>; Senthil kumar; dev@kafka.apache.org<mailto:dev@kafka.apache.org>
Subject: Handling 2 to 3 Million Events before Kafka

Hi Team ,   Sorry if this question is irrelevant to Kafka Group ...

I have been trying to solve problem of handling 5 GB/sec ingestion. Kafka is really good candidate
for us to handle this ingestion rate ..


100K machines ----> { Http Server (Jetty/Netty) } --> Kafka Cluster..

I see the problem in Http Server where it can't handle beyond 50K events per instance .. 
I'm thinking some other solution would be right choice before Kafka ..

Anyone worked on similar use case and similar load ? Suggestions/Thoughts ?

--Senthil
This e-mail and any files transmitted with it are confidential, may contain sensitive information,
and are intended solely for the use of the individual or entity to whom they are addressed.
If you have received this e-mail in error, please notify the sender by reply e-mail immediately
and destroy all copies of the e-mail and any attachments.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message