incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Update of "KafkaProposal" by junrao
Date Tue, 28 Jun 2011 17:00:02 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "KafkaProposal" page has been changed by junrao:
http://wiki.apache.org/incubator/KafkaProposal?action=diff&rev1=8&rev2=9

  == Background ==
  Kafka was developed at LinkedIn to process the large amounts of events generated by that
company's website and provide a common repository for many types of consumers to access and
process those events. Kafka has been used in production at LinkedIn scale to handle dozens
of types of events including page views, searches and social network activity. Kafka clusters
at LinkedIn currently process more than two billion events per day.
  
- Kafka fills the gap between messaging systems such as Apache ActiveMQ, which provides low
latency message delivery but don't focus on throughput, and log processing systems such as
Scribe and Flume, which do not provide adequate latency for our diverse set of consumers.
 Kafka can also be inserted into traditional log-processing systems, acting as an intermediate
step before further processing. Kafka focuses relentlessly on performance and throughput by
not introspecting into message content, nor indexing them on the broker.  We also achieve
high performance by depending on Java's sendFile/transferTo capabilities to minimize intermediate
buffer copies and relying on the OS's pagecache to efficiently serve up message contents to
consumers. Kafka is also designed to be scalable and it depends on Apache ZooKeeper for coordination
amongst its producers, brokers and consumers.
+ Kafka fills the gap between messaging systems such as Apache ActiveMQ, which provide low
latency message delivery but don't focus on throughput, and log processing systems such as
Scribe and Flume, which do not provide adequate latency for our diverse set of consumers.
 Kafka can also be inserted into traditional log-processing systems, acting as an intermediate
step before further processing. Kafka focuses relentlessly on performance and throughput by
not introspecting into message content, nor indexing them on the broker.  We also achieve
high performance by depending on Java's sendFile/transferTo capabilities to minimize intermediate
buffer copies and relying on the OS's pagecache to efficiently serve up message contents to
consumers. Kafka is also designed to be scalable and it depends on Apache ZooKeeper for coordination
amongst its producers, brokers and consumers.
  
  Kafka is written in Scala. It was developed internally at LinkedIn to meet our particular
use cases, but will be useful to many organizations facing a similar need to reliably process
large amounts of streaming data.  Therefore, we would like to share it the ASF and begin developing
a community of developers and users within Apache.
  

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message