incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Update of "KafkaProposal" by junrao
Date Tue, 28 Jun 2011 16:36:12 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "KafkaProposal" page has been changed by junrao:
http://wiki.apache.org/incubator/KafkaProposal?action=diff&rev1=7&rev2=8

  == Background ==
  Kafka was developed at LinkedIn to process the large amounts of events generated by that
company's website and provide a common repository for many types of consumers to access and
process those events. Kafka has been used in production at LinkedIn scale to handle dozens
of types of events including page views, searches and social network activity. Kafka clusters
at LinkedIn currently process more than two billion events per day.
  
- Kafka fills the gap between messaging systems such as Apache ActiveMQ, which can provide
high-volume messaging systems but lack persistence of those messages, and log processing systems
such as Scribe and Flume, which do not provide adequate latency for our diverse set of consumers.
 Kafka can also be inserted into traditional log-processing systems, acting as an intermediate
step before further processing. Kafka focuses relentlessly on performance and throughput by
not introspecting into message content, nor indexing them on the broker.  We also achieve
high performance by depending on Java's sendFile/transferTo capabilities to minimize intermediate
buffer copies and relying on the OS's pagecache to efficiently serve up message contents to
consumers.
+ Kafka fills the gap between messaging systems such as Apache ActiveMQ, which provides low
latency message delivery but don't focus on throughput, and log processing systems such as
Scribe and Flume, which do not provide adequate latency for our diverse set of consumers.
 Kafka can also be inserted into traditional log-processing systems, acting as an intermediate
step before further processing. Kafka focuses relentlessly on performance and throughput by
not introspecting into message content, nor indexing them on the broker.  We also achieve
high performance by depending on Java's sendFile/transferTo capabilities to minimize intermediate
buffer copies and relying on the OS's pagecache to efficiently serve up message contents to
consumers. Kafka is also designed to be scalable and it depends on Apache ZooKeeper for coordination
amongst its producers, brokers and consumers.
  
- Kafka is written in Scala and depends on Apache ZooKeeper for coordination amongst its producers,
brokers and consumers.
- 
- Kafka was developed internally at LinkedIn to meet our particular use cases, but will be
useful to many organizations facing a similar need to reliably process large amounts of streaming
data.  Therefore, we would like to share it the ASF and begin developing a community of developers
and users within Apache.
+ Kafka is written in Scala. It was developed internally at LinkedIn to meet our particular
use cases, but will be useful to many organizations facing a similar need to reliably process
large amounts of streaming data.  Therefore, we would like to share it the ASF and begin developing
a community of developers and users within Apache.
  
  == Rationale ==
  Many organizations can benefit from a reliable stream processing system such as Kafka. 
While our use case of processing events from a very large website like LinkedIn has driven
the design of Kafka, its uses are varied and we expect many new use cases to emerge.  Kafka
provides a natural bridge between near real-time event processing and offline batch processing
and will appeal to many users.
@@ -74, +72 @@

  
  == Required Resources ==
  === Mailing Lists ===
-  * kafka-private for private PMC discussions (with moderated subscriptions)   
+  * kafka-private for private PMC discussions (with moderated subscriptions)
   * kafka-dev
   * kafka-commits
   * kafka-user

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message