flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Flume scalability & performance
Date Thu, 19 Apr 2012 14:44:01 GMT

On Thu, Apr 19, 2012 at 10:04 AM, M. Karthikeyan
<m.karthikeyan@ericsson.com> wrote:
> Im trying to choose between Flume and JMS for data collection framework in
> our multi-node network.
> I have the following questions:
> 1) From a scalability point of view, how does Flume compare with JMS? Are
> there any numbers that can be referred to
> 2) My typical payload for a single message is 2 KB. I expect traffic of
> approx. 50 million messages/day. The messages are usually one sender one
> receiver type. I require a reasonable level of reliability (atleast the
> store-and-forward mode in Flume & durable/persistent messages in JMS). With
> these considerations, which will give better performance: Flume or JMS?

All of this is extremely dependent on the implementation of JMS you
use. JMS is a specification, there are many implementations. Looking
at your numbers and assumption all the messages come in 8 hours
(representing peak load) that is about 4MB/second.

Both Flume and most JMS implementations should be able to handle this
throughput. The advantage of Flume is really configuration. Purchasing
and configuring a JMS server and then writing code to interact with
the JMS Server is, IMHO, going to be less work than installing and
configuring Flume.


Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

View raw message