flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bojan Kostić <blood9ra...@gmail.com>
Subject Can Flume handle +100k events per seccond?
Date Mon, 14 Oct 2013 12:00:42 GMT
Hi, this is my first post here. But i play with flume for some time now.
My question is how well flume scale?
Can Flume ingest +100k events per seccond? Has anyone tried something like
this?

I created simple test and results are really slow.
I wrote simple app with rpc client with fallback using flume sdk which is
reading dummy log file.
In the end i have two flume agents which are writing to hdfs.
rollInterval = 60
And in hdfs i get files with ~12MB.

Do i need to use some complex topology with 3 tier?
How many flume agents should write to hdfs?

Best regards.

Mime
View raw message