flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject Re: Syslog TCP performances issue with filechannel
Date Wed, 04 Mar 2015 19:08:50 GMT
You should probably increase the batch size, since each batch causes an fsync which slows things
down.




Thanks, Hari

On Wed, Mar 4, 2015 at 6:28 AM, Smaine Kahlouch
<smaine.kahlouch@smartjog.com> wrote:

> Hi all,
> I'm currently doing benchmarks on flume.
> We're planning to use flume with syslogtcp as source and filechannel in 
> order to have avoid data loss.
> The performances are quiet good when a memorychannel is used :
> ~*100 000events/sec* (event size = 600bytes)
> But as soon as i switch to filechannel the performances drop drammatically:
> ~*300events/sec*
> Despite this poor result, the behaviour is really strange because i have 
> a heavy disk usage (all the disks), near 100%.
> I use a tool provided by syslog-ng in order to generate syslog logs : 
> loggen 
> <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/loggen.1.html>
> ex : loggen -i -I 3000000 --size 600 --active-connections 200 
> myflumehost 20515
> Flume version : 1.5.2
> Operating System : Centos 6
> Please find my flume configuration enclosed. The filechannel is spread 
> over 5 disks in order to improve performance.
> Could you please help me to configure properly syslogtcp source with 
> filechannel ?
> Regards,
> -- 
> Smaine Kahlouch - Engineer, Research & Engineering
> Arkena | T: +33 1 5868 6196
> 27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
> arkena.com
Mime
View raw message