flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject Re: Perfromacnce issue in file channel
Date Mon, 10 Nov 2014 19:34:57 GMT
Sending one event by one event is likely to cause performance issues. In addition, I have no
idea of the performance on Windows. Btw, 100-200 request per second is really orders of magnitude
lower than what File Channel can handle.


Thanks,
Hari

On Sun, Nov 9, 2014 at 7:49 AM, mahendran m <mahendranec@hotmail.com>
wrote:

> Hi All ,I have tested the file channel and file channel performance. below the code snippet
to generate 1000 event per sec 
> while (watch.Elapsed.TotalSeconds < sec){	if (counter <= 1000)	{		uuid = uuid +
1;
> 		var text = HttpUtility.UrlEncode("HI , How are you", Encoding.UTF8);
> 		var flumeEvent = new ThriftFlumeEvent();		flumeEvent.Headers = new Dictionary<string,
string>() { { "Host", "http://www.bing.com/" } };		flumeEvent.Body = Encoding.ASCII.GetBytes(text);
	Client.append(flumeEvent);		counter += 1;	}}
> i succeed in sending the 1000 event per sec when using memory channel , but in case if
i used file channel  i can  only reach upto 40 to 50 events per second
> config for memory channel# Use memory channela1.channels.c1.type = memorya1.channels.c1.capacity
= 1000000a1.channels.c1.transactionCapacity = 1000a1.channels.c1.byteCapacityBufferPercentage
= 10a1.channels.c1.byteCapacity = 5368709120
> config for file channel
> #using the file channela1.channels.c1.type = filea1.channels.c1.capacity = 500000a1.channels.c1.transactionCapacity
= 10000a1.channels.c1.useDualCheckpoints = truea1.channels.c1.backupCheckpointDir = F:/.checkPoint
> Can anyone explain why file channel is this much channel ? . if i used file channel in
server for directly logging each request is it better option as my server receive 100 to 200
request per second
> Regards,Mahendran
>  		 	   		   		 	   		  
Mime
View raw message