flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Bihani <jagadish.bih...@pubmatic.com>
Subject Re: Recommendation of parameters for better performance with File Channel
Date Wed, 12 Dec 2012 10:08:13 GMT

I am using latest release of flume. (Flume 1.3.0)  and hadoop 1.0.3.

On 12/12/2012 03:35 PM, Jagadish Bihani wrote:
> Hi
>
> I am able to write maximum 1.5 MB/sec data to HDFS (without compression)
> using File Channel. Are there any recommendations to improve the 
> performance?
> Has anybody achieved around 10 MB/sec with file channel ? If yes 
> please share the
> configuration like (Hardware used, RAM allocated and batch sizes of 
> source,sink and channels).
>
> Following are the configuration details :
> ========================
>
> I am using a machine with reasonable hardware configuration:
> Quadcore 2.00 GHz processors and 4 GB RAM.
>
> Command line options passed to flume agent :
> -DJAVA_OPTS="-Xms1g -Xmx4g -Dcom.sun.management.jmxremote 
> -XX:MaxDirectMemorySize=2g"
>
> Agent Configuration:
> =============
> agent.sources = avro-collection-source spooler
> agent.channels = fileChannel
> agent.sinks = hdfsSink fileSink
>
> # For each one of the sources, the type is defined
>
> agent.sources.spooler.type = spooldir
> agent.sources.spooler.spoolDir =/root/test_data
> agent.sources.spooler.batchSize = 1000
> agent.sources.spooler.channels = fileChannel
>
> # Each sink's type must be defined
> agent.sinks.hdfsSink.type = hdfs
> agent.sinks.hdfsSink.hdfs.path=hdfs://mltest2001/flume/release3Test
>
> agent.sinks.hdfsSink.hdfs.fileType =DataStream
> agent.sinks.hdfsSink.hdfs.rollSize=0
> agent.sinks.hdfsSink.hdfs.rollCount=0
> agent.sinks.hdfsSink.hdfs.batchSize=1000
> agent.sinks.hdfsSink.hdfs.rollInterval=60
>
> agent.sinks.hdfsSink.channel= fileChannel
>
> agent.channels.fileChannel.type=file
> agent.channels.fileChannel.dataDirs=/root/flume_channel/dataDir13
> agent.channels.fileChannel.checkpointDir=/root/flume_channel/checkpointDir13 
>
>
> Regards,
> Jagadish


Mime
View raw message