flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Bihani <jagadish.bih...@pubmatic.com>
Subject Recommendation of parameters for better performance with File Channel
Date Wed, 12 Dec 2012 10:05:47 GMT
Hi

I am able to write maximum 1.5 MB/sec data to HDFS (without compression)
using File Channel. Are there any recommendations to improve the 
performance?
Has anybody achieved around 10 MB/sec with file channel ? If yes please 
share the
configuration like (Hardware used, RAM allocated and batch sizes of 
source,sink and channels).

Following are the configuration details :
========================

I am using a machine with reasonable hardware configuration:
Quadcore 2.00 GHz processors and 4 GB RAM.

Command line options passed to flume agent :
-DJAVA_OPTS="-Xms1g -Xmx4g -Dcom.sun.management.jmxremote 
-XX:MaxDirectMemorySize=2g"

Agent Configuration:
=============
agent.sources = avro-collection-source spooler
agent.channels = fileChannel
agent.sinks = hdfsSink fileSink

# For each one of the sources, the type is defined

agent.sources.spooler.type = spooldir
agent.sources.spooler.spoolDir =/root/test_data
agent.sources.spooler.batchSize = 1000
agent.sources.spooler.channels = fileChannel

# Each sink's type must be defined
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.hdfs.path=hdfs://mltest2001/flume/release3Test

agent.sinks.hdfsSink.hdfs.fileType =DataStream
agent.sinks.hdfsSink.hdfs.rollSize=0
agent.sinks.hdfsSink.hdfs.rollCount=0
agent.sinks.hdfsSink.hdfs.batchSize=1000
agent.sinks.hdfsSink.hdfs.rollInterval=60

agent.sinks.hdfsSink.channel= fileChannel

agent.channels.fileChannel.type=file
agent.channels.fileChannel.dataDirs=/root/flume_channel/dataDir13
agent.channels.fileChannel.checkpointDir=/root/flume_channel/checkpointDir13

Regards,
Jagadish

Mime
View raw message