flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanzhi.liu" <404307...@qq.com>
Subject About flume Host Interceptor and end-to-end source
Date Mon, 06 Aug 2012 09:07:14 GMT
hello people:
   I am using the exec source 、avro source、avro sink and hdfs sink to configuration two
computer.
   This is mine configuration:
     A computer is:
        agent_foo.sources = tailsource-1
       agent_foo.channels = memoryChannel-1
       agent_foo.sinks = Sink-1
        agent_foo.sources.tailsource-1.type = exec
       agent_foo.sources.tailsource-1.command = tail -F /home/hadoop/hello.txt
       agent_foo.sources.tailsource-1.channels = memoryChannel-1
       agent_foo.sources.tailsource-1.batchSize = 1
        agent_foo.channels.memoryChannel-1.type = memory
       agent_foo.channels.memoryChannel-1.capacity = 1000
        agent_foo.sinks.Sink-1.type=avro
       agent_foo.sinks.Sink-1.channel=memoryChannel-1
       agent_foo.sinks.Sink-1.hostname=221.130.18.211
       agent_foo.sinks.Sink-1.port=4545
       agent_foo.sinks.Sink-1.batch-size=1
     And the other computer is configuration:
        agent_fo.sources = tailsource-1
       agent_fo.channels = memoryChannel-1
       agent_fo.sinks = hdfsSink-1
        agent_fo.sources.tailsource-1.type = avro
       agent_fo.sources.tailsource-1.channels=memoryChannel-1
       agent_fo.sources.tailsource-1.bind=221.130.18.211
       agent_fo.sources.tailsource-1.port=4545
        agent_fo.channels.memoryChannel-1.type = memory
       agent_fo.channels.memoryChannel-1.capacity = 1000
        agent_fo.sinks.hdfsSink-1.type = hdfs
       agent_fo.sinks.hdfsSink-1.channel = memoryChannel-1
       agent_fo.sinks.hdfsSink-1.hdfs.path = hdfs://CMN-NJ-2-579:9000/user/hadoop/flume
       agent_fo.sinks.hdfsSink-1.hdfs.rollInterval=600
       agent_fo.sinks.hdfsSink-1.hdfs.rollCount=0
       agent_fo.sinks.hdfsSink-1.hdfs.rollSize = 1048576
       agent_fo.sinks.hdfsSink-1.hdfs.fileType=CompressedStream
       agent_fo.sinks.hdfsSink-1.hdfs.codeC=gzip
       agent_fo.sinks.hdfsSink-1.hdfs.writeFormat=Text
       agent_fo.sinks.hdfsSink-1.hdfs.batchSize=10
       agent_fo.sinks.hdfsSink-1.serializer=avro_event
    My problem is that hello.txt don't delete old data .About the problem I should do what
could solve.
    I want to know that Flume 1.2.0 User Guide's end-to-end is how to configuration.
    The second problem is about Host Interceptor.What configuration could get the host-name
and IP-address.
    I hope to get your helping.
 Thank you very much!
 My Name:
 Yanzhi Liu
Mime
View raw message