flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: Configure and start a Flume sink + agent from java.
Date Tue, 20 May 2014 14:55:58 GMT
Have you looked at some of the test classes?
That may be a good way to see how you can accomplish this with straight
java.

https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java


On Tue, May 20, 2014 at 6:46 AM, Jay Vyas <jayunit100@gmail.com> wrote:

> Hi flume !
>
> I'd like to implement a simple flume sink:
>
> agent.channels.memory-channel.type = memory
> agent.channels.memory-channel.capacity = 2000
> agent.channels.memory-channel.transactionCapacity = 100
> agent.sources.tail-source.type = exec
> agent.sources.tail-source.command = tail -F /tmp/flume-smoke
> agent.sources.tail-source.channels = memory-channel
>
> agent.sinks.log-sink.channel = memory-channel
> agent.sinks.log-sink.type = logger
>
> # Define a sink that outputs to the DFS
> agent.sinks.hdfs-sink.channel = memory-channel
> agent.sinks.hdfs-sink.type = hdfs
> agent.sinks.hdfs-sink.hdfs.path = myfs:///tmp/flume-test
> agent.sinks.hdfs-sink.hdfs.fileType = DataStream
>
>
> # activate the channels/sinks/sources
> agent.channels = memory-channel
> agent.sources = tail-source
> agent.sinks = log-sink hdfs-sink
>
>
> But id like to do this dynamically, i.e. without having to package any
> static text files.
>
> Is there a simple way to
>
> 1) Create the above sink definition and
> 2) To start something like "timeout 60 flume-ng agent --conf ./flume/conf/
> -f ./flume/conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent"
>
> All in Java, ?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>

Mime
View raw message