flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cochran, David M (Contractor)" <David.Coch...@bsee.gov>
Subject conf help needed
Date Thu, 23 Aug 2012 14:08:54 GMT
Let me see if I can explain what I'd like to accomplish, and yes I"m certain I'm just not grasping
something here and making a mess of it all, and mayhaps ya'll can point me straight...

Simple.. grab this log file xxx.log from host foo and store it in a on the sink server as

Where the xxx.log and hostname are dynamic so multiple hosts/logs can be templated.  I guess
that makes sense... afterall that's what this whole project is about...  

My troubles are coming from sorting the conf:

here's where I've started... a combination of a bunch of conf's I've borrowed bits from.

agent1.channels.ch1.type = memory

agent1.sources =  tailsource-1
agent1.sources.channels = ch1
agent1.sources.tailsource-1.type = exec
agent1.sources.tailsource-1.command = tail -F /root/Desktop/apache-flume-1.3.0-SNAPSHOT/test.log
agent1.sources.tailsource-1.channels = ch1
agent1.sources.tailsource1.bind =
agent1.sources.tailsource-1.port = 41414

agent1.sources.tailsource-1.interceptors = hostint
agent1.sources.tailsource-1.interceptors.hostint.type = org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.tailsource-1.interceptors.hostint.preserveExisting = true
agent1.sources.tailsource-1.interceptors.hostint.useIP = false
agent1.sources.tailsource-1.interceptors.timestamp.type = timestamp

agent1.sinks.log-sink1.channel = ch1
agent1.sinks.log-sink1.type = FILE_ROLL
agent1.sinks.log-sink1.sink.directory = /root/Desktop/apache-flume-1.3.0-SNAPSHOT/logs/%{host}
agent1.sinks.log-sink1.sink.rollInterval = 86400
agent1.sinks.log-sink1.sink.filePrefix = %{filename}.%Y-%m-%d

Somehow I'm missing something as the %host never makes it forward, nor does the log filename....
before adding the host/filename stuff in it everything worked, just using filenames like 1345725771524-1
which aren't very intuitive by themselves but lumped into an unnamed directory makes them
even less so.

This is the resulting logfile...

23 Aug 2012 08:29:49,146 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160)
 - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to open file /root/Desktop/apache-flume-1.3.0-SNAPSHOT/logs/%{host}/1345728588635-1
while delivering event
        at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:166)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /root/Desktop/apache-flume-1.3.0-SNAPSHOT/logs/%{host}/1345728588635-1
(No such file or directory)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
        at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:160)
        ... 3 more

Anyone care to knock some the the dummy-dust loose that's accumulated in my head and point
me in the right direction to get this sorted?


View raw message