flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject Re: Question about Flume and Ambari
Date Thu, 05 Feb 2015 23:54:01 GMT
The directories you listed are on HDFS and you are using the file_roll sink which writes to
local FS, not to HDFS. You need to use the HDFS sink to write to HDFS.




Thanks, Hari

On Thu, Feb 5, 2015 at 3:51 PM, David Novogrodsky
<david.novogrodsky@gmail.com> wrote:

> I have the misfortune to be learning Ambari, Hadoop, flume and Sqoop
> at the same time.  I am running into a problem with a simple agent
> configuration for Flume:
> I have a very simple agent I am running from the command line using
> this command:
> [root@namenode lib]# flume-ng agent –conf conf –conf-file
> /etc/flume/conf/a1/flume.conf -n a1
> Here is the configuration file that is being managed by Ambari:
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
> a1.sources.r1.type = seq
> a1.sinks.k1.type = file_roll
> a1.sinks.k1.channel = c1
> a1.sinks.k1.sink.directory = /tmp/flume
> a1.channels.c1.type = memory
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
> ##
> Ok, but I am getting this error:
> 15/02/05 17:25:52 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
> org.apache.flume.EventDeliveryException: Failed to open file
> /tmp/flume/1423178752254-1 while delivering event
> at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: /tmp/flume/1423178752254-1
> (No such file or directory)
> But here are the permissions for that directory:
> [root@namenode a1]# hadoop fs -ls /tmp/
> Found 7 items
> drwxrwxrwx – root hdfs 0 2015-02-04 15:53 /tmp/flume
> drwxr-xr-x – admin hdfs 0 2015-01-09 18:12 /tmp/hive-beeswax-admin
> drwxr-xr-x – hue hdfs 0 2014-12-19 11:49 /tmp/hive-beeswax-hue
> I changed ownership of the directory
> [root@namenode a1]# sudo hadoop fs -ls /tmp/flume
> [root@namenode a1]# sudo hadoop fs -ls /tmp/
> Found 7 items
> drwxrwxrwx – root root 0 2015-02-04 15:53 /tmp/flume
> drwxr-xr-x – admin hdfs 0 2015-01-09 18:12 /tmp/hive-beeswax-admin
> But I got the same results.
> David Novogrodsky
> david.novogrodsky@gmail.com
> http://www.linkedin.com/in/davidnovogrodsky
Mime
View raw message