flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From prabhu k <prabhu.fl...@gmail.com>
Subject Re: tail source exec unable to HDFS sink.
Date Tue, 18 Sep 2012 11:27:20 GMT
Hi,

Please find the following flume.conf & flume.log files.

I have marked in red colour below is that having any issue?

flume.conf
=============
agent1.sources = tail
agent1.channels = MemoryChannel-2
agent1.sinks = HDFS
agent1.sources.tail.type = exec
agent1.sources.tail.command = tail
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
agent1.sources.tail.channels = MemoryChannel-2
agent1.sources.tail.interceptors = hostint
agent1.sources.tail.interceptors.hostint.type =
org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.tail.interceptors.hostint.preserveExisting = true
agent1.sources.tail.interceptors.hostint.useIP = false
agent1.sinks.HDFS.hdfs.channel = MemoryChannel-2
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://<hostname>:54310/user/
agent1.sinks.HDFS.hdfs.fileType = dataStream
agent1.sinks.HDFS.hdfs.writeFormat = text
agent1.channels.MemoryChannel-2.type = memory

flume.log
================
12/09/18 16:52:16 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 1
12/09/18 16:52:16 INFO node.FlumeNode: Flume node starting - agent1
12/09/18 16:52:16 INFO nodemanager.DefaultLogicalNodeManager: Node manager
starting
12/09/18 16:52:16 INFO properties.PropertiesFileConfigurationProvider:
Configuration provider starting
12/09/18 16:52:16 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 10
12/09/18 16:52:16 INFO properties.PropertiesFileConfigurationProvider:
Reloading configuration file:conf/flume.conf
12/09/18 16:52:16 INFO conf.FlumeConfiguration: Processing:HDFS
12/09/18 16:52:16 INFO conf.FlumeConfiguration: Processing:HDFS
12/09/18 16:52:16 INFO conf.FlumeConfiguration: Processing:HDFS
12/09/18 16:52:16 INFO conf.FlumeConfiguration: Processing:HDFS
12/09/18 16:52:16 INFO conf.FlumeConfiguration: Processing:HDFS
12/09/18 16:52:16 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
agent1
12/09/18 16:52:16 WARN conf.FlumeConfiguration: Configuration empty for:
HDFS.Removed.
12/09/18 16:52:16 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration  for agents: [agent1]
12/09/18 16:52:16 INFO properties.PropertiesFileConfigurationProvider:
Creating channels
12/09/18 16:52:16 INFO properties.PropertiesFileConfigurationProvider:
created channel MemoryChannel-2
12/09/18 16:52:16 INFO nodemanager.DefaultLogicalNodeManager: Starting new
configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: {
source:org.apache.flume.source.ExecSource@a1d1f4 }} sinkRunners:{}
channels:{MemoryChannel-2=org.apache.flume.channel.MemoryChannel@1df280b} }
12/09/18 16:52:16 INFO nodemanager.DefaultLogicalNodeManager: Starting
Channel MemoryChannel-2
12/09/18 16:52:16 INFO nodemanager.DefaultLogicalNodeManager: Starting
Source tail
12/09/18 16:52:16 INFO source.ExecSource: Exec source starting with
command:tail
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt

-Prabhu

On Tue, Sep 18, 2012 at 4:40 PM, Brock Noland <brock@cloudera.com> wrote:

> Yes they should work together. Please send the updated conf and log file.
>
> --
> Brock Noland
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
>  On Tuesday, September 18, 2012 at 5:49 AM, prabhu k wrote:
>
>  I have tried both ways, but still not working....
> can you please confirm flume 1.2.0 support hadoop 1.0.3 version?
>
> Thanks,
> Prabhu.
> On Tue, Sep 18, 2012 at 3:32 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>
> can you write something in file continuously after you start flume-ng
>
> if you do tail -f it will start getting only new entries
> or you can just change the command  in the config file from tail -f to
> tail so each time it bring default last 10 lines from the the file
>
> ~nitin
>
> On Tue, Sep 18, 2012 at 2:51 PM, prabhu k <prabhu.flume@gmail.com> wrote:
> > Hi Nitin,
> >
> > While executing flume-ng, i have updated the flume_test.txt file,still
> > unable to do HDFS sink.
> >
> > Thanks,
> > Prabhu.
> >
> > On Tue, Sep 18, 2012 at 2:35 PM, Nitin Pawar <nitinpawar432@gmail.com>
> > wrote:
> >>
> >> Hi Prabhu,
> >>
> >> are you sure there is continuous text being written to your file
> >> flume_test.txt.
> >>
> >> if nothing is written to that file, flume will not write anything into
> >> hdfs.
> >>
> >> On Tue, Sep 18, 2012 at 2:31 PM, prabhu k <prabhu.flume@gmail.com>
> wrote:
> >> > Hi Brock,
> >> >
> >> > Thanks for the reply.
> >> >
> >> > As per your suggestion, i have modified,but still same issue.
> >> >
> >> > My hadoop version is : 1.0.3 & Flume version is 1.2.0. Please let us
> >> > know is
> >> > there any incompatible version?
> >> >
> >> > On Mon, Sep 17, 2012 at 8:01 PM, Brock Noland <brock@cloudera.com>
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> I believe, this line:
> >> >> agent1.sinks.HDFS.hdfs.type = hdfs
> >> >>
> >> >> should be:
> >> >> agent1.sinks.HDFS.type = hdfs
> >> >>
> >> >> Brock
> >> >>
> >> >> On Mon, Sep 17, 2012 at 5:17 AM, prabhu k <prabhu.flume@gmail.com>
> >> >> wrote:
> >> >> > Hi Users,
> >> >> >
> >> >> > I have followed the below link for sample text file to HDFS sink
> >> >> > using
> >> >> > tail
> >> >> > source.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html#more
> >> >> >
> >> >> > I have executed flume-ng like as below command. it seems got stuck.
> >> >> > and
> >> >> > attached flume.conf file.
> >> >> >
> >> >> > #bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf
> >> >> >
> >> >> >
> >> >> > flume.conf
> >> >> > ==========
> >> >> > agent1.sources = tail
> >> >> > agent1.channels = MemoryChannel-2
> >> >> > agent1.sinks = HDFS
> >> >> >
> >> >> > agent1.sources.tail.type = exec
> >> >> > agent1.sources.tail.command = tail -F
> >> >> >
> >> >> >
> >> >> >
> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
> >> >> > agent1.sources.tail.channels = MemoryChannel-2
> >> >> >
> >> >> > agent1.sources.tail.interceptors = hostint
> >> >> > agent1.sources.tail.interceptors.hostint.type =
> >> >> > org.apache.flume.interceptor.HostInterceptor$Builder
> >> >> > agent1.sources.tail.interceptors.hostint.preserveExisting = true
> >> >> > agent1.sources.tail.interceptors.hostint.useIP = false
> >> >> >
> >> >> > agent1.sinks.HDFS.hdfs.channel = MemoryChannel-2
> >> >> > agent1.sinks.HDFS.hdfs.type = hdfs
> >> >> > agent1.sinks.HDFS.hdfs.path = hdfs://<hostname>:54310/user
> >> >> >
> >> >> > agent1.sinks.HDFS.hdfs.fileType = dataStream
> >> >> > agent1.sinks.HDFS.hdfs.writeFormat = text
> >> >> > agent1.channels.MemoryChannel-2.type = memory
> >> >> >
> >> >> >
> >> >> >
> >> >> > flume.log
> >> >> > ==========
> >> >> > 12/09/17 15:40:05 INFO lifecycle.LifecycleSupervisor: Starting
> >> >> > lifecycle
> >> >> > supervisor 1
> >> >> > 12/09/17 15:40:05 INFO node.FlumeNode: Flume node starting - agent1
> >> >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager:
Node
> >> >> > manager
> >> >> > starting
> >> >> > 12/09/17 15:40:05 INFO
> >> >> > properties.PropertiesFileConfigurationProvider:
> >> >> > Configuration provider starting
> >> >> > 12/09/17 15:40:05 INFO lifecycle.LifecycleSupervisor: Starting
> >> >> > lifecycle
> >> >> > supervisor 10
> >> >> > 12/09/17 15:40:05 INFO
> >> >> > properties.PropertiesFileConfigurationProvider:
> >> >> > Reloading configuration file:conf/flume.conf
> >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
> >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
> >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
> >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
> >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
> >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Added sinks: HDFS
> >> >> > Agent:
> >> >> > agent1
> >> >> > 12/09/17 15:40:05 WARN conf.FlumeConfiguration: Configuration
empty
> >> >> > for:
> >> >> > HDFS.Removed.
> >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Post-validation
> flume
> >> >> > configuration contains configuration  for agents: [agent1]
> >> >> > 12/09/17 15:40:05 INFO
> >> >> > properties.PropertiesFileConfigurationProvider:
> >> >> > Creating channels
> >> >> > 12/09/17 15:40:05 INFO
> >> >> > properties.PropertiesFileConfigurationProvider:
> >> >> > created channel MemoryChannel-2
> >> >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager:
> >> >> > Starting
> >> >> > new
> >> >> > configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: {
> >> >> > source:org.apache.flume.source.ExecSource@c24c0 }} sinkRunners:{}
> >> >> >
> >> >> >
> >> >> >
> channels:{MemoryChannel-2=org.apache.flume.channel.MemoryChannel@140c281}
> }
> >> >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager:
> >> >> > Starting
> >> >> > Channel MemoryChannel-2
> >> >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager:
> >> >> > Starting
> >> >> > Source tail
> >> >> > 12/09/17 15:40:05 INFO source.ExecSource: Exec source starting
with
> >> >> > command:tail -F
> >> >> >
> >> >> >
> >> >> >
> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
> >> >> >
> >> >> > Please suggest and help me on this issue.
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Apache MRUnit - Unit testing MapReduce -
> >> >> http://incubator.apache.org/mrunit/
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Nitin Pawar
> >
> >
>
>
>
> --
> Nitin Pawar
>
>
>
>

Mime
View raw message