flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Anchlia <mohitanch...@gmail.com>
Subject Re: Flume ng not running
Date Sun, 01 Apr 2012 14:44:42 GMT
Thanks for the post.

I have some questions:

1. Is writing to the SequenceFile better? I thought sequenceFile is
generally good for binary data.

2. Generally is one agent sufficient for one client? If not then idea is to
just start multiple agents?

3. How to deal with failures of agents or clients?

On Sun, Apr 1, 2012 at 2:39 AM, alo alt <wget.null@googlemail.com> wrote:

> I wrote a couple of weeks ago a blog post:
> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html
>
> here u found all what you need.
>
> - Alex
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
> On Apr 1, 2012, at 8:26 AM, shekhar sharma wrote:
>
> > Hello Mohit,
> > Create a property file "foo.properties" and then run the following
> command to start your agent
> > ./flume-ng  node --conf ../conf/ -f  ../conf/foo.properties -n foo
> >
> > and then you can run ur avro client to send some sample data which will
> be collected by the foo's source an it agent i.e. avro source which then it
> will send to HDFS.
> >
> > flume-ng avro-client --conf ../conf -H localhost -p 41414 -F /etc/passwd
> >
> >
> > you need to define the host and the port for foo.source in your
> foo.properties file. Hope that helps.
> >
> > Regards,
> > Som
> >
> >
> > On Sat, Mar 31, 2012 at 5:27 AM, Mohit Anchlia <mohitanchlia@gmail.com>
> wrote:
> > I am trying to test how flume ng works so I create a conf and tried to
> run:
> >
> >  foo.sources = avroSrc
> > foo.channels = memoryChannel
> > foo.sinks = hdfsSink
> > # For each one of the sources, the type is defined
> > foo.sources.avroSrc.type = avro
> > # The channel can be defined as follows.
> > foo.sources.avroSrc.channels = memoryChannel
> > # Each sink's type must be defined
> > foo.sinks.hdfsSink.type = hdfs
> > foo.sinks.hdfsSink.path = hdfs://dsdb1:9000/flume
> > #Specify the channel the sink should use
> > foo.sinks.hdfsSink.channel = memoryChannel
> > # Each channel's type is defined.
> > foo.channels.memoryChannel.type = memory
> > # Other config values specific to each type of channel(sink or source)
> > # can be defined as well
> > # In this case, it specifies the capacity of the memory channel
> > foo.channels.memoryChannel.capacity = 100
> >
> > But java process is not running:
> >
> > [root@dslg1 flume-1.2.0-incubating-SNAPSHOT]# ./bin/flume-ng  node
> --conf ../conf/flume-conf.properties
> > + exec /usr/java/jdk1.6.0_24/bin/java -Xmx20m -cp
> '../conf/flume-conf.properties:/root/.mohit/flume-1.2.0-incubating-SNAPSHOT/lib/*'
> -Djava.library.path= org.apache.flume.node.Application
> > log4j:WARN No appenders could be found for logger
> (org.apache.flume.node.Application).
> > log4j:WARN Please initialize the log4j system properly.
> > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfigfor more info.
> > --
> > Can you please suggest what I might be doing wrong?
> >
> >
> >
>
>

Mime
View raw message