flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From prabhu k <prabhu.fl...@gmail.com>
Subject Re: Flume - ng running issue
Date Wed, 11 Jul 2012 08:30:07 GMT
Hi Alex,

Thanks for the response.

I have followed this link
http://mapredit.blogspot.de/2012/03/flumeng-evolution.html
and configured the syslog-agent.cnf and executed by using
bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf

but, script running from past 45 mints, it seems struck up flume-ng script.

Here is the syslog-agent.cnf file.

syslog-agent.cnf
=====================
syslog-agent.sources = Syslog
syslog-agent.channels = MemoryChannel-1
syslog-agent.sinks = Console

syslog-agent.sources.Syslog.type = syslogTcp
syslog-agent.sources.Syslog.port = 5140

syslog-agent.sources.Syslog.channels = MemoryChannel-1
syslog-agent.sinks.Console.channel = MemoryChannel-1

syslog-agent.sinks.Console.type = logger
syslog-agent.channels.MemoryChannel-1.type = memory

Execution log:
===================
root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf
Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hadoop libraries found via
(/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
Info: Excluding
/usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from
classpath
Info: Excluding
/usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
classpath
+ exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp
'/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
-Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
org.apache.flume.node.Application -n syslog-agent -f conf/syslog-agent.cnf
12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 1
12/07/11 12:40:27 INFO node.FlumeNode: Flume node starting - syslog-agent
12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Node manager
starting
12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 11
12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider:
Configuration provider starting
12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider:
Reloading configuration file:conf/syslog-agent.cnf
12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console
12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console
12/07/11 12:40:27 INFO conf.FlumeConfiguration: Added sinks: Console Agent:
syslog-agent
12/07/11 12:40:27 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration  for agents: [syslog-agent]
12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider:
Creating channels
12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider:
created channel MemoryChannel-1
12/07/11 12:40:27 INFO sink.DefaultSinkFactory: Creating instance of sink
Console typelogger
12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting new
configuration:{ sourceRunners:{Syslog=EventDrivenSourceRunner: {
source:org.apache.flume.source.SyslogTcpSource@110c424 }}
sinkRunners:{Console=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@1bd2664 counterGroup:{
name:null counters:{} } }}
channels:{MemoryChannel-1=org.apache.flume.channel.MemoryChannel@1238bd2} }
12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting
Channel MemoryChannel-1
12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink
Console
12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting
Source Syslog
12/07/11 12:40:28 INFO source.SyslogTcpSource: Syslog TCP Source starting...

Please let me know, if i am doing anything wrong?

Thanks,
Prabhu



On Wed, Jul 11, 2012 at 11:48 AM, alo alt <wget.null@gmail.com> wrote:

> Hi,
>
> what exactly is the issue? Error messages, things like that.
>
> You posted only debugging and exec logs, thats pretty normal. For a first
> short overview about flume take a look into:
>
> https://cwiki.apache.org/FLUME/getting-started.html
>
> http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html
> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html
>
> regards,
>  Alex
>
> On Jul 10, 2012, at 3:09 PM, prabhu k wrote:
>
> > Hi,
> >
> > While i am executing flume-ng, i am getting following out put on my
> > terminal, it seems got struck.
> >
> > root@md-trngpoc1
> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
> > bin/flume-ng agent -n agent1 -c conf -f conf/agent1.conf
> > Info: Sourcing environment configuration script
> >
> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env.sh
> > Info: Including Hadoop libraries found via
> > (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
> > Info: Excluding
> > /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from
> > classpath
> > Info: Excluding
> > /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
> > classpath
> > + exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m -Xmx200m
> -cp
> >
> '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
> >
> -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
> > org.apache.flume.node.Application -n agent1 -f conf/agent1.conf
> >
> > in execution summary:
> > ----------------------------------------
> > Info: Including Hadoop libraries found via
> > (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
> >
> > hadoop lib path pointing to /usr/local/hadoop_dir/hadoop/bin/hadoop
> > directory insted of /usr/local/hadoop_dir/hadoop/lib path.
> >
> > Here my questions are;
> >
> > 1. in flume-ng script anything changes need in add_hadoop_paths function.
> > 2. anything wrong in flume-env.sh script
> > 3. anything wrong in agent1.conf file.
> >
> > Please find the attached configuaration details and let me know any more
> > details.
> >
> > Could you pleae suggest me on this issue.
> >
> > Thanks,
> > Prabhu
> > <flume_config.txt>
>
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>

Mime
View raw message