flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From prabhu k <prabhu.fl...@gmail.com>
Subject Re: Flume - ng running issue
Date Wed, 11 Jul 2012 12:00:43 GMT
Hi Alex,

I have executed like below with the agent ip, connection got succeeded, but
where can i include this command?

root@md-trngpoc1:/tmp# nc -v 10.5.114.110 54310 < /tmp/foo
Connection to 10.5.114.110 54310 port [tcp/*] succeeded!


On Wed, Jul 11, 2012 at 3:14 PM, alo alt <wget.null@gmail.com> wrote:

>
> example:
>
> >>>> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" >
/tmp/foo
> >>>> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo
>
> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
> nc -v 10.10.10.10 5140 < /tmp/foo
>
> 10.10.10.10 is the IP who the flume agent is waiting for a connection.
>
> man nc will help too.
>
>
> On Jul 11, 2012, at 11:36 AM, prabhu k wrote:
>
> > I did not get you, could you please elaborate bit more?
> >
> > you should replace FLUME SYSLOG AGENT IP with the ip of the agent!
> >
> > where i can add the FLUME SYSLOG AGENT IP ?
> >
> > On Wed, Jul 11, 2012 at 2:33 PM, alo alt <wget.null@gmail.com> wrote:
> >
> >> you should replace FLUME SYSLOG AGENT IP with the ip of the agent!
> >>
> >>
> >> On Jul 11, 2012, at 11:01 AM, prabhu k wrote:
> >>
> >>> Hi Alex,
> >>>
> >>> I have executed the below commands, i am getting following output:
> >>>
> >>> root@md-trngpoc1
> >> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
> >>> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo
> >>> nc: port range not valid
> >>> root@md-trngpoc1:/tmp# more foo
> >>> <13>Jun 20 12:12:12 host foo[345]: a syslog message
> >>>
> >>> Please suggest any idea on this issue.
> >>>
> >>> On Wed, Jul 11, 2012 at 2:15 PM, alo alt <wget.null@gmail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> there stuck nothing, flume is waiting for a event. Point a syslog
> server
> >>>> to the configured port or use the netcat example to see if the event
> can
> >>>> delivered:
> >>>>
> >>>> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" >
/tmp/foo
> >>>> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo
> >>>>
> >>>> - Alex
> >>>>
> >>>>
> >>>> On Jul 11, 2012, at 10:30 AM, prabhu k wrote:
> >>>>
> >>>>> Hi Alex,
> >>>>>
> >>>>> Thanks for the response.
> >>>>>
> >>>>> I have followed this link
> >>>>> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html
> >>>>> and configured the syslog-agent.cnf and executed by using
> >>>>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf
> >>>>>
> >>>>> but, script running from past 45 mints, it seems struck up flume-ng
> >>>> script.
> >>>>>
> >>>>> Here is the syslog-agent.cnf file.
> >>>>>
> >>>>> syslog-agent.cnf
> >>>>> =====================
> >>>>> syslog-agent.sources = Syslog
> >>>>> syslog-agent.channels = MemoryChannel-1
> >>>>> syslog-agent.sinks = Console
> >>>>>
> >>>>> syslog-agent.sources.Syslog.type = syslogTcp
> >>>>> syslog-agent.sources.Syslog.port = 5140
> >>>>>
> >>>>> syslog-agent.sources.Syslog.channels = MemoryChannel-1
> >>>>> syslog-agent.sinks.Console.channel = MemoryChannel-1
> >>>>>
> >>>>> syslog-agent.sinks.Console.type = logger
> >>>>> syslog-agent.channels.MemoryChannel-1.type = memory
> >>>>>
> >>>>> Execution log:
> >>>>> ===================
> >>>>> root@md-trngpoc1
> >>>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
> >>>>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf
> >>>>> Warning: No configuration directory set! Use --conf <dir>
to
> override.
> >>>>> Info: Including Hadoop libraries found via
> >>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
> >>>>> Info: Excluding
> >>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
from
> >>>>> classpath
> >>>>> Info: Excluding
> >>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar
> >> from
> >>>>> classpath
> >>>>> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp
> >>>>>
> >>>>
> >>
> '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
> >>>>>
> >>>>
> >>
> -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
> >>>>> org.apache.flume.node.Application -n syslog-agent -f
> >>>> conf/syslog-agent.cnf
> >>>>> 12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting
> >> lifecycle
> >>>>> supervisor 1
> >>>>> 12/07/11 12:40:27 INFO node.FlumeNode: Flume node starting -
> >> syslog-agent
> >>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Node
> >>>> manager
> >>>>> starting
> >>>>> 12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting
> >> lifecycle
> >>>>> supervisor 11
> >>>>> 12/07/11 12:40:27 INFO
> properties.PropertiesFileConfigurationProvider:
> >>>>> Configuration provider starting
> >>>>> 12/07/11 12:40:27 INFO
> properties.PropertiesFileConfigurationProvider:
> >>>>> Reloading configuration file:conf/syslog-agent.cnf
> >>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console
> >>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console
> >>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Added sinks: Console
> >>>> Agent:
> >>>>> syslog-agent
> >>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Post-validation
flume
> >>>>> configuration contains configuration  for agents: [syslog-agent]
> >>>>> 12/07/11 12:40:27 INFO
> properties.PropertiesFileConfigurationProvider:
> >>>>> Creating channels
> >>>>> 12/07/11 12:40:27 INFO
> properties.PropertiesFileConfigurationProvider:
> >>>>> created channel MemoryChannel-1
> >>>>> 12/07/11 12:40:27 INFO sink.DefaultSinkFactory: Creating instance
of
> >> sink
> >>>>> Console typelogger
> >>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager:
> Starting
> >>>> new
> >>>>> configuration:{ sourceRunners:{Syslog=EventDrivenSourceRunner: {
> >>>>> source:org.apache.flume.source.SyslogTcpSource@110c424 }}
> >>>>> sinkRunners:{Console=SinkRunner: {
> >>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@1bd2664counterGroup
> :{
> >>>>> name:null counters:{} } }}
> >>>>>
> >> channels:{MemoryChannel-1=org.apache.flume.channel.MemoryChannel@1238bd2
> }
> >>>> }
> >>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager:
> Starting
> >>>>> Channel MemoryChannel-1
> >>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager:
> Starting
> >>>> Sink
> >>>>> Console
> >>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager:
> Starting
> >>>>> Source Syslog
> >>>>> 12/07/11 12:40:28 INFO source.SyslogTcpSource: Syslog TCP Source
> >>>> starting...
> >>>>>
> >>>>> Please let me know, if i am doing anything wrong?
> >>>>>
> >>>>> Thanks,
> >>>>> Prabhu
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Jul 11, 2012 at 11:48 AM, alo alt <wget.null@gmail.com>
> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> what exactly is the issue? Error messages, things like that.
> >>>>>>
> >>>>>> You posted only debugging and exec logs, thats pretty normal.
For a
> >>>> first
> >>>>>> short overview about flume take a look into:
> >>>>>>
> >>>>>> https://cwiki.apache.org/FLUME/getting-started.html
> >>>>>>
> >>>>>>
> >>>>
> >>
> http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html
> >>>>>> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html
> >>>>>>
> >>>>>> regards,
> >>>>>> Alex
> >>>>>>
> >>>>>> On Jul 10, 2012, at 3:09 PM, prabhu k wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> While i am executing flume-ng, i am getting following out
put on my
> >>>>>>> terminal, it seems got struck.
> >>>>>>>
> >>>>>>> root@md-trngpoc1
> >>>>>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
> >>>>>>> bin/flume-ng agent -n agent1 -c conf -f conf/agent1.conf
> >>>>>>> Info: Sourcing environment configuration script
> >>>>>>>
> >>>>>>
> >>>>
> >>
> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env.sh
> >>>>>>> Info: Including Hadoop libraries found via
> >>>>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
> >>>>>>> Info: Excluding
> >>>>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
> from
> >>>>>>> classpath
> >>>>>>> Info: Excluding
> >>>>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar
> >>>> from
> >>>>>>> classpath
> >>>>>>> + exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m
> >> -Xmx200m
> >>>>>> -cp
> >>>>>>>
> >>>>>>
> >>>>
> >>
> '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
> >>>>>>>
> >>>>>>
> >>>>
> >>
> -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
> >>>>>>> org.apache.flume.node.Application -n agent1 -f conf/agent1.conf
> >>>>>>>
> >>>>>>> in execution summary:
> >>>>>>> ----------------------------------------
> >>>>>>> Info: Including Hadoop libraries found via
> >>>>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
> >>>>>>>
> >>>>>>> hadoop lib path pointing to /usr/local/hadoop_dir/hadoop/bin/hadoop
> >>>>>>> directory insted of /usr/local/hadoop_dir/hadoop/lib path.
> >>>>>>>
> >>>>>>> Here my questions are;
> >>>>>>>
> >>>>>>> 1. in flume-ng script anything changes need in add_hadoop_paths
> >>>> function.
> >>>>>>> 2. anything wrong in flume-env.sh script
> >>>>>>> 3. anything wrong in agent1.conf file.
> >>>>>>>
> >>>>>>> Please find the attached configuaration details and let
me know any
> >>>> more
> >>>>>>> details.
> >>>>>>>
> >>>>>>> Could you pleae suggest me on this issue.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Prabhu
> >>>>>>> <flume_config.txt>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Alexander Alten-Lorenz
> >>>>>> http://mapredit.blogspot.com
> >>>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Alexander Alten-Lorenz
> >>>> http://mapredit.blogspot.com
> >>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >>>>
> >>>>
> >>
> >>
> >> --
> >> Alexander Alten-Lorenz
> >> http://mapredit.blogspot.com
> >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >>
> >>
>
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>

Mime
View raw message