Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4340D970D for ; Wed, 11 Jul 2012 09:36:34 +0000 (UTC) Received: (qmail 44140 invoked by uid 500); 11 Jul 2012 09:36:34 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 43857 invoked by uid 500); 11 Jul 2012 09:36:33 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 43834 invoked by uid 99); 11 Jul 2012 09:36:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Jul 2012 09:36:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of prabhu.flume@gmail.com designates 209.85.161.179 as permitted sender) Received: from [209.85.161.179] (HELO mail-gg0-f179.google.com) (209.85.161.179) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Jul 2012 09:36:28 +0000 Received: by ggnk3 with SMTP id k3so944402ggn.38 for ; Wed, 11 Jul 2012 02:36:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=BqBQ848LYkVDSxXaFU/WMaa3Wv6jXUymufEr7QS4uQ0=; b=WYadpKgePDb4vRor8eiBQyFbCEbABDBZo9NoRVxAU4CWQekcQtqO78WtgLHGmgi9a6 ntHcyfIgBxt7VSvYpXPodsqNkjLs6r3TQIuZgldQB2i41TDompXFZNxBmq2n7HcWWf3p llw31hsSB0dqCQt73etb5STHdE/kTcQ5xXcG4SuF2Oor20tvLdLpJR9LD+eIgl75cSIU FL7/QFPBvi03mLvzbWtTuG1CAe9Asc/EBe97k4+dKyhTmaeMaFjF5tsu14rPBYTV2aov bnsajhr5GfVjQzyGhlBAE8nwj3rfMabUS8hKmLc2xv83ziT9KQwBsNxYgGcTQ8yVI9EZ bYZw== MIME-Version: 1.0 Received: by 10.50.202.5 with SMTP id ke5mr13884937igc.64.1341999367627; Wed, 11 Jul 2012 02:36:07 -0700 (PDT) Received: by 10.64.95.163 with HTTP; Wed, 11 Jul 2012 02:36:07 -0700 (PDT) In-Reply-To: References: Date: Wed, 11 Jul 2012 15:06:07 +0530 Message-ID: Subject: Re: Flume - ng running issue From: prabhu k To: user@flume.apache.org Content-Type: multipart/alternative; boundary=f46d04479f9198a59604c48a9303 X-Virus-Checked: Checked by ClamAV on apache.org --f46d04479f9198a59604c48a9303 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I did not get you, could you please elaborate bit more? you should replace FLUME SYSLOG AGENT IP with the ip of the agent! where i can add the FLUME SYSLOG AGENT IP ? On Wed, Jul 11, 2012 at 2:33 PM, alo alt wrote: > you should replace FLUME SYSLOG AGENT IP with the ip of the agent! > > > On Jul 11, 2012, at 11:01 AM, prabhu k wrote: > > > Hi Alex, > > > > I have executed the below commands, i am getting following output: > > > > root@md-trngpoc1 > :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# > > nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo > > nc: port range not valid > > root@md-trngpoc1:/tmp# more foo > > <13>Jun 20 12:12:12 host foo[345]: a syslog message > > > > Please suggest any idea on this issue. > > > > On Wed, Jul 11, 2012 at 2:15 PM, alo alt wrote: > > > >> Hi, > >> > >> there stuck nothing, flume is waiting for a event. Point a syslog serv= er > >> to the configured port or use the netcat example to see if the event c= an > >> delivered: > >> > >> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo > >> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo > >> > >> - Alex > >> > >> > >> On Jul 11, 2012, at 10:30 AM, prabhu k wrote: > >> > >>> Hi Alex, > >>> > >>> Thanks for the response. > >>> > >>> I have followed this link > >>> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html > >>> and configured the syslog-agent.cnf and executed by using > >>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf > >>> > >>> but, script running from past 45 mints, it seems struck up flume-ng > >> script. > >>> > >>> Here is the syslog-agent.cnf file. > >>> > >>> syslog-agent.cnf > >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >>> syslog-agent.sources =3D Syslog > >>> syslog-agent.channels =3D MemoryChannel-1 > >>> syslog-agent.sinks =3D Console > >>> > >>> syslog-agent.sources.Syslog.type =3D syslogTcp > >>> syslog-agent.sources.Syslog.port =3D 5140 > >>> > >>> syslog-agent.sources.Syslog.channels =3D MemoryChannel-1 > >>> syslog-agent.sinks.Console.channel =3D MemoryChannel-1 > >>> > >>> syslog-agent.sinks.Console.type =3D logger > >>> syslog-agent.channels.MemoryChannel-1.type =3D memory > >>> > >>> Execution log: > >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >>> root@md-trngpoc1 > >> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# > >>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf > >>> Warning: No configuration directory set! Use --conf to override= . > >>> Info: Including Hadoop libraries found via > >>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access > >>> Info: Excluding > >>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from > >>> classpath > >>> Info: Excluding > >>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar > from > >>> classpath > >>> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp > >>> > >> > '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/lo= cal/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_= dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local= /hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoo= p-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/u= sr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/ha= doop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir= /hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/= hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libe= xec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../li= b/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib= /commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/= commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1= .0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1= .jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/lo= cal/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop= _dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/= hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/h= adoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libe= xec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../li= b/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacit= y-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fa= irscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-th= riftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.1= 0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.ja= r:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:= /usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/loca= l/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoo= p/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/.= ./lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jerse= y-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.j= ar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_di= r/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.= 2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/loc= al/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop= _dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libex= ec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec= /../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1= /jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.= 1.jar' > >>> > >> > -Djava.library.path=3D/usr/local/flume_dir/flume/flume-1.2.0-incubating-S= NAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-3= 2 > >>> org.apache.flume.node.Application -n syslog-agent -f > >> conf/syslog-agent.cnf > >>> 12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting > lifecycle > >>> supervisor 1 > >>> 12/07/11 12:40:27 INFO node.FlumeNode: Flume node starting - > syslog-agent > >>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Node > >> manager > >>> starting > >>> 12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting > lifecycle > >>> supervisor 11 > >>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider= : > >>> Configuration provider starting > >>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider= : > >>> Reloading configuration file:conf/syslog-agent.cnf > >>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console > >>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console > >>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Added sinks: Console > >> Agent: > >>> syslog-agent > >>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Post-validation flume > >>> configuration contains configuration for agents: [syslog-agent] > >>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider= : > >>> Creating channels > >>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider= : > >>> created channel MemoryChannel-1 > >>> 12/07/11 12:40:27 INFO sink.DefaultSinkFactory: Creating instance of > sink > >>> Console typelogger > >>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Startin= g > >> new > >>> configuration:{ sourceRunners:{Syslog=3DEventDrivenSourceRunner: { > >>> source:org.apache.flume.source.SyslogTcpSource@110c424 }} > >>> sinkRunners:{Console=3DSinkRunner: { > >>> policy:org.apache.flume.sink.DefaultSinkProcessor@1bd2664counterGroup= :{ > >>> name:null counters:{} } }} > >>> > channels:{MemoryChannel-1=3Dorg.apache.flume.channel.MemoryChannel@1238bd= 2} > >> } > >>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Startin= g > >>> Channel MemoryChannel-1 > >>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Startin= g > >> Sink > >>> Console > >>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Startin= g > >>> Source Syslog > >>> 12/07/11 12:40:28 INFO source.SyslogTcpSource: Syslog TCP Source > >> starting... > >>> > >>> Please let me know, if i am doing anything wrong? > >>> > >>> Thanks, > >>> Prabhu > >>> > >>> > >>> > >>> On Wed, Jul 11, 2012 at 11:48 AM, alo alt wrote= : > >>> > >>>> Hi, > >>>> > >>>> what exactly is the issue? Error messages, things like that. > >>>> > >>>> You posted only debugging and exec logs, thats pretty normal. For a > >> first > >>>> short overview about flume take a look into: > >>>> > >>>> https://cwiki.apache.org/FLUME/getting-started.html > >>>> > >>>> > >> > http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUse= rGuide.html > >>>> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html > >>>> > >>>> regards, > >>>> Alex > >>>> > >>>> On Jul 10, 2012, at 3:09 PM, prabhu k wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> While i am executing flume-ng, i am getting following out put on my > >>>>> terminal, it seems got struck. > >>>>> > >>>>> root@md-trngpoc1 > >>>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# > >>>>> bin/flume-ng agent -n agent1 -c conf -f conf/agent1.conf > >>>>> Info: Sourcing environment configuration script > >>>>> > >>>> > >> > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env= .sh > >>>>> Info: Including Hadoop libraries found via > >>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access > >>>>> Info: Excluding > >>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar fro= m > >>>>> classpath > >>>>> Info: Excluding > >>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar > >> from > >>>>> classpath > >>>>> + exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m > -Xmx200m > >>>> -cp > >>>>> > >>>> > >> > '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/loc= al/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_d= ir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/= libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir= /hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.= 3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/ha= doop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/lib= exec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libe= xec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/= commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-c= ollections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-co= nfiguration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-dae= mon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-= 1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/l= ocal/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_= dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop= /libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libe= xec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libex= ec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/= commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.= 1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler= -1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler= -1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.= 3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/loca= l/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_di= r/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/.= ./lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jerse= y-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.= 8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/loc= al/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/= hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/li= bexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/ju= nit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_d= ir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop= /libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/s= ervlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xml= enc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.ja= r:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' > >>>>> > >>>> > >> > -Djava.library.path=3D/usr/local/flume_dir/flume/flume-1.2.0-incubating-S= NAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-3= 2 > >>>>> org.apache.flume.node.Application -n agent1 -f conf/agent1.conf > >>>>> > >>>>> in execution summary: > >>>>> ---------------------------------------- > >>>>> Info: Including Hadoop libraries found via > >>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access > >>>>> > >>>>> hadoop lib path pointing to /usr/local/hadoop_dir/hadoop/bin/hadoop > >>>>> directory insted of /usr/local/hadoop_dir/hadoop/lib path. > >>>>> > >>>>> Here my questions are; > >>>>> > >>>>> 1. in flume-ng script anything changes need in add_hadoop_paths > >> function. > >>>>> 2. anything wrong in flume-env.sh script > >>>>> 3. anything wrong in agent1.conf file. > >>>>> > >>>>> Please find the attached configuaration details and let me know any > >> more > >>>>> details. > >>>>> > >>>>> Could you pleae suggest me on this issue. > >>>>> > >>>>> Thanks, > >>>>> Prabhu > >>>>> > >>>> > >>>> > >>>> -- > >>>> Alexander Alten-Lorenz > >>>> http://mapredit.blogspot.com > >>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >>>> > >>>> > >> > >> > >> -- > >> Alexander Alten-Lorenz > >> http://mapredit.blogspot.com > >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >> > >> > > > -- > Alexander Alten-Lorenz > http://mapredit.blogspot.com > German Hadoop LinkedIn Group: http://goo.gl/N8pCF > > --f46d04479f9198a59604c48a9303 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I did not get you, could you please elaborate bit more?
=A0
you should replace FLUME SYSLOG AGENT IP with the ip of the agent!
=A0
where i can add the FLUME SYSLOG AGENT IP ?

On Wed, Jul 11, 2012 at 2:33 PM, alo alt <wge= t.null@gmail.com> wrote:
you should replace FLUME SYSLOG AGENT= IP with the ip of the agent!


On Jul 11, 2012, at 11:01 AM, prabhu k wrote:
=
> Hi Alex,
>
> I have executed the below commands, i am = getting following output:
>
> root@md-trngpoc1:/usr/local/flume= _dir/flume/flume-1.2.0-incubating-SNAPSHOT#
> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo
> nc: port range = not valid
> root@md-trngpoc1:/tmp# more foo
> <13>Jun 20 = 12:12:12 host foo[345]: a syslog message
>
> Please suggest any= idea on this issue.
>
> On Wed, Jul 11, 2012 at 2:15 PM, alo alt <wget.null@gmail.com> wrote:
>
>>= ; Hi,
>>
>> there stuck nothing, flume is waiting for a e= vent. Point a syslog server
>> to the configured port or use the netcat example to see if the eve= nt can
>> delivered:
>>
>> echo "<13>= Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
>> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo
>>
>= > - Alex
>>
>>
>> On Jul 11, 2012, at 10:30 A= M, prabhu k wrote:
>>
>>> Hi Alex,
>>>
>>> Thanks for the response.
>>>
>>> I hav= e followed this link
>>> http://mapredit.blogspot= .de/2012/03/flumeng-evolution.html
>>> and configured the syslog-agent.cnf and executed by using
&= gt;>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf
= >>>
>>> but, script running from past 45 mints, it see= ms struck up flume-ng
>> script.
>>>
>>> Here is the syslog-agent.c= nf file.
>>>
>>> syslog-agent.cnf
>>> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>= > syslog-agent.sources =3D Syslog
>>> syslog-agent.channels = =3D MemoryChannel-1
>>> syslog-agent.sinks =3D Console
>>>
>>>= syslog-agent.sources.Syslog.type =3D syslogTcp
>>> syslog-agen= t.sources.Syslog.port =3D 5140
>>>
>>> syslog-agent= .sources.Syslog.channels =3D MemoryChannel-1
>>> syslog-agent.sinks.Console.channel =3D MemoryChannel-1
>= >>
>>> syslog-agent.sinks.Console.type =3D logger
>= >> syslog-agent.channels.MemoryChannel-1.type =3D memory
>>&= gt;
>>> Execution log:
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> root@md-trngpoc1
>> :/= usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
>>> = bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf
>>> Warning: No configuration directory set! Use --conf <dir>= ; to override.
>>> Info: Including Hadoop libraries found via>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access >>> Info: Excluding
>>> /usr/local/hadoop_dir/hadoop/l= ibexec/../lib/slf4j-api-1.4.3.jar from
>>> classpath
>>= ;> Info: Excluding
>>> /usr/local/hadoop_dir/hadoop/libexec/= ../lib/slf4j-log4j12-1.4.3.jar from
>>> classpath
>>> + exec /usr/lib/jvm/java-6-sun/bin/j= ava -Xmx20m -cp
>>>
>> '/usr/local/flume_dir/flume= /flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2= .0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/u= sr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..= :/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/h= adoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/li= bexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../li= b/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-b= eanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons= -cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.= jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.j= ar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.ja= r:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local= /hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/= hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec= /../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/co= mmons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons= -logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-= math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.= jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/h= adoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/lo= cal/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/lo= cal/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/h= adoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/h= adoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoo= p/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/= libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libe= xec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/.= ./lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-= 1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/us= r/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/h= adoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hado= op/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-= 0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/l= ocal/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/h= adoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/= ../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/or= o-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-200= 81211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoo= p_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
>>>
>> -Djava.library.path=3D/usr/local/flume_dir/flume/f= lume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../= lib/native/Linux-i386-32
>>> org.apache.flume.node.Application = -n syslog-agent -f
>> conf/syslog-agent.cnf
>>> 12/07/11 12:40:27 INFO lifec= ycle.LifecycleSupervisor: Starting lifecycle
>>> supervisor 1>>> 12/07/11 12:40:27 INFO node.FlumeNode: Flume node starting -= syslog-agent
>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: = Node
>> manager
>>> starting
>>> 12/07/11 = 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
>>= > supervisor 11
>>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationP= rovider:
>>> Configuration provider starting
>>> 12= /07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider:
>>> Reloading configuration file:conf/syslog-agent.cnf
>>= > 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console
= >>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Con= sole
>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Added sinks: C= onsole
>> Agent:
>>> syslog-agent
>>> 12/0= 7/11 12:40:27 INFO conf.FlumeConfiguration: Post-validation flume
>&g= t;> configuration contains configuration =A0for agents: [syslog-agent] >>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationP= rovider:
>>> Creating channels
>>> 12/07/11 12:40:2= 7 INFO properties.PropertiesFileConfigurationProvider:
>>> crea= ted channel MemoryChannel-1
>>> 12/07/11 12:40:27 INFO sink.DefaultSinkFactory: Creating insta= nce of sink
>>> Console typelogger
>>> 12/07/11 12:= 40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting
>> new<= br> >>> configuration:{ sourceRunners:{Syslog=3DEventDrivenSourceRunne= r: {
>>> source:org.apache.flume.source.SyslogTcpSource@110c424= }}
>>> sinkRunners:{Console=3DSinkRunner: {
>>> po= licy:org.apache.flume.sink.DefaultSinkProcessor@1bd2664 counterGroup:{
>>> name:null counters:{} } }}
>>> channels:{MemoryCha= nnel-1=3Dorg.apache.flume.channel.MemoryChannel@1238bd2}
>> }
&= gt;>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: S= tarting
>>> Channel MemoryChannel-1
>>> 12/07/11 12:40:27 INFO= nodemanager.DefaultLogicalNodeManager: Starting
>> Sink
>&g= t;> Console
>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLo= gicalNodeManager: Starting
>>> Source Syslog
>>> 12/07/11 12:40:28 INFO source.Sy= slogTcpSource: Syslog TCP Source
>> starting...
>>>>>> Please let me know, if i am doing anything wrong?
>>= >
>>> Thanks,
>>> Prabhu
>>>
>>>=
>>>
>>> On Wed, Jul 11, 2012 at 11:48 AM, alo alt = <wget.null@gmail.com> wrot= e:
>>>
>>>> Hi,
>>>>
>>>>= ; what exactly is the issue? Error messages, things like that.
>>&= gt;>
>>>> You posted only debugging and exec logs, thats = pretty normal. For a
>> first
>>>> short overview about flume take a look i= nto:
>>>>
>>>> https://cwiki.apache.or= g/FLUME/getting-started.html
>>>>
>>>>
>> http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.= 0b2/FlumeUserGuide.html
>>>> http://mapredit.blogspot.de/2012/03/flumeng-= evolution.html
>>>>
>>>> regards,
>= >>> Alex
>>>>
>>>> On Jul 10, 2012, at 3:09 PM, prabhu k = wrote:
>>>>
>>>>> Hi,
>>>>&= gt;
>>>>> While i am executing flume-ng, i am getting fol= lowing out put on my
>>>>> terminal, it seems got struck.
>>>>>=
>>>>> root@md-trngpoc1
>>>> :/usr/local/f= lume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
>>>>> bin= /flume-ng agent -n agent1 -c conf -f conf/agent1.conf
>>>>> Info: Sourcing environment configuration script
>= ;>>>>
>>>>
>> /usr/local/flume_dir/flum= e/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env.sh
>>>>>= Info: Including Hadoop libraries found via
>>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS acc= ess
>>>>> Info: Excluding
>>>>> /usr/lo= cal/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from
>>&g= t;>> classpath
>>>>> Info: Excluding
>>>>> /usr/local/had= oop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar
>> from
&= gt;>>>> classpath
>>>>> + exec /usr/lib/jvm/j= ava-6-sun-1.6.0.26/jre/bin/java -Xms100m -Xmx200m
>>>> -cp
>>>>>
>>>>
>>= ; '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr= /local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flu= me_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/had= oop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop= _dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-= 1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/loca= l/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_di= r/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop= /libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/li= bexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/= libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commo= ns-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons= -daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-diges= ter-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:= /usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/u= sr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/had= oop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/= libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/l= ibexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-= 3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-sched= uler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairsched= uler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-= 1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/= usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/lo= cal/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoo= p_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libex= ec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/j= ersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-serve= r-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_= dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoo= p/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../li= b/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/= usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../l= ib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib= /xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.= 1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar&#= 39;
>>>>>
>>>>
>> -Djava.library.path=3D= /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/h= adoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
>>>>>= ; org.apache.flume.node.Application -n agent1 -f conf/agent1.conf
>>>>>
>>>>> in execution summary:
>&= gt;>>> ----------------------------------------
>>>>= ;> Info: Including Hadoop libraries found via
>>>>> (/= usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
>>>>>
>>>>> hadoop lib path pointing to /u= sr/local/hadoop_dir/hadoop/bin/hadoop
>>>>> directory ins= ted of /usr/local/hadoop_dir/hadoop/lib path.
>>>>>
>>>>> Here my questions are;
>>>>>
>= >>>> 1. in flume-ng script anything changes need in add_hadoop_= paths
>> function.
>>>>> 2. anything wrong in fl= ume-env.sh script
>>>>> 3. anything wrong in agent1.conf file.
>>>= >>
>>>>> Please find the attached configuaration de= tails and let me know any
>> more
>>>>> details.=
>>>>>
>>>>> Could you pleae suggest me on = this issue.
>>>>>
>>>>> Thanks,
>= >>>> Prabhu
>>>>> <flume_config.txt> >>>>
>>>>
>>>> --
>>>= > Alexander Alten-Lorenz
>>>> http://mapredit.blogspot.com
>&g= t;>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>>>
>>>>
>>
>>
>> --=
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>&g= t; German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>


--
Alexander Alten-Lorenz
http://mapredit.blogspot.c= om
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


--f46d04479f9198a59604c48a9303--