flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@gmail.com>
Subject Re: Flume - ng running issue
Date Wed, 11 Jul 2012 09:44:54 GMT

example:

>>>> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
>>>> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo

echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
nc -v 10.10.10.10 5140 < /tmp/foo

10.10.10.10 is the IP who the flume agent is waiting for a connection.

man nc will help too.


On Jul 11, 2012, at 11:36 AM, prabhu k wrote:

> I did not get you, could you please elaborate bit more?
> 
> you should replace FLUME SYSLOG AGENT IP with the ip of the agent!
> 
> where i can add the FLUME SYSLOG AGENT IP ?
> 
> On Wed, Jul 11, 2012 at 2:33 PM, alo alt <wget.null@gmail.com> wrote:
> 
>> you should replace FLUME SYSLOG AGENT IP with the ip of the agent!
>> 
>> 
>> On Jul 11, 2012, at 11:01 AM, prabhu k wrote:
>> 
>>> Hi Alex,
>>> 
>>> I have executed the below commands, i am getting following output:
>>> 
>>> root@md-trngpoc1
>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
>>> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo
>>> nc: port range not valid
>>> root@md-trngpoc1:/tmp# more foo
>>> <13>Jun 20 12:12:12 host foo[345]: a syslog message
>>> 
>>> Please suggest any idea on this issue.
>>> 
>>> On Wed, Jul 11, 2012 at 2:15 PM, alo alt <wget.null@gmail.com> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> there stuck nothing, flume is waiting for a event. Point a syslog server
>>>> to the configured port or use the netcat example to see if the event can
>>>> delivered:
>>>> 
>>>> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
>>>> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo
>>>> 
>>>> - Alex
>>>> 
>>>> 
>>>> On Jul 11, 2012, at 10:30 AM, prabhu k wrote:
>>>> 
>>>>> Hi Alex,
>>>>> 
>>>>> Thanks for the response.
>>>>> 
>>>>> I have followed this link
>>>>> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html
>>>>> and configured the syslog-agent.cnf and executed by using
>>>>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf
>>>>> 
>>>>> but, script running from past 45 mints, it seems struck up flume-ng
>>>> script.
>>>>> 
>>>>> Here is the syslog-agent.cnf file.
>>>>> 
>>>>> syslog-agent.cnf
>>>>> =====================
>>>>> syslog-agent.sources = Syslog
>>>>> syslog-agent.channels = MemoryChannel-1
>>>>> syslog-agent.sinks = Console
>>>>> 
>>>>> syslog-agent.sources.Syslog.type = syslogTcp
>>>>> syslog-agent.sources.Syslog.port = 5140
>>>>> 
>>>>> syslog-agent.sources.Syslog.channels = MemoryChannel-1
>>>>> syslog-agent.sinks.Console.channel = MemoryChannel-1
>>>>> 
>>>>> syslog-agent.sinks.Console.type = logger
>>>>> syslog-agent.channels.MemoryChannel-1.type = memory
>>>>> 
>>>>> Execution log:
>>>>> ===================
>>>>> root@md-trngpoc1
>>>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
>>>>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf
>>>>> Warning: No configuration directory set! Use --conf <dir> to override.
>>>>> Info: Including Hadoop libraries found via
>>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
>>>>> Info: Excluding
>>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from
>>>>> classpath
>>>>> Info: Excluding
>>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar
>> from
>>>>> classpath
>>>>> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp
>>>>> 
>>>> 
>> '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
>>>>> 
>>>> 
>> -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
>>>>> org.apache.flume.node.Application -n syslog-agent -f
>>>> conf/syslog-agent.cnf
>>>>> 12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting
>> lifecycle
>>>>> supervisor 1
>>>>> 12/07/11 12:40:27 INFO node.FlumeNode: Flume node starting -
>> syslog-agent
>>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Node
>>>> manager
>>>>> starting
>>>>> 12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting
>> lifecycle
>>>>> supervisor 11
>>>>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider:
>>>>> Configuration provider starting
>>>>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider:
>>>>> Reloading configuration file:conf/syslog-agent.cnf
>>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console
>>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console
>>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Added sinks: Console
>>>> Agent:
>>>>> syslog-agent
>>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Post-validation flume
>>>>> configuration contains configuration  for agents: [syslog-agent]
>>>>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider:
>>>>> Creating channels
>>>>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider:
>>>>> created channel MemoryChannel-1
>>>>> 12/07/11 12:40:27 INFO sink.DefaultSinkFactory: Creating instance of
>> sink
>>>>> Console typelogger
>>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting
>>>> new
>>>>> configuration:{ sourceRunners:{Syslog=EventDrivenSourceRunner: {
>>>>> source:org.apache.flume.source.SyslogTcpSource@110c424 }}
>>>>> sinkRunners:{Console=SinkRunner: {
>>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@1bd2664counterGroup:{
>>>>> name:null counters:{} } }}
>>>>> 
>> channels:{MemoryChannel-1=org.apache.flume.channel.MemoryChannel@1238bd2}
>>>> }
>>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting
>>>>> Channel MemoryChannel-1
>>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting
>>>> Sink
>>>>> Console
>>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting
>>>>> Source Syslog
>>>>> 12/07/11 12:40:28 INFO source.SyslogTcpSource: Syslog TCP Source
>>>> starting...
>>>>> 
>>>>> Please let me know, if i am doing anything wrong?
>>>>> 
>>>>> Thanks,
>>>>> Prabhu
>>>>> 
>>>>> 
>>>>> 
>>>>> On Wed, Jul 11, 2012 at 11:48 AM, alo alt <wget.null@gmail.com>
wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> what exactly is the issue? Error messages, things like that.
>>>>>> 
>>>>>> You posted only debugging and exec logs, thats pretty normal. For
a
>>>> first
>>>>>> short overview about flume take a look into:
>>>>>> 
>>>>>> https://cwiki.apache.org/FLUME/getting-started.html
>>>>>> 
>>>>>> 
>>>> 
>> http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html
>>>>>> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html
>>>>>> 
>>>>>> regards,
>>>>>> Alex
>>>>>> 
>>>>>> On Jul 10, 2012, at 3:09 PM, prabhu k wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> While i am executing flume-ng, i am getting following out put
on my
>>>>>>> terminal, it seems got struck.
>>>>>>> 
>>>>>>> root@md-trngpoc1
>>>>>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
>>>>>>> bin/flume-ng agent -n agent1 -c conf -f conf/agent1.conf
>>>>>>> Info: Sourcing environment configuration script
>>>>>>> 
>>>>>> 
>>>> 
>> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env.sh
>>>>>>> Info: Including Hadoop libraries found via
>>>>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
>>>>>>> Info: Excluding
>>>>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
from
>>>>>>> classpath
>>>>>>> Info: Excluding
>>>>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar
>>>> from
>>>>>>> classpath
>>>>>>> + exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m
>> -Xmx200m
>>>>>> -cp
>>>>>>> 
>>>>>> 
>>>> 
>> '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
>>>>>>> 
>>>>>> 
>>>> 
>> -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
>>>>>>> org.apache.flume.node.Application -n agent1 -f conf/agent1.conf
>>>>>>> 
>>>>>>> in execution summary:
>>>>>>> ----------------------------------------
>>>>>>> Info: Including Hadoop libraries found via
>>>>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
>>>>>>> 
>>>>>>> hadoop lib path pointing to /usr/local/hadoop_dir/hadoop/bin/hadoop
>>>>>>> directory insted of /usr/local/hadoop_dir/hadoop/lib path.
>>>>>>> 
>>>>>>> Here my questions are;
>>>>>>> 
>>>>>>> 1. in flume-ng script anything changes need in add_hadoop_paths
>>>> function.
>>>>>>> 2. anything wrong in flume-env.sh script
>>>>>>> 3. anything wrong in agent1.conf file.
>>>>>>> 
>>>>>>> Please find the attached configuaration details and let me know
any
>>>> more
>>>>>>> details.
>>>>>>> 
>>>>>>> Could you pleae suggest me on this issue.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Prabhu
>>>>>>> <flume_config.txt>
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Alexander Alten-Lorenz
>>>>>> http://mapredit.blogspot.com
>>>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alexander Alten-Lorenz
>>>> http://mapredit.blogspot.com
>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>>> 
>>>> 
>> 
>> 
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>> 
>> 


--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


Mime
View raw message