Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1DF70D896 for ; Thu, 12 Jul 2012 09:41:41 +0000 (UTC) Received: (qmail 3564 invoked by uid 500); 12 Jul 2012 09:41:40 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 3496 invoked by uid 500); 12 Jul 2012 09:41:40 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 3456 invoked by uid 99); 12 Jul 2012 09:41:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Jul 2012 09:41:39 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of prabhu.flume@gmail.com designates 209.85.217.179 as permitted sender) Received: from [209.85.217.179] (HELO mail-lb0-f179.google.com) (209.85.217.179) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Jul 2012 09:41:31 +0000 Received: by lban1 with SMTP id n1so2702046lba.38 for ; Thu, 12 Jul 2012 02:41:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=cyvSFJhMyhcCSG7AfvJQKd+cEyWMTm0O7H6YTLEMtGU=; b=XV170jobcXQxOHO2PNVsFZb1w2E2R2HOcGtQgtB0XPx+xXh5vqjD3NZifvF8Um5s8+ OSWeWSFyJPFNxBTGaMWcQSsQY50jOe7mxgxNUvg3PtSBMETA0USi0HoRu0eeg3kkw0St 5bYh0JW5SMKIrfjg0Nnd5mSLzG99WdSH1eSQgfUKoV6xY5UMfqJ2Mod+datYE8VwXwJz MjoG8J84Zi+JDXQVexIqrlOm+TaDS2rqOt/uh1Nreqv05DrJcrsbhE8VT0yygJXUL4VZ zXrAq6EeHLOwzDkVW37llqCLlQPwbyYljLvYmKR5KaI0dKaCI6XLgFRjPxgJf6iezmgJ StTA== MIME-Version: 1.0 Received: by 10.112.42.41 with SMTP id k9mr739224lbl.90.1342086070867; Thu, 12 Jul 2012 02:41:10 -0700 (PDT) Received: by 10.112.60.98 with HTTP; Thu, 12 Jul 2012 02:41:10 -0700 (PDT) In-Reply-To: References: <4FFE5809.3080003@cyberagent.co.jp> Date: Thu, 12 Jul 2012 15:11:10 +0530 Message-ID: Subject: Re: using tail to HDFS sink From: prabhu k To: user@flume.apache.org Content-Type: multipart/alternative; boundary=e0cb4efe33a48317e704c49ec3c1 --e0cb4efe33a48317e704c49ec3c1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable can you please any one respond on the below issue? On Thu, Jul 12, 2012 at 12:57 PM, prabhu k wrote: > As per mohammad suggestion i have executed like below. > > root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSH= OT# > bin/flume-ng agent -n agent2 -c /conf -f conf/agent2.conf > > Info: Including Hadoop libraries found via > (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access > Info: Excluding > /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from > classpath > Info: Excluding > /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from > classpath > + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp > '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/= usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/h= adoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr= /local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/..= /hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.= jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/lo= cal/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_= dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoo= p_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoo= p/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec= /../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/= ../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/.= ./lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/= commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient= -3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/= usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoo= p_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop= _dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoo= p/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec= /../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-c= apacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/had= oop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/had= oop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1= .8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.= 8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.= 8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.ja= r:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/us= r/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir= /hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/lib= exec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib= /jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0= .6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/had= oop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/= libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kf= s-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/u= sr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop= /libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/l= ibexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/j= sp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-= api-2.1.jar' > -Djava.library.path=3D/usr/local/flume_dir/flume/flume-1.2.0-incubating-S= NAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-3= 2 > org.apache.flume.node.Application -n agent2 -f conf/agent2.conf > 12/07/12 12:48:45 ERROR node.Application: The specified configuration fil= e > does not exist: > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/agent2.co= nf > root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSH= OT# > > > > getting error; ERROR node.Application: The specified configuration file > does not exist: > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/agent2.co= nf > > my configuaration file is /conf/flume.conf, this is reason getting error= . > > and i have tryied following commands > > bin/flume-ng agent -n agent2 -c /conf -f conf/flume.conf -----flume.log > file not genrated and sample file not able to hdfs sink. > > root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSH= OT# > bin/flume-ng agent -n agent2 -c /conf -f conf/flume.conf > > Info: Including Hadoop libraries found via > (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access > Info: Excluding > /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from > classpath > Info: Excluding > /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from > classpath > + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp > '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/= usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/h= adoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr= /local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/..= /hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.= jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/lo= cal/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_= dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoo= p_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoo= p/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec= /../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/= ../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/.= ./lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/= commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient= -3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/= usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoo= p_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop= _dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoo= p/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec= /../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-c= apacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/had= oop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/had= oop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1= .8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.= 8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.= 8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.ja= r:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/us= r/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir= /hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/lib= exec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib= /jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0= .6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/had= oop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/= libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kf= s-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/u= sr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop= /libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/l= ibexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/j= sp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-= api-2.1.jar' > -Djava.library.path=3D/usr/local/flume_dir/flume/flume-1.2.0-incubating-S= NAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-3= 2 > org.apache.flume.node.Application -n agent2 -f conf/flume.conf > 12/07/12 12:43:25 INFO lifecycle.LifecycleSupervisor: Starting lifecycle > supervisor 1 > 12/07/12 12:43:25 INFO node.FlumeNode: Flume node starting - agent2 > 12/07/12 12:43:25 INFO nodemanager.DefaultLogicalNodeManager: Node manage= r > starting > 12/07/12 12:43:25 INFO properties.PropertiesFileConfigurationProvider: > Configuration provider starting > 12/07/12 12:43:25 INFO lifecycle.LifecycleSupervisor: Starting lifecycle > supervisor 10 > 12/07/12 12:43:25 INFO properties.PropertiesFileConfigurationProvider: > Reloading configuration file:conf/flume.conf > 12/07/12 12:43:25 INFO conf.FlumeConfiguration: Processing:HDFS > 12/07/12 12:43:25 INFO conf.FlumeConfiguration: Processing:HDFS > 12/07/12 12:43:25 INFO conf.FlumeConfiguration: Processing:HDFS > 12/07/12 12:43:25 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: > agent2 > 12/07/12 12:43:25 INFO conf.FlumeConfiguration: Processing:HDFS > 12/07/12 12:43:25 INFO conf.FlumeConfiguration: Post-validation flume > configuration contains configuration for agents: [agent2] > 12/07/12 12:43:25 INFO properties.PropertiesFileConfigurationProvider: > Creating channels > 12/07/12 12:43:25 INFO properties.PropertiesFileConfigurationProvider: > created channel memoryChannel > 12/07/12 12:43:25 INFO sink.DefaultSinkFactory: Creating instance of sink > HDFS typehdfs > > > > bin/flume-ng agent --conf conf/ -f conf/flume.conf -n agent2 -- flume.log > file created and sample file not able to hdfs sink. > > root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSH= OT# > bin/flume-ng agent --conf conf/ -f conf/flume.conf -n agent2 > > Info: Sourcing environment configuration script > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env= .sh > Info: Including Hadoop libraries found via > (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access > Info: Excluding > /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from > classpath > Info: Excluding > /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from > classpath > + exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m -Xmx200m -c= p > '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/loc= al/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_d= ir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/= libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir= /hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.= 3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/ha= doop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/lib= exec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libe= xec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/= commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-c= ollections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-co= nfiguration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-dae= mon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-= 1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/l= ocal/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_= dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop= /libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libe= xec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libex= ec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/= commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.= 1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler= -1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler= -1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.= 3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/loca= l/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_di= r/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/.= ./lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jerse= y-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.= 8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/loc= al/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/= hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/li= bexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/ju= nit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_d= ir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop= /libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/s= ervlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xml= enc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.ja= r:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' > -Djava.library.path=3D/usr/local/flume_dir/flume/flume-1.2.0-incubating-S= NAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-3= 2 > org.apache.flume.node.Application -f conf/flume.conf -n agent2 > > flume.log > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSH= OT# > more flume.log > 2012-07-12 12:46:24,415 INFO lifecycle.LifecycleSupervisor: Starting > lifecycle supervisor 1 > 2012-07-12 12:46:24,416 INFO node.FlumeNode: Flume node starting - agent2 > 2012-07-12 12:46:24,418 INFO nodemanager.DefaultLogicalNodeManager: Node > manager starting > 2012-07-12 12:46:24,418 INFO lifecycle.LifecycleSupervisor: Starting > lifecycle supervisor 10 > 2012-07-12 12:46:24,418 INFO > properties.PropertiesFileConfigurationProvider: Configuration provider > starting > 2012-07-12 12:46:24,420 INFO > properties.PropertiesFileConfigurationProvider: Reloading configuration > file:conf/flume.conf > 2012-07-12 12:46:24,426 INFO conf.FlumeConfiguration: Processing:HDFS > 2012-07-12 12:46:24,427 INFO conf.FlumeConfiguration: Processing:HDFS > 2012-07-12 12:46:24,428 INFO conf.FlumeConfiguration: Processing:HDFS > 2012-07-12 12:46:24,428 INFO conf.FlumeConfiguration: Added sinks: HDFS > Agent: agent2 > 2012-07-12 12:46:24,428 INFO conf.FlumeConfiguration: Processing:HDFS > 2012-07-12 12:46:24,443 INFO conf.FlumeConfiguration: Post-validation > flume configuration contains configuration for agents: [agent2] > 2012-07-12 12:46:24,443 INFO > properties.PropertiesFileConfigurationProvider: Creating channels > 2012-07-12 12:46:24,447 INFO > properties.PropertiesFileConfigurationProvider: created channel > memoryChannel > 2012-07-12 12:46:24,455 INFO sink.DefaultSinkFactory: Creating instance o= f > sink HDFS typehdfs > > > Please suggest me on this issue. > > Thanks, > Prabhu. > > > On Thu, Jul 12, 2012 at 10:22 AM, Juhani Connolly < > juhani_connolly@cyberagent.co.jp> wrote: > >> Hello Prabhu, >> >> You should have a look at your logs. Unless you've changed the settings, >> they'll be in flume.log in whatever directory you executed from. >> >> It looks to me like you are trying to use a header(%{host}) in the path, >> but you are not providing that header. If you want to use it, you should >> try using the host interceptor(which is better documented in the most >> recent revisions). There may be other issues too, but it's hard to tell >> without seeing the logs. Check them out and see if there's any exception= s >> popping up >> >> >> On 07/12/2012 03:19 AM, prabhu k wrote: >> >> Hi, >> >> While i am trying to load sample text file to HDFS sink, after executing >> bin/flume-ng node --conf conf/ -f conf/flume.conf -n agent2 >> command, Am unable to view the file in hdfs. >> >> i have executed like below. >> >> >> root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPS= HOT#bin/flume-ng node --conf conf/ -f conf/flume.conf -n agent2 >> Warning: The "node" command is deprecated. Please use "agent" instead. >> >> Info: Sourcing environment configuration script >> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-en= v.sh >> Info: Including Hadoop libraries found via >> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access >> Info: Excluding >> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from >> classpath >> Info: Excluding >> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from >> classpath >> + exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m -Xmx200m >> -cp >> '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/lo= cal/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_= dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop= /libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_di= r/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0= .3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/h= adoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/h= adoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/li= bexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libex= ec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/lib= exec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib= /commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-= collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-c= onfiguratio! >> n-1.6.jar: >> /usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/us= r/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/loca= l/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir= /hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/h= adoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/c= ommons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons= -math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1= .jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/l= ocal/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/l= ocal/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/= hadoop_dir/hadoop! >> /libexec/. >> ./lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/ja= ckson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackso= n-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-c= ompiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runti= me-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hado= op/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty= -6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.j= ar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/h= adoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/= libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/lo= g4j-1.2.15.jar:/u! >> sr/local/h >> adoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_= dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/= ../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/= jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1= .jar' >> -Djava.library.path=3D/usr/local/flume_dir/flume/flume-1.2.0-incubating-= SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-= 32 >> org.apache.flume.node.Application -f conf/flume.conf -n agent2 >> >> >> flume.conf >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> agent2.sources =3D tail >> agent2.channels =3D memoryChannel >> agent2.sinks =3D HDFS >> agent2.sources.tail.type =3D exec >> agent2.sources.tail.command =3D tail -F >> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.tx= t >> agent2.sources.tail.channels =3D memoryChannel >> agent2.sinks.HDFS.channel =3D memoryChannel >> agent2.sinks.HDFS.type =3D hdfs >> agent2.sinks.HDFS.hdfs.path =3D hdfs:// >> 10.5.114.110:54310/user/flume/'%{host}' >> agent2.sinks.HDFS.hdfs.file.Type =3D DataStream >> agent2.channels.memoryChannel.type =3D memory >> >> >> Please help me, if i have missed anything? >> Thanks, >> Prabhu >> >> >> >> > --e0cb4efe33a48317e704c49ec3c1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable can you please any one respond on the below issue?

On Thu, Jul 12, 2012 at 12:57 PM, prabhu k <prabhu.flume@gmail.com> wrote:

As per mohammad suggestion i have executed like below.

root@md-trngpoc1:/usr/local/flume_= dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng agent -n agent2= -c /conf -f conf/agent2.conf=20


Info: Including Hadoop libraries found via (/usr/loca= l/hadoop_dir/hadoop/bin/hadoop) for HDFS access
Info: Excluding /usr/loc= al/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1= .4.3.jar from classpath
+ exec /usr/lib/jvm/java-6-sun/bin/java -X= mx20m -cp '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAP= SHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:= /usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/to= ols.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoo= p/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5= .jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/us= r/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/l= ocal/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/had= oop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hado= op/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/lib= exec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/.= ./lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-= io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar= :/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson= -core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-map= per-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compil= er-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.= 5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/loca= l/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_di= r/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/li= bexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.= 26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/u= sr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop= _dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libex= ec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1= .2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar= :/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoo= p_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_= dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libe= xec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/= jsp-2.1/jsp-api-2.1.jar' -Djava.library.path=3D/usr/local/flume_dir/flu= me/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec= /../lib/native/Linux-i386-32 org.apache.flume.node.Application -n agent2 -f= conf/agent2.conf
12/07/12 12:48:45 ERROR node.Application: The specified configuration file = does not exist: /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/= conf/agent2.conf
root@md-trngpoc1:= /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#=20


getting error; ERROR node.Application: The specified configuration f= ile does not exist: /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPS= HOT/conf/agent2.conf

my configuaration file is=A0 /conf/flume.conf, this is reason getting er= ror.

and i have tryied following commands

bin/flume-ng agent -n agent2 -c /conf -f conf/flume.conf -----flume.log = file not genrated and sample file not able to hdfs sink.

root@md-trngpoc1:/usr/local/flume_= dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng agent -n agent2= -c /conf -f conf/flume.conf=20


Info: Including Hadoop libraries found via (/usr/loca= l/hadoop_dir/hadoop/bin/hadoop) for HDFS access
Info: Excluding /usr/loc= al/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1= .4.3.jar from classpath
+ exec /usr/lib/jvm/java-6-sun/bin/java -X= mx20m -cp '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAP= SHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:= /usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/to= ols.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoo= p/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5= .jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/us= r/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/l= ocal/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/had= oop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hado= op/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/lib= exec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/.= ./lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-= io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar= :/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson= -core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-map= per-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compil= er-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.= 5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/loca= l/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_di= r/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/li= bexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../= lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.= 26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/u= sr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop= _dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libex= ec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1= .2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar= :/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoo= p_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_= dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libe= xec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/= jsp-2.1/jsp-api-2.1.jar' -Djava.library.path=3D/usr/local/flume_dir/flu= me/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec= /../lib/native/Linux-i386-32 org.apache.flume.node.Application -n agent2 -f= conf/flume.conf
12/07/12 12:43:25 INFO lifecycle.LifecycleSupervisor: Starting lifecycle su= pervisor 1
12/07/12 12:43:25 INFO node.FlumeNode: Flume node starting - = agent2
12/07/12 12:43:25 INFO nodemanager.DefaultLogicalNodeManager: Nod= e manager starting
12/07/12 12:43:25 INFO properties.PropertiesFileConfigurationProvider: Conf= iguration provider starting
12/07/12 12:43:25 INFO lifecycle.LifecycleSu= pervisor: Starting lifecycle supervisor 10
12/07/12 12:43:25 INFO proper= ties.PropertiesFileConfigurationProvider: Reloading configuration file:conf= /flume.conf
12/07/12 12:43:25 INFO conf.FlumeConfiguration: Processing:HDFS
12/07/12= 12:43:25 INFO conf.FlumeConfiguration: Processing:HDFS
12/07/12 12:43:2= 5 INFO conf.FlumeConfiguration: Processing:HDFS
12/07/12 12:43:25 INFO c= onf.FlumeConfiguration: Added sinks: HDFS Agent: agent2
12/07/12 12:43:25 INFO conf.FlumeConfiguration: Processing:HDFS
12/07/12= 12:43:25 INFO conf.FlumeConfiguration: Post-validation flume configuration= contains configuration=A0 for agents: [agent2]
12/07/12 12:43:25 INFO p= roperties.PropertiesFileConfigurationProvider: Creating channels
12/07/12 12:43:25 INFO properties.PropertiesFileConfigurationProvider: crea= ted channel memoryChannel
12/07/12 12:43:25 INFO sink.DefaultSinkFactory= : Creating instance of sink HDFS typehdfs=20

=A0

bin/flume-ng agent --conf conf/ -f conf/flume.conf -n agent2 -- flume.lo= g file created and sample file not able to hdfs sink.

root@md-trngpoc1:/usr/local/flume_= dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng agent --conf co= nf/ -f conf/flume.conf -n agent2=20


Info: Sourcing environment configuration script /usr/= local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env.sh
= Info: Including Hadoop libraries found via (/usr/local/hadoop_dir/hadoop/bi= n/hadoop) for HDFS access
Info: Excluding /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3= .jar from classpath
Info: Excluding /usr/local/hadoop_dir/hadoop/libexec= /../lib/slf4j-log4j12-1.4.3.jar from classpath
+ exec /usr/lib/jvm= /java-6-sun-1.6.0.26/jre/bin/java -Xms100m -Xmx200m -cp '/usr/local/flu= me_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/flume_dir/flum= e/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.= 2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/= usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/.= .:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/l= ibexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../l= ib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commo= ns-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-= beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4= .jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.= jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.j= ar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/us= r/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/loca= l/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir= /hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/h= adoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/c= ommons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons= -math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1= .jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/l= ocal/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/l= ocal/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/= hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hado= op/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop= /libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/lib= exec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/= ../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core= -1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/u= sr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/had= oop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/..= /lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch= -0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/= hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec= /../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/o= ro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20= 081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr= /local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hado= op_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' -Djava.library.pa= th=3D/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/lo= cal/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32 org.apache.flume.= node.Application -f conf/flume.conf -n agent2=20

flume.log
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1= .2.0-incubating-SNAPSHOT# more flume.log
2012-07-12 12:46:24,415 INFO lifecycle.LifecycleSupervisor: Starting lifecy= cle supervisor 1
2012-07-12 12:46:24,416 INFO node.FlumeNode: Flume node= starting - agent2
2012-07-12 12:46:24,418 INFO nodemanager.DefaultLogic= alNodeManager: Node manager starting
2012-07-12 12:46:24,418 INFO lifecycle.LifecycleSupervisor: Starting lifecy= cle supervisor 10
2012-07-12 12:46:24,418 INFO properties.PropertiesFile= ConfigurationProvider: Configuration provider starting
2012-07-12 12:46:= 24,420 INFO properties.PropertiesFileConfigurationProvider: Reloading confi= guration file:conf/flume.conf
2012-07-12 12:46:24,426 INFO conf.FlumeConfiguration: Processing:HDFS
20= 12-07-12 12:46:24,427 INFO conf.FlumeConfiguration: Processing:HDFS
2012= -07-12 12:46:24,428 INFO conf.FlumeConfiguration: Processing:HDFS
2012-0= 7-12 12:46:24,428 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: ag= ent2
2012-07-12 12:46:24,428 INFO conf.FlumeConfiguration: Processing:HDFS
20= 12-07-12 12:46:24,443 INFO conf.FlumeConfiguration: Post-validation flume c= onfiguration contains configuration=A0 for agents: [agent2]
2012-07-12 1= 2:46:24,443 INFO properties.PropertiesFileConfigurationProvider: Creating c= hannels
2012-07-12 12:46:24,447 INFO properties.PropertiesFileConfigurationProvider= : created channel memoryChannel
2012-07-12 12:46:24,455 INFO sink.Defaul= tSinkFactory: Creating instance of sink HDFS typehdfs


Please sug= gest me on this issue.
=A0
Thanks,
Prabhu.
=A0
=A0
On Thu, Jul 12, 2012 at 10:22 AM, Juhani Connoll= y <juhani_connolly@cyberagent.co.jp> wrote:
Hello Prabhu,

You should have a look at your logs. Unless you&#= 39;ve changed the settings, they'll be in flume.log in whatever directo= ry you executed from.

It looks to me like you are trying to use a he= ader(%{host}) in the path, but you are not providing that header. If you wa= nt to use it, you should try using the host interceptor(which is better doc= umented in the most recent revisions). There may be other issues too, but i= t's hard to tell without seeing the logs. Check them out and see if the= re's any exceptions popping up=20


On 07/12/2012 03:19 AM, prabhu k wrote:
Hi,
=A0
While i am trying to load sample text file to HDFS sink,= after executing bin/flume-ng node --conf conf/ -f conf/flume.conf -n agent= 2
command, Am unable to view the file in hdfs.
=A0
i have executed= like below.
=A0
root@md-trngpoc1:/usr/local/f= lume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng node --con= f conf/ -f conf/flume.conf -n agent2
Warning: The "node" command is deprecated. Please use "agent= " instead.

Info: Sourcing environment configuration script /usr= /local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env.sh Info: Including Hadoop libraries found via (/usr/local/hadoop_dir/hadoop/bi= n/hadoop) for HDFS access
Info: Excluding /usr/local/hadoop_dir/hadoop/l= ibexec/../lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /usr/lo= cal/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from classpath=
+ exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m -Xmx200= m -cp '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:= /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local= /flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir= /hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/ha= doop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-c= ore-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/= local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoo= p_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoo= p/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/had= oop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec= /../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/c= ommons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/co= mmons-configuratio! n-1.6.jar: /usr/local/hadoop_dir/hadoop/libexec/../lib/= commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/common= s-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1= .0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1= .jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/lo= cal/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop= _dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/= hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/h= adoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libe= xec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../li= b/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacit= y-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fa= irscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-th= riftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop! /libexec/. ./lib/hsqldb-1.8.= 0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8= .jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.j= ar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/= usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/l= ocal/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/ha= doop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexe= c/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/je= rsey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.= 1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/loc= al/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop= _dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/lib= exec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0= .2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/u! s= r/local/h adoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/= hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop= /libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/l= ibexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/j= sp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-= api-2.1.jar' -Djava.library.path=3D/usr/local/flume_dir/flume/flume-1.2= .0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/nati= ve/Linux-i386-32 org.apache.flume.node.Application -f conf/flume.conf -n ag= ent2=20


flume.conf
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
agent2.sources =3D tail
agent2.channels =3D memoryChannel
agent2.= sinks =3D HDFS
agent2.sources.tail.type =3D exec
agent2.sources.tail.= command =3D tail -F /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPS= HOT/flume_test.txt
agent2.sources.tail.channels =3D memoryChannel
agent2.sinks.HDFS.channel= =3D memoryChannel
agent2.sinks.HDFS.type =3D hdfs
agent2.sinks.HDFS.= hdfs.path =3D hdfs://10.5.114.110:54310/user/flume/'%{host}&= #39;
agent2.sinks.HDFS.hdfs.file.Type =3D DataStream
agent2.channels.memoryCh= annel.type =3D memory


Please help me, if i have missed anything?=
Thanks,
Prabhu




--e0cb4efe33a48317e704c49ec3c1--