Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 936AA9184 for ; Thu, 4 Oct 2012 22:02:37 +0000 (UTC) Received: (qmail 53767 invoked by uid 500); 4 Oct 2012 22:02:37 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 53729 invoked by uid 500); 4 Oct 2012 22:02:37 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 53719 invoked by uid 99); 4 Oct 2012 22:02:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Oct 2012 22:02:37 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of hshreedharan@cloudera.com designates 209.85.220.51 as permitted sender) Received: from [209.85.220.51] (HELO mail-pa0-f51.google.com) (209.85.220.51) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Oct 2012 22:02:27 +0000 Received: by mail-pa0-f51.google.com with SMTP id kq12so962644pab.38 for ; Thu, 04 Oct 2012 15:02:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=date:from:to:message-id:in-reply-to:references:subject:x-mailer :mime-version:content-type:x-gm-message-state; bh=fK609/AmqhZYxrf8daXrOz6LA6qbZLeO0vYauNWdAhg=; b=EWJBgiMeo5Zz53Je3NWSpYW2uL27Xm9fLpIZyAbc+2i/eFFpn9ROTlwy+uiAW9VfDn OOwuPXHQVKBOvQyBaiZ3g6jkvvA66KfrFMK5OVIAeSBe0ZuCbVbmMCRg0SJxYja7h0lk Yb4u1+ThHgvej1aKcqEhg5v2gHImPjdaywAdoJItEkQyNH5zheYg9N8SMV9xvefvsl0r NW/3ncCt1BZovLQ3+g7kmMHujDOyI+tJWuDmN/SHkGXWE3AWEeYqXH69YQrgxlsc3PlW NLiG+Tp0BcgPkuBVJaiJY24eLwveILSVfpbv7jiabt2PSypFlL49I8cUb9uTAQ4eIVtt yTSg== Received: by 10.68.232.131 with SMTP id to3mr25749366pbc.58.1349388125887; Thu, 04 Oct 2012 15:02:05 -0700 (PDT) Received: from [172.22.1.152] ([65.50.196.130]) by mx.google.com with ESMTPS id te6sm4930673pbc.29.2012.10.04.15.02.04 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 04 Oct 2012 15:02:05 -0700 (PDT) Date: Thu, 4 Oct 2012 15:02:02 -0700 From: Hari Shreedharan To: user@flume.apache.org Message-ID: <01DF0EFF6FC24B039D43D11B304054A1@cloudera.com> In-Reply-To: <8AA8DD4A17F49645938C17986485E501015940AD@CORPUSMX30B.corp.emc.com> References: <8AA8DD4A17F49645938C17986485E501015940AD@CORPUSMX30B.corp.emc.com> Subject: Re: Flume Source and Sink in different hosts X-Mailer: sparrow 1.6.3 (build 1172) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="506e075a_2cb6a6a4_d4" X-Gm-Message-State: ALoCoQnW1FStBtwruGzrMFxPNAO8/rYDjJj+I9bIIjbkjKZD4KGifYPsKzppzWNeOuf3I7RjSNX8 --506e075a_2cb6a6a4_d4 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Can you send the logs also, of both agents=3F Does your Hbase cluster hav= e the said column family and table with that family=3F =20 Also are you sure the files are not getting rotated out. You should use t= ail -=46 so that your code works even with files getting rotated out. Hari -- =20 Hari Shreedharan On Thursday, October 4, 2012 at 2:53 PM, Kumar, Suresh wrote: > Hello: > =20 > I have just downloaded and build flume-ng (apache-flume-1.3.0-SNAPSHOT)= . > =20 > My goal is to collect log data from HostA (source) and send it to HostB= (sink), my initial test (sending /etc/passwd) =20 > from HostA to HostB worked fine, I was also able to load the passwd fil= e into my HBase in HostB. > =20 > Now, I want to load a continuous stream of log data (using tail =E2=80=93= f), but I was not able to replicate the above process. > =46lume just started fine in HostA, but I do not see any data being rec= eived by HostB or in my HBase. > =20 > What is wrong with my configuration=3F > =20 > Thanks, > Suresh > =20 > Here is my flume.conf in HostA > =20 > agent3.sources =3D tail > agent3.channels =3D MemoryChannel-1 > agent3.sinks =3D avro-sink > =20 > =23 Define source flow > agent3.sources.tail.type =3D exec > agent3.sources.tail.command =3D tail -f /var/log/auth.log > agent3.sources.tail.channels =3D MemoryChannel-1 > =20 > =23 What kind of channel > agent3.channels.MemoryChannel-1.type =3D memory > =20 > =23 avro sink properties > agent3.sinks.avro-sink.type =3D avro > agent3.sinks.avro-sink.channel =3D MemoryChannel-1 > agent3.sinks.avro-sink.hostname =3D hostb > agent3.sinks.avro-sink.port =3D 41414 > =20 > Here is my flume.conf in HostB > =20 > =23 Define a memory channel called ch1 on agent1 > agent1.channels.ch1.type =3D memory > =20 > =23 Define an Avro source called avro-source1 on agent1 and tell it > =23 to bind to 0.0.0.0:41414. Connect it to channel ch1. > agent1.sources.avro-source1.channels =3D ch1 > agent1.sources.avro-source1.type =3D avro > agent1.sources.avro-source1.bind =3D 0.0.0.0 > agent1.sources.avro-source1.port =3D 41414 > =20 > =23 Define a logger sink that simply logs all events it receives > =23 and connect it to the other end of the same channel. > agent1.sinks.log-sink1.channel =3D ch1 > agent1.sinks.log-sink1.type =3D logger > =20 > =23 =46inally, now that we've defined all of our components, tell > =23 agent1 which ones we want to activate. > agent1.channels =3D ch1 > agent1.sources =3D avro-source1 > =23agent1.sources =3D avro-source1 > agent1.sinks =3D sink1 > =20 > agent1.sinks.sink1.type =3D org.apache.flume.sink.hbase.HBaseSink > agent1.sinks.sink1.channel =3D ch1 > agent1.sinks.sink1.table =3D flumedemo > agent1.sinks.sink1.column=46amily =3D testing > agent1.sinks.sink1.column =3D foo > agent1.sinks.sink1.serializer =3D org.apache.flume.sink.hbase.SimpleHba= seEventSerializer > agent1.sinks.sink1.serializer.payloadColumn =3D col1 > agent1.sinks.sink1.serializer.keyType =3D timestamp > agent1.sinks.sink1.serializer.rowPrefix =3D 1 > agent1.sinks.sink1.serializer.suffix =3D timestamp > agent1.sinks.sink1.serializer.payloadColumn =3D pcol > agent1.sinks.sink1.serializer.incrementColumn =3D icol > =20 > =20 > =20 > =20 --506e075a_2cb6a6a4_d4 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Can you send the= logs also, of both agents=3F Does your Hbase cluster have the said colum= n family and table with that family=3F

Also are you sure the files are not getting rotated out. You sh= ould use tail -=46 so that your code works even with files getting rotate= d out.

<= /div>

Hari
-- 
Hari Shreedharan

=20

On Thursday, October 4= , 2012 at 2:53 PM, Kumar, Suresh wrote:

<=21--=5Bif gte mso = 9=5D> <=21=5Bendif=5D--><=21--=5Bif gte mso 9=5D> <=21=5Bendif=5D-->

Hello:

 

I have just downloaded and build flume-ng= (apache-flume-1.3.0-SNAPSHOT).

 

My goal is to col= lect log data from HostA (source) and send it to HostB(sink), my initial = test (sending /etc/passwd)

= from HostA to HostB worked fine, I was also able to load the passwd file = into my HBase in HostB.

 

Now, I want to load a con= tinuous stream of log data (using tail =E2=80=93f), but I was not able to= replicate the above process.

=46lume just started fine in HostA, but I do not see any data being rece= ived by HostB or in my HBase.

 

What is wrong with m= y configuration=3F

&nbs= p;

Thanks,

Suresh

 

Here is my flume.= conf in HostA

 = ;

agent3.sources =3D tail

agent3.channels =3D MemoryChannel-1=

agent3.sinks =3D avro-sink<= o:p>

 

=23 Define source flow

agent3.sources.tail.type =3D exec

agent3.sources.tail.command =3D tail -f /var/log/a= uth.log

agent3.sources.tail.= channels =3D MemoryChannel-1

 

=23 What kind of cha= nnel

agent3.channels.MemoryC= hannel-1.type =3D memory

 

=23 avro sink properties=

agent3.sinks.avro-sink.type= =3D avro

agent3.sinks.avro-= sink.channel =3D MemoryChannel-1

agent3.sinks.avro-sink.hostname =3D hostb

agent3.sinks.avro-sink.port =3D 41414

 

Here is my flume.conf in HostB

 

=23 Define a memory channel called ch1 on agent1

agent1.channels.ch1.type =3D memory

 

=23 Define an Avro source called avro-source1 on agent1 and tell i= t

=23 to bind to 0.0.0.0:414= 14. Connect it to channel ch1.

agent1.sources.avro-source1.channels =3D ch1

agent1.sources.avro-source1.type =3D avro

=

agent1.sources.avro-source1.bind =3D 0.0.0= .0

agent1.sources.avro-sourc= e1.port =3D 41414

 = ;

=23 Define a logger sink that s= imply logs all events it receives

=23 and connect it to the other end of the same channel.=

agent1.sinks.log-sink1.channel =3D ch1=

agent1.sinks.log-sink1.type= =3D logger

 

=23 =46inally, now that we've defined= all of our components, tell

=23 agent1 which ones we want to activate.

agent1.channels =3D ch1

agent1.sources =3D avro-source1

=23agent1.sources =3D avro-source1

agent1.sinks =3D sink1

 

agen= t1.sinks.sink1.type =3D org.apache.flume.sink.hbase.HBaseSink<= /p>

agent1.sinks.sink1.channel =3D ch1=

agent1.sinks.sink1.table =3D flu= medemo

agent1.sinks.sink1.co= lumn=46amily =3D testing

age= nt1.sinks.sink1.column =3D foo

agent1.sinks.sink1.serializer =3D org.apache.flume.sink.hbase.SimpleHbas= eEventSerializer

agent1.sink= s.sink1.serializer.payloadColumn =3D col1

agent1.sinks.sink1.serializer.keyType =3D timestamp

agent1.sinks.sink1.serializer.rowPr= efix =3D 1

agent1.sinks.sink= 1.serializer.suffix =3D timestamp

agent1.sinks.sink1.serializer.payloadColumn =3D pcol

=

agent1.sinks.sink1.serializer.incrementCol= umn =3D icol

 

 =20 =20 =20 =20
=20

--506e075a_2cb6a6a4_d4--