Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F15A2E254 for ; Wed, 20 Feb 2013 01:45:33 +0000 (UTC) Received: (qmail 30506 invoked by uid 500); 20 Feb 2013 01:45:33 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 30472 invoked by uid 500); 20 Feb 2013 01:45:33 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 30461 invoked by uid 99); 20 Feb 2013 01:45:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Feb 2013 01:45:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jlord@cloudera.com designates 209.85.217.172 as permitted sender) Received: from [209.85.217.172] (HELO mail-lb0-f172.google.com) (209.85.217.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Feb 2013 01:45:26 +0000 Received: by mail-lb0-f172.google.com with SMTP id n8so5641014lbj.31 for ; Tue, 19 Feb 2013 17:45:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type:x-gm-message-state; bh=EavXH/+tRjHL4bpBxnQIZWRd6+lO2d1k6PDPkQoHFNk=; b=VDs7L1AuLNfiZ2394MSgk88iuze0QDHX9WjBql6irYWwMWKae2G/5jzarfUR+ErZra Yion2tYXanyGf4DeqgJxRcy2tfuk4/7DfZpCEok+pjqIMvsHlU3FioTAUBfUiUr4YIEz /lUNLiJtqFLFcJLIYJXkL2PL5YRl6rkTxD/YnE1anRFNmxyo/L5lC7iW3G9pAEnSH0v8 ovZrRaXioSx+9YsvyyX23Z4A2kzI8tQ7qMBoVV5VNog1QxRkTVKjRK6+pFm++CVCUEpq 9okdKmaaizq/WViYBfn1TCCG78Z+MWOMjC7m15kbUOQ71x3OsSOkx9i9xTpT5ta6W5oj 2k/w== MIME-Version: 1.0 X-Received: by 10.112.50.169 with SMTP id d9mr7915639lbo.57.1361324705960; Tue, 19 Feb 2013 17:45:05 -0800 (PST) Received: by 10.112.49.168 with HTTP; Tue, 19 Feb 2013 17:45:05 -0800 (PST) In-Reply-To: <64C8D4EE364ADF498AF23BC6A6619B9E4C5C1FE5@ex-nz1.globalbrain.net> References: <64C8D4EE364ADF498AF23BC6A6619B9E4C5C1EFD@ex-nz1.globalbrain.net> <64C8D4EE364ADF498AF23BC6A6619B9E4C5C1F85@ex-nz1.globalbrain.net> <5BCBB1BC0F5744CFB5731BFD97930C83@cloudera.com> <64C8D4EE364ADF498AF23BC6A6619B9E4C5C1F9D@ex-nz1.globalbrain.net> <64C8D4EE364ADF498AF23BC6A6619B9E4C5C1FE5@ex-nz1.globalbrain.net> Date: Tue, 19 Feb 2013 17:45:05 -0800 Message-ID: Subject: Re: Architecting Flume for failover From: Jeff Lord To: user@flume.apache.org Content-Type: multipart/alternative; boundary=f46d0401678d85fcac04d61e1bfa X-Gm-Message-State: ALoCoQklpwuQLYn9UzeoLdzlwVsCvfYBFqaz5SBhx2ryfIqotbuPnmzPA2o+Ty/LG/2Gy2ihyKry X-Virus-Checked: Checked by ClamAV on apache.org --f46d0401678d85fcac04d61e1bfa Content-Type: text/plain; charset=ISO-8859-1 Maybe post your entire config? On Tue, Feb 19, 2013 at 5:35 PM, Noel Duffy wrote: > In my tests I set up two Flume agents connected to two different HDFS > clusters. The configuration of both Flume agents is identical. They read > events from the same RabbitMQ server. In my test, both agent hosts wrote > the event to their respective HDFS servers using hdfsSink-2, but I expected > the failover sinkgroup configuration would mean only one host would write > the event. In other words, I thought that a failover sinkgroup could be > configured to have sinks on different hosts but that only one sink on one > host would actually write the event and that the other host would not do > anything. > > All the examples in the documentation have all sinks in a sinkgroup on a > single host. I want to have the sinks on different hosts. I've seen a > number of assertions online that this can be done, but so far, I've not > seen any examples of how to actually configure it. > > From: Jeff Lord [mailto:jlord@cloudera.com] > Sent: Wednesday, 20 February 2013 2:17 p.m. > To: user@flume.apache.org > Subject: Re: Architecting Flume for failover > > Noel, > > What test did you perform? > Did you stop sink-2? > Currently you have set a higher priority for sink-2 so it will be the > default sink so long as it is up and running. > > -Jeff > > http://flume.apache.org/FlumeUserGuide.html#failover-sink-processor > > On Tue, Feb 19, 2013 at 5:03 PM, Noel Duffy > wrote: > The ip addresses 10.20.30.81 and 10.20.30.119 are the addresses of the > Flume agents. The first agent is on 10.20.30.81, the second on > 10.20.30.119. The idea was to have two sinks on different hosts and to > configure Flume to failover to the second host if the first host should > disappear. Although the documentation does not say so explicitly, I have > read posts online which say that such a configuration is possible. I am > running Flume-ng 1.2.0. > > It may be that I am approaching this problem in the wrong way. We need to > have Flume reading events from RabbitMQ and writing them to HDFS. We want > to have two different hosts running Flume so that if one dies for any > reason, the other would take over and no events should be lost or delayed. > Later we may have more Flume hosts, depending on how well they cope with > the expected traffic, but for now two will suffice to prove the concept. A > load-balancing sink processor sounds like it might also be a solution, but > again, I do not see how to configure this to work across more than one host. > > > From: Hari Shreedharan [mailto:hshreedharan@cloudera.com] > Sent: Wednesday, 20 February 2013 1:31 p.m. > To: user@flume.apache.org > Subject: Re: Architecting Flume for failover > > Can you change the hdfs.path to hdfs://10.20.30.81/flume/localbrain-eventsand hdfs:// > 10.20.30.119/flume/localbrain-events on hdfsSink-1 and hdfsSink-2 > respectively (assuming those are your namenodes)? The "bind" configuration > param does not really exist for HDFS Sink (it is only for the IPC sources). > > > Thanks > Hari > > -- > Hari Shreedharan > > On Tuesday, February 19, 2013 at 4:05 PM, Noel Duffy wrote: > If I disable the agent.sinks line, both my sinks are disabled and nothing > gets written to HDFS. The status page no longer shows me any sinks. > > From: Yogi Nerella [mailto:ynerella999@gmail.com] > Sent: Wednesday, 20 February 2013 12:40 p.m. > To: user@flume.apache.org > Subject: Re: Architecting Flume for failover > > Hi Noel, > > May be you are specifying both sinkgroups and sinks. > > Can you try removing the sinks. > #agent.sinks = hdfsSink-1 hdfsSink-2 > > Yogi > > > On Tue, Feb 19, 2013 at 1:32 PM, Noel Duffy > wrote: > I have a Flume agent that pulls events from RabbitMQ and pushes them into > HDFS. So far so good, but now I want to have a second Flume agent on a > different host acting as a hot backup for the first agent such that the > loss of the first host running Flume would not cause any events to be lost. > In the testing I've done I've gotten two Flume agents on separate hosts to > read the same events from the RabbitMQ queue, but it's not clear to me how > to configure the sinks such that only one of the sinks actually does > something and the other does nothing. > > From reading the documentation, I supposed that a sinkgroup configured for > failover was what I needed, but the documentation examples only cover the > case where the sinks in a failover group are all on the same agent on the > same host. I've seen messages online which seem to say that sinks in a > sinkgroup can be on different hosts, but I can find no clear explanation of > how to configure such a sinkgroup. How would sinks on different hosts > communicate with one another? Would the sinks in the sinkgroup have to use > a JDBC channel? Would the sinks have to be non-terminal sinks, like Avro? > > In my testing I set up two agents on different hosts and configured a > sinkgroup containing two sinks, both HDFS sinks. > > agent.sinkgroups = sinkgroup1 > agent.sinkgroups.sinkgroup1.sinks = hdfsSink-1 hdfsSink-2 > agent.sinkgroups.sinkgroup1.processor.priority.hdfsSink-1 = 5 > agent.sinkgroups.sinkgroup1.processor.priority.hdfsSink-2 = 10 > agent.sinkgroups.sinkgroup1.processor.type=failover > > agent.sinks = hdfsSink-1 hdfsSink-2 > agent.sinks.hdfsSink-1.type = hdfs > agent.sinks.hdfsSink-1.bind = 10.20.30.81 > agent.sinks.hdfsSink-1.channel = fileChannel-1 > agent.sinks.hdfsSink-1.hdfs.path = /flume/localbrain-events > agent.sinks.hdfsSink-1.hdfs.filePrefix = lb-events > agent.sinks.hdfsSink-1.hdfs.round = false > agent.sinks.hdfsSink-1.hdfs.rollCount=50 > agent.sinks.hdfsSink-1.hdfs.fileType=SequenceFile > agent.sinks.hdfsSink-1.hdfs.writeFormat=Text > agent.sinks.hdfsSink-1.hdfs.codeC = lzo > agent.sinks.hdfsSink-1.hdfs.rollInterval=30 > agent.sinks.hdfsSink-1.hdfs.rollSize=0 > agent.sinks.hdfsSink-1.hdfs.batchSize=1 > > agent.sinks.hdfsSink-2.bind = 10.20.30.119 > agent.sinks.hdfsSink-2.type = hdfs > agent.sinks.hdfsSink-2.channel = fileChannel-1 > agent.sinks.hdfsSink-2.hdfs.path = /flume/localbrain-events > agent.sinks.hdfsSink-2.hdfs.filePrefix = lb-events > agent.sinks.hdfsSink-2.hdfs.round = false > agent.sinks.hdfsSink-2.hdfs.rollCount=50 > agent.sinks.hdfsSink-2.hdfs.fileType=SequenceFile > agent.sinks.hdfsSink-2.hdfs.writeFormat=Text > agent.sinks.hdfsSink-2.hdfs.codeC = lzo > agent.sinks.hdfsSink-2.hdfs.rollInterval=30 > agent.sinks.hdfsSink-2.hdfs.rollSize=0 > agent.sinks.hdfsSink-2.hdfs.batchSize=1 > > However, this does not achieve the failover I hoped for. The sink > hdfsSink-2 on both agents writes the events to HDFS. The agents are not > communicating, so the binding of the sink to an ip address is not doing > anything. > > --f46d0401678d85fcac04d61e1bfa Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Maybe post your entire config?


On Tue, Feb 19, 2013 at 5:35 PM, No= el Duffy <noel.duffy@sli-systems.com> wrote:
In my tests I set up two Flume agents connec= ted to two different HDFS clusters. The configuration of both Flume agents = is identical. They read events from the same RabbitMQ server. In my test, b= oth agent hosts wrote the event to their respective HDFS servers using hdfs= Sink-2, but I expected the failover sinkgroup configuration would mean only= one host would write the event. In other words, I thought that a failover = sinkgroup could be configured to have sinks on different hosts but that onl= y one sink on one host would actually write the event and that the other ho= st would not do anything.

All the examples in the documentation have all sinks in a sinkgroup on a si= ngle host. I want to have the sinks on different hosts. I've seen a num= ber of assertions online that this can be done, but so far, I've not se= en any examples of how to actually configure it.

From: Jeff Lord [mailto:jlord@clouder= a.com]
Sent: Wednesday, 20 February 2013 2:17 p.m.
To: user@flume.apache.org
Subject: Re: Architecting Flume for failover

Noel,

What test did you perform?
Did you stop sink-2?=A0
Currently you have set a higher priority for sink-2 so it will be the defau= lt sink so long as it is up and running.

-Jeff

http://flume.apache.org/FlumeUserGuide.html#failover= -sink-processor

On Tue, Feb 19, 2013 at 5:03 PM, Noel Duffy <noel.duffy@sli-systems.com> wrote:
The ip addresses 10.20.30.81 and 10.20.30.119 are the addresses of the Flum= e agents. The first agent is on 10.20.30.81, the second on 10.20.30.119. Th= e idea was to have two sinks on different hosts and to configure Flume to f= ailover to the second host if the first host should disappear. Although the= documentation does not say so explicitly, I have read posts online which s= ay that such a configuration is possible. I am running Flume-ng 1.2.0.

It may be that I am approaching this problem in the wrong way. We need to h= ave Flume reading events from RabbitMQ and writing them to HDFS. We want to= have two different hosts running Flume so that if one dies for any reason,= the other would take over and no events should be lost or delayed. Later w= e may have more Flume hosts, depending on how well they cope with the expec= ted traffic, but for now two will suffice to prove the concept. A load-bala= ncing sink processor sounds like it might also be a solution, but again, I = do not see how to configure this to work across more than one host.


From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Wednesday, 20 February 2013 1:31 p.m.
To: user@flume.apache.org
Subject: Re: Architecting Flume for failover

Can you change the hdfs.path to hdfs://10.20.30.81/flume/localbrain-events and hdfs://10.20.30.119/flume/localbrain-events on hdfsSink-1 and hdfs= Sink-2 respectively (assuming those are your namenodes)? The "bind&quo= t; configuration param does not really exist for HDFS Sink (it is only for = the IPC sources).=A0


Thanks
Hari

--=A0
Hari Shreedharan

On Tuesday, February 19, 2013 at 4:05 PM, Noel Duffy wrote:
If I disable the agent.sinks line, both my sinks are disabled and nothing g= ets written to HDFS. The status page no longer shows me any sinks.

From: Yogi Nerella [mailto:ynerell= a999@gmail.com]
Sent: Wednesday, 20 February 2013 12:40 p.m.
To: user@flume.apache.org
Subject: Re: Architecting Flume for failover

Hi Noel,

May be you are specifying =A0both sinkgroups and sinks. =A0

Can you try removing the sinks.
#agent.sinks =3D hdfsSink-1 hdfsSink-2

Yogi


On Tue, Feb 19, 2013 at 1:32 PM, Noel Duffy <noel.duffy@sli-systems.com> wrote:
I have a Flume agent that pulls events from RabbitMQ and pushes them into H= DFS. So far so good, but now I want to have a second Flume agent on a diffe= rent host acting as a hot backup for the first agent such that the loss of = the first host running Flume would not cause any events to be lost. In the = testing I've done I've gotten two Flume agents on separate hosts to= read the same events from the RabbitMQ queue, but it's not clear to me= how to configure the sinks such that only one of the sinks actually does s= omething and the other does nothing.

>From reading the documentation, I supposed that a sinkgroup configured for = failover was what I needed, but the documentation examples only cover the c= ase where the sinks in a failover group are all on the same agent on the sa= me host. I've seen messages online which seem to say that sinks in a si= nkgroup can be on different hosts, but I can find no clear explanation of h= ow to configure such a sinkgroup. How would sinks on different hosts commun= icate with one another? Would the sinks in the sinkgroup have to use a JDBC= channel? Would the sinks have to be non-terminal sinks, like Avro?

In my testing I set up two agents on different hosts and configured a sinkg= roup containing two sinks, both HDFS sinks.

agent.sinkgroups =3D sinkgroup1
agent.sinkgroups.sinkgroup1.sinks =3D hdfsSink-1 hdfsSink-2
agent.sinkgroups.sinkgroup1.processor.priority.hdfsSink-1 =3D 5
agent.sinkgroups.sinkgroup1.processor.priority.hdfsSink-2 =3D 10
agent.sinkgroups.sinkgroup1.processor.type=3Dfailover

agent.sinks =3D hdfsSink-1 hdfsSink-2
agent.sinks.hdfsSink-1.type =3D hdfs
agent.sinks.hdfsSink-1.bind =3D 10.20.30.81
agent.sinks.hdfsSink-1.channel =3D fileChannel-1
agent.sinks.hdfsSink-1.hdfs.path =3D /flume/localbrain-events
agent.sinks.hdfsSink-1.hdfs.filePrefix =3D lb-events
agent.sinks.hdfsSink-1.hdfs.round =3D false
agent.sinks.hdfsSink-1.hdfs.rollCount=3D50
agent.sinks.hdfsSink-1.hdfs.fileType=3DSequenceFile
agent.sinks.hdfsSink-1.hdfs.writeFormat=3DText
agent.sinks.hdfsSink-1.hdfs.codeC =3D lzo
agent.sinks.hdfsSink-1.hdfs.rollInterval=3D30
agent.sinks.hdfsSink-1.hdfs.rollSize=3D0
agent.sinks.hdfsSink-1.hdfs.batchSize=3D1

agent.sinks.hdfsSink-2.bind =3D 10.20.30.119
agent.sinks.hdfsSink-2.type =3D hdfs
agent.sinks.hdfsSink-2.channel =3D fileChannel-1
agent.sinks.hdfsSink-2.hdfs.path =3D /flume/localbrain-events
agent.sinks.hdfsSink-2.hdfs.filePrefix =3D lb-events
agent.sinks.hdfsSink-2.hdfs.round =3D false
agent.sinks.hdfsSink-2.hdfs.rollCount=3D50
agent.sinks.hdfsSink-2.hdfs.fileType=3DSequenceFile
agent.sinks.hdfsSink-2.hdfs.writeFormat=3DText
agent.sinks.hdfsSink-2.hdfs.codeC =3D lzo
agent.sinks.hdfsSink-2.hdfs.rollInterval=3D30
agent.sinks.hdfsSink-2.hdfs.rollSize=3D0
agent.sinks.hdfsSink-2.hdfs.batchSize=3D1

However, this does not achieve the failover I hoped for. The sink hdfsSink-= 2 on both agents writes the events to HDFS. The agents are not communicatin= g, so the binding of the sink to an ip address is not doing anything.


--f46d0401678d85fcac04d61e1bfa--