Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7AAB9D779 for ; Thu, 15 Nov 2012 17:27:39 +0000 (UTC) Received: (qmail 1819 invoked by uid 500); 15 Nov 2012 17:27:39 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 1744 invoked by uid 500); 15 Nov 2012 17:27:39 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 1734 invoked by uid 99); 15 Nov 2012 17:27:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Nov 2012 17:27:39 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [98.139.212.170] (HELO nm11.bullet.mail.bf1.yahoo.com) (98.139.212.170) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Nov 2012 17:27:29 +0000 Received: from [98.139.212.148] by nm11.bullet.mail.bf1.yahoo.com with NNFMP; 15 Nov 2012 17:27:08 -0000 Received: from [98.139.212.195] by tm5.bullet.mail.bf1.yahoo.com with NNFMP; 15 Nov 2012 17:27:08 -0000 Received: from [127.0.0.1] by omp1004.mail.bf1.yahoo.com with NNFMP; 15 Nov 2012 17:27:08 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 205073.25816.bm@omp1004.mail.bf1.yahoo.com Received: (qmail 58706 invoked by uid 60001); 15 Nov 2012 17:27:08 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1353000428; bh=ZeHzExJ+4r57KyoTjrgy15cUOIrm1RjAi8eyflG75zU=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=VG+n4zGwHSbYLQ+CYjlB07ancGou9/Ki+l0ORdt80dJOsBOmYYyxpOXf0/CcgcgMIOePH7miyPSszX528QRWluklsygHtPnqMW6JW+EqpO1zZ+ciPsXBHIhyLgB1twHdihV4gn4Y4klF6Y7q9jKtuuGu6xUBf/yHS+n3dKyRWKU= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=L3A3srf05EjSw7eUJWx7ZJ8WwUuEZqrzXU9seTgtmxwPZuGOhaGXDmY/CfbzsvCxCP/R/fTk7RQlgilsmd51KBx4wSSTDC6+ARms5XoBu/GyrMXDT+hCTFN4xgOHTnKl0mDz3lEi7oY+SWdGb00RKBGFWWobLmBhwPYQAWZPMCc=; X-YMail-OSG: RFK1GkEVM1kxJKItiB6rgt5OV40hIXkGAI00axZt.uzY7YN epyN7xkEksdr5xGX0660TXsjAu2UpfSG0c89dquhdoyZpMo_498KTl6KJVuu UhuvC.aHiU8ijAxhzRFmVQb987x7Do9RxECrTkh4h_38cgLwz0eereAsPTXV lAPXsYHDnrpyK9WWbD66xaHt9G3iLqhdog7Cqg0ABVGaUc0AOKi.vECDD.0U tgO9spHBCmAC5PIrbUMdg_x9Z.AT7umhZCfN9WhgGVNHPJAkfm5PRKoq4I9i zKLp5qioajhr3qhwrIYoeCATF2ngkQcecIg7SPp0_ysftrWSQgbgYk.r78Ml mIjhv4xps.sbjvfbxkneIY9UKu24v4zbzrf_8WVzlKRZRWIaOXwqtXqfMR_9 AxaLMaYb6DrA5z5yUXNGDuyeubR3e4ObEbo8atoyKgfZDgxE0JleiRvmEXZ9 DANop6rvUuVKI60_0BiK9gL7o0e0QrvClKTizoeNJ09bZy1xMa5SErmYB7A- - Received: from [24.5.86.198] by web162404.mail.bf1.yahoo.com via HTTP; Thu, 15 Nov 2012 09:27:07 PST X-Rocket-MIMEInfo: 001.001,SEFQcm94eSBoYXMgYSBUQ1AgbW9kZSB3aGVyZSBpdCByb3VuZCByb2JpbnMgVENQIGNvbm5lY3Rpb25zLiBEb2VzIGl0IG5lZWQgdG8gdW5kZXJzdGFuZCBzb21ldGhpbmcgc3BlY2lmaWMgYWJvdXQgdGhlIHdpcmUgcHJvdG9jb2wgdXNlZCBieSBGbHVtZT8KCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwogRnJvbTogQnJvY2sgTm9sYW5kIDxicm9ja0BjbG91ZGVyYS5jb20.ClRvOiB1c2VyQGZsdW1lLmFwYWNoZS5vcmc7IFJhaHVsIFJhdmluZHJhbiA8cmFodWxydkB5YWhvby5jb20.IApTZW4BMAEBAQE- X-Mailer: YahooMailWebService/0.8.123.460 References: <1352857973.87906.YahooMailNeo@web162403.mail.bf1.yahoo.com> <1352926009.90577.YahooMailNeo@web162402.mail.bf1.yahoo.com> <4131B936-FAC7-4503-9076-FFA771E239BB@yahoo.com> <1352933198.97671.YahooMailNeo@web162403.mail.bf1.yahoo.com> Message-ID: <1353000427.57505.YahooMailNeo@web162404.mail.bf1.yahoo.com> Date: Thu, 15 Nov 2012 09:27:07 -0800 (PST) From: Rahul Ravindran Reply-To: Rahul Ravindran Subject: Re: Flume hops behind HAProxy To: "user@flume.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="736800001-1653398931-1353000427=:57505" X-Virus-Checked: Checked by ClamAV on apache.org --736800001-1653398931-1353000427=:57505 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable HAProxy has a TCP mode where it round robins TCP connections. Does it need = to understand something specific about the wire protocol used by Flume?=0A= =0A=0A________________________________=0A From: Brock Noland =0ATo: user@flume.apache.org; Rahul Ravindran =0A= Sent: Wednesday, November 14, 2012 6:20 PM=0ASubject: Re: Flume hops behind= HAProxy=0A =0A> It would be round robin but not sticky sessions( so each r= equest could Goto any random flume hop)=0A=0ADoes HAProxy understand the pr= otocol?=A0 To round robin requests like=0Athis it needs to understand the c= ommunication protocol, which I=0Asuppose would work if you were using the H= TTPSource.=0A=0AOn Wed, Nov 14, 2012 at 4:46 PM, Rahul Ravindran wrote:=0A> HAProxy is widely deployed already in our deployment a= nd Ops is familiar=0A> with dealing with it for hosts which go down etc.=0A= >=0A> ________________________________=0A> From: "Camp, Roy" =0A> To: "user@flume.apache.org" =0A> Sent: Wednes= day, November 14, 2012 2:15 PM=0A> Subject: RE: Flume hops behind HAProxy= =0A>=0A> Out of curiosity, what is the use case vs using the built in load = balancing?=0A>=0A>=0A>=0A> -----Original Message-----=0A> From: Rahul Ravin= dran [mailto:rahulrv@yahoo.com]=0A> Sent: Wednesday, November 14, 2012 1:49= PM=0A> To: user@flume.apache.org=0A> Cc: user@flume.apache.org=0A> Subject= : Re: Flume hops behind HAProxy=0A>=0A> It would be round robin but not sti= cky sessions( so each request could Goto=0A> any random flume hop)=0A>=0A> = Sent from my phone.Excuse the terseness.=0A>=0A> On Nov 14, 2012, at 1:33 P= M, Brock Noland wrote:=0A>=0A>> I assume it would be c= onnection based round robin?=A0 Might work just=0A>> fine, but probably bes= t to the use built-in support.=0A>>=0A>> On Wed, Nov 14, 2012 at 2:46 PM, R= ahul Ravindran =0A>> wrote:=0A>>> Resending given I sent= it during off-hours.=0A>>>=0A>>> ________________________________=0A>>> Fr= om: Rahul Ravindran =0A>>> To: "user@flume.apache.org" <= user@flume.apache.org>=0A>>> Sent: Tuesday, November 13, 2012 5:52 PM=0A>>>= Subject: Flume hops behind HAProxy=0A>>>=0A>>> Hi,=0A>>>=A0 Before I try i= t, I wanted to check if there were any known issues=0A>>> with this. We wil= l have multiple flume agents sending an Avro stream=0A>>> each to a smaller= set of intermediate flume hops. Are there any=0A>>> issues/concerns around= having the flume agents send their streams to=0A>>> an HAProxy which will = round robin between the different flume hops.=0A>>> Any issue around the tr= ansaction mechanism with this setup?=0A>>>=0A>>> I know that there is a sel= ector mechanism in Flume to do this, but=0A>>> our operations extensively u= se HAProxy, and are most familiar with it.=0A>>>=0A>>> Thanks,=0A>>> ~Rahul= .=0A>>=0A>>=0A>>=0A>> --=0A>> Apache MRUnit - Unit testing MapReduce -=0A>>= http://incubator.apache.org/mrunit/=0A>=0A>=0A=0A=0A=0A-- =0AApache MRUnit= - Unit testing MapReduce - http://incubator.apache.org/mrunit/ --736800001-1653398931-1353000427=:57505 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
HAProxy ha= s a TCP mode where it round robins TCP connections. Does it need to underst= and something specific about the wire protocol used by Flume?
<= div>
From: Brock Noland <brock@cloudera.com>
To: user@flume.apache.org; Rahul Ravi= ndran <rahulrv@yahoo.com>
= Sent: Wednesday, November 14, 2012 6:20 PM
Subject: Re: Flume hops behind HAProxy
=

=0A> It would be round robin but not sticky sessions( so each reque= st could Goto any random flume hop)

Does HAProxy understand the prot= ocol?  To round robin requests like
this it needs to understand the= communication protocol, which I
suppose would work if you were using th= e HTTPSource.

On Wed, Nov 14, 2012 at 4:46 PM, Rahul Ravindran <<= a ymailto=3D"mailto:rahulrv@yahoo.com" href=3D"mailto:rahulrv@yahoo.com">ra= hulrv@yahoo.com> wrote:
> HAProxy is widely deployed already i= n our deployment and Ops is familiar
> with dealing with it for hosts= which go down etc.
>
> ________________________________
>= ; From: "Camp, Roy" <rcamp@ebay.com>
> To: "user@flume.apac= he.org" <user@flume.apache.org>
>= ; Sent: Wednesday, November 14, 2012 2:15 PM
> Subject: RE: Flume hop= s behind HAProxy
>
> Out of curiosity, what is the use case vs = using the built in load balancing?
>
>
>
> -----Ori= ginal Message-----
> From: Rahul Ravindran [mailto:rahulrv@yahoo.com]
> Sent: Wednesday, November 14, 2012 1:49 PM
> To: u= ser@flume.apache.org
> Cc: user@flume.apache.org
&g= t; Subject: Re: Flume hops behind HAProxy
>
> It would be round= robin but not sticky sessions( so each request could Goto
> any rand= om flume hop)
>
> Sent from my phone.Excuse the terseness.
>
> On Nov 14, 2012, at 1:33 PM, Brock Noland <<= a ymailto=3D"mailto:brock@cloudera.com" href=3D"mailto:brock@cloudera.com">= brock@cloudera.com> wrote:
>
>> I assume it would be = connection based round robin?  Might work just
>> fine, but p= robably best to the use built-in support.
>>
>> On Wed, N= ov 14, 2012 at 2:46 PM, Rahul Ravindran <rahulrv@yahoo.com>
>= ;> wrote:
>>> Resending given I sent it during off-hours.>>>
>>> ________________________________
>>&= gt; From: Rahul Ravindran <rahulrv@yahoo.com>
>>> To:= "user@flume.apache.org" <user@flume.apache.org>
>= ;>> Sent: Tuesday, November 13, 2012 5:52 PM
>>> Subject:= Flume hops behind HAProxy
>>>
>>> Hi,
>>&= gt;  Before I try it, I wanted to check if there were any known issues=
>>> with this. We will have multiple flume agents sending an A= vro stream
>>> each to a smaller set of intermediate flume hops= . Are there any
>>> issues/concerns around having the flume age= nts send their streams to
>>> an HAProxy which will round robin= between the different flume hops.
>>> Any issue around the tra= nsaction mechanism with this setup?
>>>
>>> I know = that there is a selector mechanism in Flume to do this, but
>>>= our operations extensively use HAProxy, and are most familiar with it.
= >>>
>>> Thanks,
>>> ~Rahul.
>>
>>
>>
>> --
>> Apa= che MRUnit - Unit testing MapReduce -
>> http://incubator.apache.o= rg/mrunit/
>
>



--
Apache MRUnit - Unit testi= ng MapReduce - http://incubator.apache.org/mrunit/


--736800001-1653398931-1353000427=:57505--