Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9DD9E17BF3 for ; Mon, 23 Mar 2015 22:32:37 +0000 (UTC) Received: (qmail 78117 invoked by uid 500); 23 Mar 2015 22:32:37 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 78050 invoked by uid 500); 23 Mar 2015 22:32:37 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 78040 invoked by uid 99); 23 Mar 2015 22:32:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Mar 2015 22:32:37 +0000 X-ASF-Spam-Status: No, hits=3.2 required=5.0 tests=FSL_HELO_BARE_IP_2,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of roshan@hortonworks.com designates 64.78.52.187 as permitted sender) Received: from [64.78.52.187] (HELO relayvx12c.securemail.intermedia.net) (64.78.52.187) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Mar 2015 22:32:32 +0000 Received: from securemail.intermedia.net (localhost [127.0.0.1]) by emg-ca-1-2.localdomain (Postfix) with ESMTP id 2C3F353E49 for ; Mon, 23 Mar 2015 15:31:09 -0700 (PDT) Subject: Re: Load Balancer MIME-Version: 1.0 x-echoworx-emg-received: Mon, 23 Mar 2015 15:31:09.146 -0700 x-echoworx-msg-id: 1e270da4-b304-4864-bd3b-c251eed768b1 x-echoworx-action: delivered Received: from 10.254.155.17 ([10.254.155.17]) by emg-ca-1-2 (JAMES SMTP Server 2.3.2) with SMTP ID 311 for ; Mon, 23 Mar 2015 15:31:09 -0700 (PDT) Received: from MBX080-W4-CO-1.exch080.serverpod.net (unknown [10.224.117.101]) by emg-ca-1-2.localdomain (Postfix) with ESMTP id D8EB953E49 for ; Mon, 23 Mar 2015 15:31:08 -0700 (PDT) Received: from MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) by MBX080-W4-CO-1.exch080.serverpod.net (10.224.117.101) with Microsoft SMTP Server (TLS) id 15.0.1044.25; Mon, 23 Mar 2015 15:31:07 -0700 Received: from MBX080-W4-CO-2.exch080.serverpod.net ([10.224.117.102]) by mbx080-w4-co-2.exch080.serverpod.net ([10.224.117.102]) with mapi id 15.00.1044.021; Mon, 23 Mar 2015 15:31:07 -0700 From: Roshan Naik To: "user@flume.apache.org" Thread-Topic: Load Balancer Thread-Index: AQHQWyN+MhSRD6kEx06gUHsal9Q7LJ0WCdOAgABMdwCAFGZygA== Date: Mon, 23 Mar 2015 22:31:07 +0000 Message-ID: In-Reply-To: <54FF1596.6010901@cloudera.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [192.175.27.17] x-source-routing-agent: Processed Content-Type: multipart/alternative; boundary="_000_D135E2C811077roshanhortonworkscom_" X-Virus-Checked: Checked by ClamAV on apache.org --_000_D135E2C811077roshanhortonworkscom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Might be good idea to make a note of this correlation between sink groups a= nd threads in the user guide. From: Hari Shreedharan > Reply-To: "user@flume.apache.org" > Date: Tuesday, March 10, 2015 9:02 AM To: "user@flume.apache.org" > Subject: Re: Load Balancer Most often, for HDFS, you'd need multiple sinks to drain the same channel t= han one with a load balancer. Load balancing is mostly meant to send data f= rom one set of Flume agents to another set (tier 1 to 2), so that even if o= ne agent goes down, the data still goes to the others. Adding more sinks or= sink groups adds multithreading - so your drain rates improve. Each sink g= roup gets a thread - the thread chooses one of the sinks to process the dat= a from the channel. So for multithreading + load balancing, you'd need seve= ral (possibly, identically configured, but with different sinks) sink group= s. Think of sinks outside of sink groups as a sink group with just 1 sink. j.guilmard@accenture.com wrote: Hello, No they are not as far as I know. That makes it very important to know when= designing Load Balanced Flume Architecture: you must count on multiple age= nts, rather than multiple sinks Best regards ________________________________________________ Jean-Fran=E7ois Guilmard Accenture Digital Email: j.guilmard@accenture.com Phone: +33 6 32 27 70 22 Accenture Analytics: Connect with us to learn more. -----Original Message----- From: Guillermo Ortiz [mailto:konstt2000@gmail.com] Sent: mardi 10 mars 2015 12:15 To: user@flume.apache.org Subject: Load Balancer I was checking the load balancing with a group of HDFS-Sink, are they multi= threading?? I mean, each sink of the group is an independent thread as when= I have many sinks? When should I use loadbalancer instance of independend sinks? ________________________________ This message is for the designated recipient only and may contain privilege= d, proprietary, or otherwise confidential information. If you have received= it in error, please notify the sender immediately and delete the original.= Any other use of the e-mail by you is prohibited. Where allowed by local l= aw, electronic communications with Accenture and its affiliates, including = e-mail and instant messaging (including content), may be scanned by our sys= tems for the purposes of information security and assessment of internal co= mpliance with Accenture policy. ___________________________________________________________________________= ___________ www.accenture.com --_000_D135E2C811077roshanhortonworkscom_ Content-Type: text/html; charset="iso-8859-1" Content-ID: <07D9643ACD97D4478412B48122D8C5CA@exch080.serverpod.net> Content-Transfer-Encoding: quoted-printable
Might be good idea to make a note of this correlation between sink gro= ups and threads in the user guide.

From: Hari Shreedharan <hshreedharan@cloudera.com>
Reply-To: "user@flume.apache.org" <user@flume.apache.org>
Date: Tuesday, March 10, 2015 9:02 = AM
To: "user@flume.apache.org" <user@flume.apache.org>
Subject: Re: Load Balancer

Most often, for HDFS, you'd need multiple sinks to drain the same chan= nel than one with a load balancer. Load balancing is mostly meant to send d= ata from one set of Flume agents to another set (tier 1 to 2), so that even= if one agent goes down, the data still goes to the others. Adding more sinks or sink groups adds multithrea= ding - so your drain rates improve. Each sink group gets a thread - the thr= ead chooses one of the sinks to process the data from the channel. So for m= ultithreading + load balancing, you'd need several (possibly, identically configured, but with different s= inks) sink groups. Think of sinks outside of sink groups as a sink group wi= th just 1 sink.

j.guilmard@accenture.com wr= ote:

Hello,

No they are not as far as I know. That makes it very important to know when= designing Load Balanced Flume Architecture: you must count on multiple age= nts, rather than multiple sinks

Best regards
________________________________________________
Jean-Fran=E7ois Guilmard
Accenture Digital

Email: j.guilmard@accenture.com=
Phone: +33 6 32 27 70 22

Accenture Analytics: Connect with us to learn more.










-----Original Message-----
From: Guillermo Ortiz [mailto:konst= t2000@gmail.com]
Sent: mardi 10 mars 2015 12:15
To: user@flume.apache.org
Subject: Load Balancer

I was checking the load balancing with a group of HDFS-Sink, are they multi= threading?? I mean, each sink of the group is an independent thread as when= I have many sinks?
When should I use loadbalancer instance of independend sinks?

________________________________

This message is for the designated recipient only and may contain privilege= d, proprietary, or otherwise confidential information. If you have received= it in error, please notify the sender immediately and delete the original.= Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications w= ith Accenture and its affiliates, including e-mail and instant messaging (i= ncluding content), may be scanned by our systems for the purposes of inform= ation security and assessment of internal compliance with Accenture policy.
___________________________________________________________________________= ___________

www.accenture.com
--_000_D135E2C811077roshanhortonworkscom_--