Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EF32F10FEB for ; Fri, 10 Jan 2014 03:25:33 +0000 (UTC) Received: (qmail 98176 invoked by uid 500); 10 Jan 2014 03:25:28 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 98145 invoked by uid 500); 10 Jan 2014 03:25:27 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 98130 invoked by uid 99); 10 Jan 2014 03:25:25 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jan 2014 03:25:25 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of chen.apache.solr@gmail.com designates 209.85.214.46 as permitted sender) Received: from [209.85.214.46] (HELO mail-bk0-f46.google.com) (209.85.214.46) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jan 2014 03:25:20 +0000 Received: by mail-bk0-f46.google.com with SMTP id u15so1374666bkz.5 for ; Thu, 09 Jan 2014 19:24:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=g72Y8/DM2gJMKghJ02hmn2wrnUj5QlYutc4mQCnZNsE=; b=LMHQ/LSx4VPB0pFkybEPHIMjtmz5+Tp0QX7T+7nB+bYLNHhqs9F+mXrNSMwa/EGgMk MnwCbaULfA1nIcBfF2KrlGwGPszX2LJOSCoMzfkCJzGzWLkDFYrPofVk69eTJ4AGwkOo Ji0QXqBCGLUSzMgxjBZJBWrzrJPFRLW+U66CGfA9bv19Fn6DOCYM9LxTMDOI1FyLFOPi b4JGRoD8J7DlmLXCTnvbMgvldQwj901KZGF6iev0Heb+o00u+defkctygwqoOOwCtp8o RTzhhg1vBSVBbjWrtPx46/O/MqswuR0mSUsch3WCGyiXPaLZ4x8vuBYv34YFoQaYtrVY 3+WQ== MIME-Version: 1.0 X-Received: by 10.205.26.67 with SMTP id rl3mr21702bkb.162.1389324299592; Thu, 09 Jan 2014 19:24:59 -0800 (PST) Received: by 10.204.172.144 with HTTP; Thu, 9 Jan 2014 19:24:59 -0800 (PST) In-Reply-To: References: Date: Thu, 9 Jan 2014 19:24:59 -0800 Message-ID: Subject: Re: seeking help on flume cluster deployment From: Chen Wang To: user@flume.apache.org Content-Type: multipart/alternative; boundary=20cf301ee8115b0b4b04ef954507 X-Virus-Checked: Checked by ClamAV on apache.org --20cf301ee8115b0b4b04ef954507 Content-Type: text/plain; charset=ISO-8859-1 Thanks Saurabh. If that is the case, I am actually thinking about using storm spout to talk to our socket server so that the storm cluster can take care of the reading socket server part. Then in each storm node, start a flume agent, listening on a RPC port and write to HDFS(with fail over) .Then in the storm bolt, simply send the data to RPC so that flume can get it. How do you think of this setup? It takes care of both failover on the source(by storm) and on the sink(by flume) But It looks a little complicated for me. Chen On Thu, Jan 9, 2014 at 7:18 PM, Saurabh B wrote: > Hi Chen, > > I think Flume doesn't have a way to configure multiple sources pointing to > same data source. Of course you can do that, but you will end up with > duplicate data. Flume offers fail over at the sink level. > > On Thu, Jan 9, 2014 at 6:56 PM, Chen Wang wrote: > >> Ok. so after more researching:) It seems that what i need is the failover >> for agent source, (not fail over for sink): >> If one agent dies, another same kind of agent will start running. >> Does flume support this scenario? >> Thanks, >> Chen >> >> >> On Thu, Jan 9, 2014 at 3:12 PM, Chen Wang wrote: >> >>> After reading more docs, it seems that if I want to achieve my goal, i >>> have to do the following: >>> 1. Having one agent with the custom source running on one node. This >>> agent reads from those 5 socket server, and sink to some kind of sink(maybe >>> another socket?) >>> 2. On another(or more) machines, setting up collectors that read from >>> the agent sink in 1, and sink to hdfs. >>> 3. Having a master node managing nodes in 1,2. >>> >>> But it seems to be overskilled in my case: in 1, i can already sink to >>> hdfs. Since the data available at socket server are much faster than the >>> data translation part. I want to be able to later add more nodes to do the >>> translation job. so what is the correct setup? >>> Thanks, >>> Chen >>> >>> >>> >>> On Thu, Jan 9, 2014 at 2:38 PM, Chen Wang wrote: >>> >>>> Guys, >>>> In my environment, the client is 5 socket servers. Thus i wrote a >>>> custom source spawning 5 threads reading from each of them infinitely,and >>>> the sink is hdfs(hive table). The work fine by running flume-ng agent. >>>> >>>> But how can i deploy this in distributed mode(cluster)? I am confused >>>> about the 3 ties(agent,collector,storage) mentioned in the doc. Does it >>>> apply to my case? How can I separate my agent/collect/storage? Apparently i >>>> can only have one agent running: multiple agent will result in getting >>>> duplicates from the socket server. But I want that if one agent dies, other >>>> agent can take it up. I would also like to be able to add horizontal >>>> scalability for writing to hdfs. How can I achieve all this? >>>> >>>> thank you very much for your advice. >>>> Chen >>>> >>> >>> >> > > > -- > Mailing List Archives, > QnaList.com > --20cf301ee8115b0b4b04ef954507 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks Saurabh.=A0
If that is the case, I am actually = thinking about using storm spout to talk to our socket server so that the s= torm cluster can take care of the reading socket server part. Then in each = storm node, start a flume agent, listening on a RPC port and write to HDFS(= with fail over) .Then in the storm bolt, simply send the data to RPC so tha= t flume can get it.=A0
How do you think of this setup? It takes care of both failover on the sourc= e(by storm) and on the sink(by flume) But It looks a little complicated for= me.
Chen


On Thu, Jan 9, 2014 at 7:18 PM, Saurabh B <qna.list.141211@gmail.c= om> wrote:
Hi Chen,

I think Flume doesn't have= a way to configure multiple sources pointing to same data source. Of cours= e you can do that, but you will end up with duplicate data. Flume offers fa= il over at the sink level.

On Thu, Jan 9, 2014 at 6:56 PM, Chen Wang <chen.apache.solr@gmail.com> wrote:
Ok. so after more researchi= ng:) It seems that what i need is the failover for agent source, (not fail = over for sink):
If one agent dies, another same kind of agent will start running.
Does flume support this scenario?
Thanks,
Chen=A0


On Thu, Jan 9, 2014 at 3:12 PM, Chen= Wang <chen.apache.solr@gmail.com> wrote:
After reading more docs, it= seems that if I want to achieve my goal, i have to do the following:
1= . Having one agent with the custom source running on one node. This agent r= eads from those 5 socket server, and sink to some kind of sink(maybe anothe= r socket?)
2. On another(or more) machines, setting up collectors that read from = the agent sink in 1, and sink to hdfs.
3. Having a master node ma= naging nodes in 1,2.

But it seems to be overskille= d in my case: in 1, i can already sink to hdfs. Since the data available at= socket server are much faster than the data translation part. =A0I want to= be able to later add more nodes to do the translation job. so what is the = correct setup?
Thanks,
Chen


On Thu, Jan 9, 2014 at 2:38 PM, Chen Wang <chen.apache.solr@gmail.com> wrote:
Guys,
In my environment= , the client is 5 socket servers. Thus i wrote a custom source spawning 5 t= hreads reading from each of them infinitely,and the sink is hdfs(hive table= ). The work fine by running=A0flume-ng agent.
=A0<= br>
But how can i deploy this in distributed mode(cluster)? I am= confused about the 3 ties(agent,collector,storage) mentioned in the doc. D= oes it apply to my case? How can I separate my agent/collect/storage? Appar= ently i can only have one agent running: multiple agent will result in gett= ing duplicates from the socket server. But I want that if one agent dies, o= ther agent can take it up. I would also like to be able to add horizontal s= calability for writing to hdfs. How can I achieve all this?

thank you very much for your advice.
<= font color=3D"#888888">
Chen





<= /div>--
Mailing List Archives,

--20cf301ee8115b0b4b04ef954507--