Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1345EDF55 for ; Wed, 15 May 2013 11:39:45 +0000 (UTC) Received: (qmail 53322 invoked by uid 500); 15 May 2013 11:39:44 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 52915 invoked by uid 500); 15 May 2013 11:39:43 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 52879 invoked by uid 99); 15 May 2013 11:39:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 May 2013 11:39:42 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of liuyongbo@baidu.com designates 220.181.50.208 as permitted sender) Received: from [220.181.50.208] (HELO baidu.com) (220.181.50.208) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 May 2013 11:39:34 +0000 Received: from M1-MAIL-HUB03.internal.baidu.com (10.44.83.14) by baidu.com (172.22.2.71) with Microsoft SMTP Server (TLS) id 8.2.176.0; Wed, 15 May 2013 19:39:10 +0800 Received: from lionlaptop (220.181.38.105) by email.baidu.com (10.44.83.6) with Microsoft SMTP Server (TLS) id 14.2.247.3; Wed, 15 May 2013 19:39:06 +0800 From: liuyongbo To: References: <011d01ce512e$4aa5d310$dff17930$@com> In-Reply-To: Subject: =?gb2312?B?tPC4tDogaG93IHRvIHByaW50IHRoZSBjaGFubmVsIGNhcGFjaXR5?= Date: Wed, 15 May 2013 19:39:07 +0800 Message-ID: <01c801ce5160$cf36fa00$6da4ee00$@com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_01C9_01CE51A3.DD5A3A00" X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac5RSSVEl2BPUv5BSfO/HCCGVuGIqQAFeRAw Content-Language: zh-cn X-Originating-IP: [220.181.38.105] X-Virus-Checked: Checked by ClamAV on apache.org ------=_NextPart_000_01C9_01CE51A3.DD5A3A00 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: quoted-printable Thanks for your answer. Additional,I=A1=AFm using mem channel, write log to mongodb, when the = input log is faster than consume(write into mongo), the queue is growing, when = reach the max,the new input log is lost. So, what I want to know is the exact point that can blance the input and output =20 =B7=A2=BC=FE=C8=CB: Nitin Pawar [mailto:nitinpawar432@gmail.com]=20 =B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA5=D4=C215=C8=D5 16:49 =CA=D5=BC=FE=C8=CB: user@flume.apache.org =D6=F7=CC=E2: Re: how to print the channel capacity =20 here is one example for the capacity defining flow=20 https://cwiki.apache.org/FLUME/flume-ng-performance-measurements.html =20 On Wed, May 15, 2013 at 2:16 PM, Nitin Pawar wrote: sorry pressed enter too soon=20 =20 as for your question: how many events a flume agent can hold?=20 sorry but I don't think there is any direct answer to that.... .I may be very well wrong there as I am myself pretty new with flume=20 =20 there was a JIRA for the capacity of file channels FLUME-1571 =20 On Wed, May 15, 2013 at 1:50 PM, Nitin Pawar wrote: for maximum performance on your data flow two things which will matter = most are: the channel and the transaction batch size. when you say losing data, are you using memory channel? or file channel? = =20 Flume can batch events. The batch size is the maximum number of events = that a sink or client will attempt to take from a channel in a single transaction. =20 What is the channel type do you have a slow sink so the # events written out are less than # = event incoming to channels so over time it piles up=20 =20 others may point out more things.=20 Also your flume conf and if you are seeing any errors on flume then that will help people to find out the problem=20 =20 On Wed, May 15, 2013 at 11:07 AM, liuyongbo wrote: Hi: I=A1=AFm using flume to pass log data to mongodb, but I find = that some data lose when the pressure is in high level, so I want to know the max request that flume can hold and need to print the capacity.but I can not find the proper way to do this instead of change the source code. Any = ideas? thanks =20 --=20 Nitin Pawar =20 --=20 Nitin Pawar =20 --=20 Nitin Pawar ------=_NextPart_000_01C9_01CE51A3.DD5A3A00 Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable

Thanks for your answer.

Additional,I=A1=AFm using mem channel, write log to = mongodb, when the input log is faster than consume(write into mongo), the queue is = growing, when reach the max,the new input log is lost.

So, what I want to know is the exact point that can = blance the input and output

 

=B7=A2=BC=FE=C8=CB: Nitin Pawar [mailto:nitinpawar432@gmail.com] =
=B7=A2=CB=CD=CA=B1=BC= =E4: = 2013=C4=EA5=D4=C215=C8=D5 16:49
=CA=D5=BC=FE=C8=CB: user@flume.apache.org
=D6=F7=CC=E2: = Re: how to print the channel capacity

 

here is one example for the = capacity defining flow 

 

On Wed, May 15, 2013 at 2:16 PM, = Nitin Pawar <nitinpawar432@gmail.com> wrote:

sorry pressed enter too = soon 

 

as for your question: how many = events a flume agent can hold? 

sorry but I don't think there is = any direct answer to that.... .I may be very well wrong there as I am myself pretty = new with flume 

 

there was a JIRA for the = capacity of file channels FLUME-1571

 

On Wed, May 15, 2013 at 1:50 PM, = Nitin Pawar <nitinpawar432@gmail.com> wrote:

for maximum performance on your data flow two things = which will matter most are: the channel and the transaction batch size.

when you say losing data, are = you using memory channel? or file channel? 

 

Flume can batch events. The batch size is the maximum = number of events that a sink or client will attempt to take from a channel in a = single transaction.

 

 What is the channel type

do you have a slow sink so the # events written out are = less than # event incoming to channels so over time it piles = up 

 

others may point out more things. 

Also your flume conf and if you are seeing any errors on = flume then that will help people to find out the problem 

 

On Wed, May 15, 2013 at 11:07 = AM, liuyongbo <liuyongbo@baidu.com> wrote:

Hi:

         I=A1=AFm = using flume to pass log data to mongodb, but I find that some data lose when the = pressure is in high level, so I want to know the max request that flume can hold and = need to print the capacity.but I can not find the proper way to do this = instead of change the source code. Any ideas?

         = thanks



 

--
Nitin Pawar



 

--
Nitin Pawar



 

--
Nitin Pawar

------=_NextPart_000_01C9_01CE51A3.DD5A3A00--