Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (athena.apache.org: domain of asim.zafir@gmail.com
 designates 74.125.82.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAFSTDUhWwPWKjF2cYua-HVAWGGYugRcoerVST_3+YfLozHnrQ@mail.gmail.com>
References: 
 <CAAFSTDUhWwPWKjF2cYua-HVAWGGYugRcoerVST_3+YfLozHnrQ@mail.gmail.com>
Date: Thu, 25 Sep 2014 02:40:21 -0700
Message-ID: 
 <CAEh7H39hfuT-d3eUeRzkOC4qWNgbg6LX8GdaO4StMQmSMrC8nQ@mail.gmail.com>
Subject: Re: Performance of Flume in production systems
From: Asim Zafir <asim.zafir@gmail.com>
To: user@flume.apache.org
Content-Type: multipart/alternative; boundary=047d7bb03cb8da6b460503e09699

--047d7bb03cb8da6b460503e09699
Content-Type: text/plain; charset=UTF-8

It really depends but couple of questions before a proper suggestion can be
made. :

What kind of agent are you using in your pipeline sinking to HDFS?
Does your pipeline involves a collector?
What kind of channel you are using accross the data pipeline?
How frequently do you want to roll the flume events?
It will be helpful to see your data pipeline architecture before making a
suggestion?

Asim Zafir

On Wed, Sep 24, 2014 at 10:53 PM, Blade Liu <hafzcdcn@gmail.com> wrote:

> Hi,
>
> I'm going to deploy Flume in production systems, but a little worried
> about its performance in real-world environment. Could anyone tell me about
> Flume's actual performance in production environment? say, if Flume can
> deal with 20,000 events per second from a single source(and what about
> 100-200 sources with one final HDFS sink).
>
> In addition, to reach good performance of tens of thousands of events per
> second, how many servers(agents) should be used?  More agents(and more
> tiers), better performance?
>
> Thanks very much for your suggestions.
>
>
> Cheers,
> Blade
>

--047d7bb03cb8da6b460503e09699
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>It really depends but couple of questions before a pr=
oper suggestion can be made. :</div><div><br></div>What kind of agent are y=
ou using in your pipeline sinking to HDFS?<div>Does your pipeline involves =
a collector? =C2=A0</div><div>What kind of channel you are using accross th=
e data pipeline?</div><div>How frequently do you want to roll the flume eve=
nts?</div><div>It will be helpful to see your data pipeline architecture be=
fore making a suggestion?</div><div><br>Asim Zafir</div></div><div class=3D=
"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Sep 24, 2014 at 10:53 =
PM, Blade Liu <span dir=3D"ltr">&lt;<a href=3D"mailto:hafzcdcn@gmail.com" t=
arget=3D"_blank">hafzcdcn@gmail.com</a>&gt;</span> wrote:<br><blockquote cl=
ass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;p=
adding-left:1ex"><div dir=3D"ltr"><div>Hi,</div><div><br></div><div>I&#39;m=
 going to deploy Flume in production systems, but a little worried about it=
s performance in real-world environment. Could anyone tell me about Flume&#=
39;s actual performance in production environment? say, if Flume can deal w=
ith 20,000 events per second from a single source(and what about 100-200 so=
urces with one final HDFS sink).</div><div><br></div><div>In addition, to r=
each good performance of tens of thousands of events per second, how many s=
ervers(agents) should be used?=C2=A0 More agents(and more tiers), better pe=
rformance?</div><div><br></div><div>Thanks very much for your suggestions.<=
br></div><div><br></div><div><br></div><div>Cheers,</div><div>Blade</div></=
div>
</blockquote></div><br></div>

--047d7bb03cb8da6b460503e09699--