Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (nike.apache.org: domain of smaine.kahlouch@arkena.com
 designates 157.56.112.144 as permitted sender)
From: Smaine KAHLOUCH <smaine.kahlouch@arkena.com>
To: "user@flume.apache.org" <user@flume.apache.org>
Subject: Re: Spreading multiple file-channels over distinct disks
Thread-Topic: Spreading multiple file-channels over distinct disks
Thread-Index: AQHQS1tT5fpmN6lpvEKIeplBx8IaMZz2mngAgAABgP8=
Date: Wed, 18 Feb 2015 16:39:50 +0000
Message-ID: <1424277570918.86352@arkena.com>
References: 
 <54E457EA.2040803@smartjog.com>,<CAH0hrPsSqyLxGjyGkxK2LkZ0WW7gD705myO+6ONfKo0j0HUiZA@mail.gmail.com>
In-Reply-To: 
 <CAH0hrPsSqyLxGjyGkxK2LkZ0WW7gD705myO+6ONfKo0j0HUiZA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_142427757091886352arkenacom_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Feb 2015 16:39:50.2307
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: b7b9e5b9-4875-440c-82e2-364b828200ef
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM3PR04MB0824
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_142427757091886352arkenacom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Thank you Ahmed, i'll try that


Regards,

Smaine

________________________________
From: Ahmed Vila <avila@devlogic.eu>
Sent: Wednesday, February 18, 2015 5:33 PM
To: user@flume.apache.org
Subject: Re: Spreading multiple file-channels over distinct disks

Hi Smaine,

There is always an overhead of running multiple flume instances, i.e. multi=
ple Java VMs. Thus, I wouldn't recommend running more of them if that's not=
 due to distribution across servers.

But, you are able to do round-robin events from source into multiple channe=
ls and those channels could have one disk per channel.
However, in that case you'll also need at least one sink per channel. Guess=
 that's even desirable if you're sinking your data into the HDFS.

Regards

On Wed, Feb 18, 2015 at 10:14 AM, Smaine Kahlouch <smaine.kahlouch@smartjog=
.com<mailto:smaine.kahlouch@smartjog.com>> wrote:
Hi all!

I would like to know what is the best way to spread the i/o on my server.
Actually it has several disks and i was wondering how to spread Flume's wor=
k over them :
* One Flume instance with multiple working directories (dataDirs), each wor=
king directory on a single disk
* Multiple flume instances with a file channel on a distinct disk (using co=
ntainers would be the best way to do so)

Could you please helpe me to find out the best architecture which allows us=
 to keep file-channel for data persistency ?

Regards,

--
Smaine Kahlouch - Engineer, Research & Engineering
Arkena | T: +33 1 5868 6196<tel:%2B33%201%205868%206196>
27 Blvd Hippolyte Marqu=E8s, 94200 Ivry-sur-Seine, France
arkena.com<http://arkena.com>


--

Best regards,

Ahmed Vila | Senior software developer
DevLogic | Sarajevo | Bosnia and Herzegovina

Office : +387 33 942 123
Mobile: +387 62 139 348

Website: www.devlogic.eu<http://www.devlogic.eu>
E-mail   : avila@devlogic.eu<mailto:avila@devlogic.eu>
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended recipi=
ent(s) only. This email contains confidential information. It should not be=
 copied, disclosed to, retained or used by, any party other than the intend=
ed recipient. Any unauthorised distribution, dissemination or copying of th=
is E-mail or its attachments, and/or any use of any information contained i=
n them, is strictly prohibited and may be illegal. If you are not an intend=
ed recipient then please promptly delete this e-mail and any attachment and=
 all copies and inform the sender directly via email. Any emails that you s=
end to us may be monitored by systems or persons other than the named commu=
nicant for the purposes of ascertaining whether the communication complies =
with the law and company policies.

---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended recipi=
ent(s) only. This email contains confidential information. It should not be=
 copied, disclosed to, retained or used by, any party other than the intend=
ed recipient. Any unauthorised distribution, dissemination or copying of th=
is E-mail or its attachments, and/or any use of any information contained i=
n them, is strictly prohibited and may be illegal. If you are not an intend=
ed recipient then please promptly delete this e-mail and any attachment and=
 all copies and inform the sender directly via email. Any emails that you s=
end to us may be monitored by systems or persons other than the named commu=
nicant for the purposes of ascertaining whether the communication complies =
with the law and company policies.

--_000_142427757091886352arkenacom_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
</head>
<body dir=3D"ltr">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;back=
ground-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Thank you Ahmed, i'll try that</p>
<p><br>
</p>
<p>Regards,</p>
<p>Smaine<br>
</p>
<div style=3D"color: rgb(33, 33, 33);">
<hr tabindex=3D"-1" style=3D"display:inline-block; width:98%">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> Ahmed Vila &lt;avila=
@devlogic.eu&gt;<br>
<b>Sent:</b> Wednesday, February 18, 2015 5:33 PM<br>
<b>To:</b> user@flume.apache.org<br>
<b>Subject:</b> Re: Spreading multiple file-channels over distinct disks</f=
ont>
<div>&nbsp;</div>
</div>
<div>
<div dir=3D"ltr">Hi Smaine,
<div><br>
</div>
<div>There is always an overhead of running multiple flume instances, i.e. =
multiple Java VMs. Thus, I wouldn't recommend running more of them if that'=
s not due to distribution across servers.</div>
<div><br>
</div>
<div>But, you are able to do round-robin events from source into multiple c=
hannels and those channels could have one disk per channel.</div>
<div>However, in that case you'll also need at least one sink per channel. =
Guess that's even desirable if you're sinking your data into the HDFS.</div=
>
<div><br>
</div>
<div>Regards</div>
</div>
<div class=3D"gmail_extra"><br>
<div class=3D"gmail_quote">On Wed, Feb 18, 2015 at 10:14 AM, Smaine Kahlouc=
h <span dir=3D"ltr">
&lt;<a href=3D"mailto:smaine.kahlouch@smartjog.com" target=3D"_blank">smain=
e.kahlouch@smartjog.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex; border-left:1=
px #ccc solid; padding-left:1ex">
Hi all!<br>
<br>
I would like to know what is the best way to spread the i/o on my server.<b=
r>
Actually it has several disks and i was wondering how to spread Flume's wor=
k over them :<br>
* One Flume instance with multiple working directories (dataDirs), each wor=
king directory on a single disk<br>
* Multiple flume instances with a file channel on a distinct disk (using co=
ntainers would be the best way to do so)<br>
<br>
Could you please helpe me to find out the best architecture which allows us=
 to keep file-channel for data persistency ?<br>
<br>
Regards,<span class=3D"HOEnZb"><font color=3D"#888888"><br>
<br>
-- <br>
Smaine Kahlouch - Engineer, Research &amp; Engineering<br>
Arkena | T: <a href=3D"tel:%2B33%201%205868%206196" value=3D"&#43;331586861=
96" target=3D"_blank">
&#43;33 1 5868 6196</a><br>
27 Blvd Hippolyte Marqu=E8s, 94200 Ivry-sur-Seine, France<br>
<a href=3D"http://arkena.com" target=3D"_blank">arkena.com</a><br>
<br>
</font></span></blockquote>
</div>
<br>
<br clear=3D"all">
<div><br>
</div>
-- <br>
<div class=3D"gmail_signature">
<div dir=3D"ltr">
<div dir=3D"ltr">
<div dir=3D"ltr">
<div>
<p style=3D"color:rgb(80,0,80)"><span style=3D"color:black">Best regards,</=
span></p>
<div>Ahmed Vila | Senior software developer</div>
<div>DevLogic | Sarajevo | Bosnia and Herzegovina</div>
<div><br>
</div>
<div>Office :&nbsp;&#43;387 33 942 123&nbsp;</div>
<div>Mobile:&nbsp;&#43;387 62 139 348</div>
<div><br>
</div>
<div>Website:&nbsp;<a href=3D"http://www.devlogic.eu" target=3D"_blank">www=
.devlogic.eu</a>&nbsp;</div>
<div>E-mail &nbsp; :&nbsp;<a href=3D"mailto:avila@devlogic.eu" target=3D"_b=
lank">avila@devlogic.eu</a></div>
</div>
</div>
</div>
------------------------------<u></u>------------------------------<u></u>-=
--------<br>
This e-mail and any attachment is for authorised use by the intended recipi=
ent(s) only. This email contains confidential information. It should not be=
 copied, disclosed to, retained or used by, any party other than the intend=
ed recipient. Any unauthorised distribution,
 dissemination or copying of this E-mail or its attachments, and/or any use=
 of any information contained in them, is strictly prohibited and may be il=
legal. If you are not an intended recipient then please promptly delete thi=
s e-mail and any attachment and
 all copies and inform the sender directly via email. Any emails that you s=
end to us may be monitored by systems or persons other than the named commu=
nicant for the purposes of ascertaining whether the communication complies =
with the law and company policies.<br>
</div>
</div>
</div>
<br>
<font size=3D"2">------------------------------<wbr>-----------------------=
-------<wbr>---------<br>
This e-mail and any attachment is for authorised use by the intended recipi=
ent(s) only. This email contains confidential information. It should not be=
 copied, disclosed to, retained or used by, any party other than the intend=
ed recipient. Any unauthorised distribution,
 dissemination or copying of this E-mail or its attachments, and/or any use=
 of any information contained in them, is strictly prohibited and may be il=
legal. If you are not an intended recipient then please promptly delete thi=
s e-mail and any attachment and
 all copies and inform the sender directly via email. Any emails that you s=
end to us may be monitored by systems or persons other than the named commu=
nicant for the purposes of ascertaining whether the communication complies =
with the law and company policies.</font></div>
</div>
</div>
</body>
</html>

--_000_142427757091886352arkenacom_--