flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Smaine KAHLOUCH <smaine.kahlo...@arkena.com>
Subject Re: Spreading multiple file-channels over distinct disks
Date Wed, 18 Feb 2015 16:39:50 GMT
Thank you Ahmed, i'll try that


Regards,

Smaine

________________________________
From: Ahmed Vila <avila@devlogic.eu>
Sent: Wednesday, February 18, 2015 5:33 PM
To: user@flume.apache.org
Subject: Re: Spreading multiple file-channels over distinct disks

Hi Smaine,

There is always an overhead of running multiple flume instances, i.e. multiple Java VMs. Thus,
I wouldn't recommend running more of them if that's not due to distribution across servers.

But, you are able to do round-robin events from source into multiple channels and those channels
could have one disk per channel.
However, in that case you'll also need at least one sink per channel. Guess that's even desirable
if you're sinking your data into the HDFS.

Regards

On Wed, Feb 18, 2015 at 10:14 AM, Smaine Kahlouch <smaine.kahlouch@smartjog.com<mailto:smaine.kahlouch@smartjog.com>>
wrote:
Hi all!

I would like to know what is the best way to spread the i/o on my server.
Actually it has several disks and i was wondering how to spread Flume's work over them :
* One Flume instance with multiple working directories (dataDirs), each working directory
on a single disk
* Multiple flume instances with a file channel on a distinct disk (using containers would
be the best way to do so)

Could you please helpe me to find out the best architecture which allows us to keep file-channel
for data persistency ?

Regards,

--
Smaine Kahlouch - Engineer, Research & Engineering
Arkena | T: +33 1 5868 6196<tel:%2B33%201%205868%206196>
27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
arkena.com<http://arkena.com>




--

Best regards,

Ahmed Vila | Senior software developer
DevLogic | Sarajevo | Bosnia and Herzegovina

Office : +387 33 942 123
Mobile: +387 62 139 348

Website: www.devlogic.eu<http://www.devlogic.eu>
E-mail   : avila@devlogic.eu<mailto:avila@devlogic.eu>
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended recipient(s) only. This
email contains confidential information. It should not be copied, disclosed to, retained or
used by, any party other than the intended recipient. Any unauthorised distribution, dissemination
or copying of this E-mail or its attachments, and/or any use of any information contained
in them, is strictly prohibited and may be illegal. If you are not an intended recipient then
please promptly delete this e-mail and any attachment and all copies and inform the sender
directly via email. Any emails that you send to us may be monitored by systems or persons
other than the named communicant for the purposes of ascertaining whether the communication
complies with the law and company policies.

---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended recipient(s) only. This
email contains confidential information. It should not be copied, disclosed to, retained or
used by, any party other than the intended recipient. Any unauthorised distribution, dissemination
or copying of this E-mail or its attachments, and/or any use of any information contained
in them, is strictly prohibited and may be illegal. If you are not an intended recipient then
please promptly delete this e-mail and any attachment and all copies and inform the sender
directly via email. Any emails that you send to us may be monitored by systems or persons
other than the named communicant for the purposes of ascertaining whether the communication
complies with the law and company policies.

Mime
View raw message