flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Gentile <juan.gent...@globant.com>
Subject Re: Flume-ng - Distributed
Date Wed, 10 Oct 2012 19:49:32 GMT
Great thanks this clears everything thanks!

On Wed, Oct 10, 2012 at 4:45 PM, Harish Mandala <mvharish14988@gmail.com>wrote:

> You would use something like puppet or chef to push such config files
> around.
>
> Regards,
> Harish
>
>
> On Wed, Oct 10, 2012 at 2:19 PM, Camp, Roy <rcamp@ebay.com> wrote:
>
>>  You have to manually start each node with its specific configuration.
>> However, you can put the configuration for all your different setups into
>> one config file, but you will still need to place a copy of it on every
>> machine.  Simply define which agent config to use with the --name param
>> when starting.  ****
>>
>> ** **
>>
>> Thanks,****
>>
>> ** **
>>
>> Roy****
>>
>> ** **
>>
>> ** **
>>
>> *From:* Juan Gentile [mailto:juan.gentile@globant.com]
>> *Sent:* Wednesday, October 10, 2012 9:54 AM
>> *To:* user@flume.apache.org
>> *Subject:* Re: Flume-ng - Distributed****
>>
>> ** **
>>
>> Thank you both very much, I've been reading the documentation you sent me
>> and this brings another question, is there a way to submit my flume
>> configuration to a cluster or I have to manually start up each node with
>> the its specific configuration?****
>>
>> ** **
>>
>> Thank you!****
>>
>> On Wed, Oct 10, 2012 at 1:51 AM, Mike Percy <mpercy@apache.org> wrote:***
>> *
>>
>> +1 on what Roy said, with a minor terminology quibble: in Flume NG the
>> Avro collector component is called the Avro Source.****
>>
>> ** **
>>
>> Also, here are links to the docs with working image links and table of
>> contents:****
>>
>> ** **
>>
>> http://flume.apache.org/FlumeUserGuide.html****
>>
>> http://flume.apache.org/FlumeDeveloperGuide.html****
>>
>> ** **
>>
>> Regards,****
>>
>> Mike****
>>
>> ** **
>>
>> On Tue, Oct 9, 2012 at 5:52 PM, Camp, Roy <rcamp@ebay.com> wrote:****
>>
>> You would run a flume-ng instance on each node with an avro-sink.  Then
>> on your collector machine you will run another flume-ng instance with an
>> avro-collector.****
>>
>>  ****
>>
>> If you run more than one collector you can setup sink groups and define
>> that it does failover or load balancing.****
>>
>>  ****
>>
>> The concept of a flume master from flume 0.9.x does not exist on
>> flume-ng.  I personally use the node and collector configs in the same
>> config file under a different agent name, and then keep them synced on all
>> machines.  ****
>>
>>  ****
>>
>> These two docs are pretty helpful:****
>>
>>
>> https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst
>>
>> https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeDeveloperGuide.rst
>> ****
>>
>>  ****
>>
>> Thanks,****
>>
>>  ****
>>
>> Roy****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>> *From:* Juan Gentile [mailto:juan.gentile@globant.com]
>> *Sent:* Tuesday, October 09, 2012 11:04 AM
>> *To:* user@flume.apache.org
>> *Subject:* Flume-ng - Distributed****
>>
>>  ****
>>
>> Hi,****
>>
>>  ****
>>
>> I'm new to Flume-ng, I'd like to ask you if you can tell me how I can
>> accomplish to have an agent distributed in a cluster. I've have developed
>> my own source and sink version that reads from a queue and the sink stores
>> the messages read to hdfs. If I want to have this running in multiple
>> instances, do I have to submit it on each node?****
>>
>>  ****
>>
>> This is my conf file:****
>>
>> agent1.channels.channel1.type = memory****
>>
>> agent1.channels.channel1.capacity = 1000****
>>
>> agent1.channels.channel1.transactionCapacity = 1000****
>>
>>  ****
>>
>> agent1.sources.source1.channels = channel1****
>>
>> agent1.sources.source1.type = MySource****
>>
>>  ****
>>
>> agent1.sinks.sink1.channel = channel1****
>>
>> agent1.sinks.sink1.type = MySink****
>>
>>  ****
>>
>> agent1.channels = channel1****
>>
>> agent1.sources = source1****
>>
>> agent1.sinks = sink1****
>>
>>  ****
>>
>>  ****
>>
>> I see that there is the concept of 'master' a 'node' in the previous
>> version of flume, do I have something similar here?****
>>
>>  ****
>>
>> Thanks,****
>>
>> Juan****
>>
>> ** **
>>
>> ** **
>>
>
>

Mime
View raw message