flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juhani Connolly <juhani_conno...@cyberagent.co.jp>
Subject Re: multiple agents
Date Fri, 09 Nov 2012 10:08:13 GMT
I can't see any obvious problem with your config.

When you start up, check your logs if all the components were correctly 
configured and started. You may need to adjust the log4j configurations 
in your conf directory.

Are all your file channels configured to write to different directories? 
If they have the same place setup, things aren't going to work well.

On 11/09/2012 05:43 PM, Ashutoshsharma(오픈플랫폼개발팀) wrote:
>
> Hi,
>
> Can I define the multiple flows with different sources, sinks and 
> channels as below:
>
> agent.sources = tx-avro dev-avro web-avro
>
> agent.sinks = tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink 
> web-es-sink web-hdfs-sink
>
> agent.channels = tx-mem-channel tx-file-channel dev-mem-channel 
> dev-file-channel web-mem-channel web-file-channel
>
> ##### Flow1 - Start #################################
>
> ## Define Avro source
>
> agent.sources.tx-avro.type = avro
>
> agent.sources.tx-avro.bind = 0.0.0.0
>
> agent.sources.tx-avro.port = 35853
>
> agent.sources.tx-avro.channels = tx-mem-channel tx-file-channel
>
> agent.sources.tx-avro.selector.type = replicating
>
> ## Define HDFS sink
>
> agent.sinks.tx-hdfs-sink.type = hdfs
>
> agent.sinks.tx-hdfs-sink.hdfs.path = hdfs://…/%{hostname}/%Y-%m-%d
>
> agent.sinks.tx-hdfs-sink.hdfs.fileType = DataStream
>
> agent.sinks.tx-hdfs-sink.hdfs.writeFormat = Text
>
> agent.sinks.tx-hdfs-sink.hdfs.filePrefix = transaction
>
> agent.sinks.tx-hdfs-sink.channel = tx-file-channel
>
> agent.sinks.tx-hdfs-sink.hdfs.rollCount = 0
>
> agent.sinks.tx-hdfs-sink.hdfs.rollSize = 0
>
> agent.sinks.tx-hdfs-sink.hdfs.rollInterval = 600
>
> ## Define es sink
>
> agent.sinks.tx-es-sink.type = org.flume.sink.ESSink
>
> agent.sinks.tx-es-sink.indexName = txlog
>
> agent.sinks.tx-es-sink.typeName = tx
>
> agent.sinks.tx-es-sink.cluster = es-cluster
>
> agent.sinks.tx-es-sink.host = 9.127.216.198
>
> agent.sinks.tx-es-sink.channel = tx-mem-channel
>
> ## Define the memory channel
>
> agent.channels.tx-mem-channel.type = memory
>
> agent.channels.tx-mem-channel.capacity = 10000
>
> agent.channels.tx-mem-channel.transactionCapacity = 20
>
> ## Define the file channel
>
> agent.channels.tx-file-channel.type = FILE
>
> agent.channels.tx-file-channel.checkpointDir = 
> /flume/agent/tx-file-channel/checkpoint
>
> agent.channels.tx-file-channel.dataDirs = 
> /flume/agent/tx-file-channel/data
>
> Same as flow1 #### Flow2 #####....#### Flow3 ######....... is defined 
> with different port for avro source. Here I am using flow1, flow2 and 
> flow3 for three different types of logs and store separately i.e. 
> different location.
>
> When I defined the flume.conf(collector) as mentioned above, agents 
> failed to connect to the avro sources. It returns the RPC connection 
> error. However, I checked that the agent is able to send the events to 
> the collector if I specify only one avro source.
>
> So, the question is, can I define the mentioned configuration to have 
> multiple agents(flows) as mentioned above?
>
> ----------------------------------------
>
> Thanks & Regards,
>
> Ashutosh Sharma
>
> ----------------------------------------
>
> *From:*Juhani Connolly [mailto:juhani_connolly@cyberagent.co.jp]
> *Sent:* Thursday, November 08, 2012 5:07 PM
> *To:* user@flume.apache.org
> *Subject:* Re: multiple agents
>
> Hi Ashutosh,
>
> as was pointed out, one configuration will work fine.
>
> There is nothing stopping you running multiple background tasks, but 
> that won't be possible with the service scripts that come with the 
> flume packaged in cdh, you'd have to write your own service scripts. 
> But really I can't think of a use case where you would want multiple 
> processes
>
> On 11/08/2012 10:39 AM, Ashutoshsharma(오픈플랫폼개발팀) wrote:
>
>     Hi,
>
>     I have sources to collect multiple types of logs(mainly three
>     types). Most of them generate at least two types of logs. That
>     mean, a server generates two types of log. For my use case, I
>     created two separate agents running on a server to collect the
>     logs. I am running these agents in foreground using “flume-ng
>     agent –n agent1” command, so two flume process.
>
>     Now, I have doubt, should I merge these two agents configuration
>     into single flume.conf file? If I want to continue to use two
>     different conf file, then how can I run the two agents in
>     background? Is there any known performance issue with any of these
>     approaches?
>
>     Please share your suggestions and thoughts.
>
>     ----------------------------------------
>
>     Thanks & Regards,
>
>     Ashutosh Sharma
>
>     ----------------------------------------
>
>     이메일은지정된수취인만을위해작성되었으며, 중요한정보나저작권을포함
>     하고있을수있습니다. 어떠한권한없이, 본문서에포함된정보의전부또는일
>     부를무단으로제3자에게공개, 배포, 복사또는사용하는것을엄격히금지합
>     니다. 만약, 본메일이잘못전송된경우, 발신인또는당사에알려주시고,
본
>     메일을즉시삭제하여주시기바랍니다.
>     This E-mail may contain confidential information and/or copyright
>     material. This email is intended for the use of the addressee
>     only. If you receive this email by mistake, please either delete
>     it without reproducing, distributing or retaining copies thereof
>     or notify the sender immediately.
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나
저작권을 
> 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된
정보의 
> 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는
것 
> 을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인
또는 당 
> 사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
> This E-mail may contain confidential information and/or copyright 
> material. This email is intended for the use of the addressee only. If 
> you receive this email by mistake, please either delete it without 
> reproducing, distributing or retaining copies thereof or notify the 
> sender immediately.
>


Mime
View raw message