Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 52786D7BF for ; Fri, 9 Nov 2012 10:08:56 +0000 (UTC) Received: (qmail 3175 invoked by uid 500); 9 Nov 2012 10:08:56 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 3055 invoked by uid 500); 9 Nov 2012 10:08:55 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 3040 invoked by uid 99); 9 Nov 2012 10:08:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Nov 2012 10:08:55 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of juhani_connolly@cyberagent.co.jp designates 210.134.177.71 as permitted sender) Received: from [210.134.177.71] (HELO ipl2.aams0.jp) (210.134.177.71) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Nov 2012 10:08:47 +0000 Received: from [192.168.177.160] (61-121-214-170.bitcat.net [61.121.214.170] (may be forged)) (authenticated bits=0) by ipl2.aams0.jp (Sentrion-MTA-4.0.2/Switch-3.2.5) with ESMTP id qA9A8Mjl010365 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 9 Nov 2012 19:08:23 +0900 Message-ID: <509CD60D.7040909@cyberagent.co.jp> Date: Fri, 09 Nov 2012 19:08:13 +0900 From: Juhani Connolly User-Agent: Mozilla/5.0 (X11; Linux i686; rv:15.0) Gecko/20120827 Thunderbird/15.0 MIME-Version: 1.0 To: user@flume.apache.org Subject: Re: multiple agents References: <509B6837.7040805@cyberagent.co.jp> In-Reply-To: Content-Type: multipart/alternative; boundary="------------040608000809010701060306" X-AAMS0-Virus-Status: clean X-AAMS0-Virus-Status: clean X-Aams0-Archive-Original-S: YES X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------040608000809010701060306 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit I can't see any obvious problem with your config. When you start up, check your logs if all the components were correctly configured and started. You may need to adjust the log4j configurations in your conf directory. Are all your file channels configured to write to different directories? If they have the same place setup, things aren't going to work well. On 11/09/2012 05:43 PM, Ashutoshsharma(오픈플랫폼개발팀) wrote: > > Hi, > > Can I define the multiple flows with different sources, sinks and > channels as below: > > agent.sources = tx-avro dev-avro web-avro > > agent.sinks = tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink > web-es-sink web-hdfs-sink > > agent.channels = tx-mem-channel tx-file-channel dev-mem-channel > dev-file-channel web-mem-channel web-file-channel > > ##### Flow1 - Start ################################# > > ## Define Avro source > > agent.sources.tx-avro.type = avro > > agent.sources.tx-avro.bind = 0.0.0.0 > > agent.sources.tx-avro.port = 35853 > > agent.sources.tx-avro.channels = tx-mem-channel tx-file-channel > > agent.sources.tx-avro.selector.type = replicating > > ## Define HDFS sink > > agent.sinks.tx-hdfs-sink.type = hdfs > > agent.sinks.tx-hdfs-sink.hdfs.path = hdfs://…/%{hostname}/%Y-%m-%d > > agent.sinks.tx-hdfs-sink.hdfs.fileType = DataStream > > agent.sinks.tx-hdfs-sink.hdfs.writeFormat = Text > > agent.sinks.tx-hdfs-sink.hdfs.filePrefix = transaction > > agent.sinks.tx-hdfs-sink.channel = tx-file-channel > > agent.sinks.tx-hdfs-sink.hdfs.rollCount = 0 > > agent.sinks.tx-hdfs-sink.hdfs.rollSize = 0 > > agent.sinks.tx-hdfs-sink.hdfs.rollInterval = 600 > > ## Define es sink > > agent.sinks.tx-es-sink.type = org.flume.sink.ESSink > > agent.sinks.tx-es-sink.indexName = txlog > > agent.sinks.tx-es-sink.typeName = tx > > agent.sinks.tx-es-sink.cluster = es-cluster > > agent.sinks.tx-es-sink.host = 9.127.216.198 > > agent.sinks.tx-es-sink.channel = tx-mem-channel > > ## Define the memory channel > > agent.channels.tx-mem-channel.type = memory > > agent.channels.tx-mem-channel.capacity = 10000 > > agent.channels.tx-mem-channel.transactionCapacity = 20 > > ## Define the file channel > > agent.channels.tx-file-channel.type = FILE > > agent.channels.tx-file-channel.checkpointDir = > /flume/agent/tx-file-channel/checkpoint > > agent.channels.tx-file-channel.dataDirs = > /flume/agent/tx-file-channel/data > > Same as flow1 #### Flow2 #####....#### Flow3 ######....... is defined > with different port for avro source. Here I am using flow1, flow2 and > flow3 for three different types of logs and store separately i.e. > different location. > > When I defined the flume.conf(collector) as mentioned above, agents > failed to connect to the avro sources. It returns the RPC connection > error. However, I checked that the agent is able to send the events to > the collector if I specify only one avro source. > > So, the question is, can I define the mentioned configuration to have > multiple agents(flows) as mentioned above? > > ---------------------------------------- > > Thanks & Regards, > > Ashutosh Sharma > > ---------------------------------------- > > *From:*Juhani Connolly [mailto:juhani_connolly@cyberagent.co.jp] > *Sent:* Thursday, November 08, 2012 5:07 PM > *To:* user@flume.apache.org > *Subject:* Re: multiple agents > > Hi Ashutosh, > > as was pointed out, one configuration will work fine. > > There is nothing stopping you running multiple background tasks, but > that won't be possible with the service scripts that come with the > flume packaged in cdh, you'd have to write your own service scripts. > But really I can't think of a use case where you would want multiple > processes > > On 11/08/2012 10:39 AM, Ashutoshsharma(오픈플랫폼개발팀) wrote: > > Hi, > > I have sources to collect multiple types of logs(mainly three > types). Most of them generate at least two types of logs. That > mean, a server generates two types of log. For my use case, I > created two separate agents running on a server to collect the > logs. I am running these agents in foreground using “flume-ng > agent –n agent1” command, so two flume process. > > Now, I have doubt, should I merge these two agents configuration > into single flume.conf file? If I want to continue to use two > different conf file, then how can I run the two agents in > background? Is there any known performance issue with any of these > approaches? > > Please share your suggestions and thoughts. > > ---------------------------------------- > > Thanks & Regards, > > Ashutosh Sharma > > ---------------------------------------- > > 이메일은지정된수취인만을위해작성되었으며, 중요한정보나저작권을포함 > 하고있을수있습니다. 어떠한권한없이, 본문서에포함된정보의전부또는일 > 부를무단으로제3자에게공개, 배포, 복사또는사용하는것을엄격히금지합 > 니다. 만약, 본메일이잘못전송된경우, 발신인또는당사에알려주시고, 본 > 메일을즉시삭제하여주시기바랍니다. > This E-mail may contain confidential information and/or copyright > material. This email is intended for the use of the addressee > only. If you receive this email by mistake, please either delete > it without reproducing, distributing or retaining copies thereof > or notify the sender immediately. > > > 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 > 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 > 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것 > 을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당 > 사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다. > This E-mail may contain confidential information and/or copyright > material. This email is intended for the use of the addressee only. If > you receive this email by mistake, please either delete it without > reproducing, distributing or retaining copies thereof or notify the > sender immediately. > --------------040608000809010701060306 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
I can't see any obvious problem with your config.

When you start up, check your logs if all the components were correctly configured and started. You may need to adjust the log4j configurations in your conf directory.

Are all your file channels configured to write to different directories? If they have the same place setup, things aren't going to work well.

On 11/09/2012 05:43 PM, Ashutoshsharma(오픈플랫폼개발팀) wrote:

Hi,

 

Can I define the multiple flows with different sources, sinks and channels as below:

 

agent.sources = tx-avro dev-avro web-avro

agent.sinks = tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink web-es-sink web-hdfs-sink

agent.channels = tx-mem-channel tx-file-channel dev-mem-channel dev-file-channel web-mem-channel web-file-channel

 

##### Flow1 - Start #################################

## Define Avro source

agent.sources.tx-avro.type = avro

agent.sources.tx-avro.bind = 0.0.0.0

agent.sources.tx-avro.port = 35853

agent.sources.tx-avro.channels = tx-mem-channel tx-file-channel

agent.sources.tx-avro.selector.type = replicating

 

## Define HDFS sink

agent.sinks.tx-hdfs-sink.type = hdfs

agent.sinks.tx-hdfs-sink.hdfs.path = hdfs://…/%{hostname}/%Y-%m-%d

agent.sinks.tx-hdfs-sink.hdfs.fileType = DataStream

agent.sinks.tx-hdfs-sink.hdfs.writeFormat = Text

agent.sinks.tx-hdfs-sink.hdfs.filePrefix = transaction

agent.sinks.tx-hdfs-sink.channel = tx-file-channel

agent.sinks.tx-hdfs-sink.hdfs.rollCount = 0

agent.sinks.tx-hdfs-sink.hdfs.rollSize = 0

agent.sinks.tx-hdfs-sink.hdfs.rollInterval = 600

 

## Define es sink

agent.sinks.tx-es-sink.type = org.flume.sink.ESSink

agent.sinks.tx-es-sink.indexName = txlog

agent.sinks.tx-es-sink.typeName = tx

agent.sinks.tx-es-sink.cluster = es-cluster

agent.sinks.tx-es-sink.host = 9.127.216.198

agent.sinks.tx-es-sink.channel = tx-mem-channel

 

## Define the memory channel

agent.channels.tx-mem-channel.type = memory

agent.channels.tx-mem-channel.capacity = 10000

agent.channels.tx-mem-channel.transactionCapacity = 20

 

## Define the file channel

agent.channels.tx-file-channel.type = FILE

agent.channels.tx-file-channel.checkpointDir = /flume/agent/tx-file-channel/checkpoint

agent.channels.tx-file-channel.dataDirs = /flume/agent/tx-file-channel/data

 

Same as flow1 #### Flow2 #####....#### Flow3 ######....... is defined with different port for avro source. Here I am using flow1, flow2 and flow3 for three different types of logs and store separately i.e. different location.

 

When I defined the flume.conf(collector) as mentioned above, agents failed to connect to the avro sources. It returns the RPC connection error. However, I checked that the agent is able to send the events to the collector if I specify only one avro source.

 

So, the question is, can I define the mentioned configuration to have multiple agents(flows) as mentioned above?

 

----------------------------------------

Thanks & Regards,

Ashutosh Sharma

----------------------------------------

 

From: Juhani Connolly [mailto:juhani_connolly@cyberagent.co.jp]
Sent: Thursday, November 08, 2012 5:07 PM
To: user@flume.apache.org
Subject: Re: multiple agents

 

Hi Ashutosh,

as was pointed out, one configuration will work fine.

There is nothing stopping you running multiple background tasks, but that won't be possible with the service scripts that come with the flume packaged in cdh, you'd have to write your own service scripts. But really I can't think of a use case where you would want multiple processes   

On 11/08/2012 10:39 AM, Ashutoshsharma(오픈플랫폼개발팀) wrote:

Hi,

 

I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent –n agent1” command, so two flume process.

 

Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?

 

Please share your suggestions and thoughts.

 

----------------------------------------

Thanks & Regards,

Ashutosh Sharma

----------------------------------------

 

 

이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.

 

 

이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.


--------------040608000809010701060306--