flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ashutosh(오픈플랫폼개발팀) <sharma.ashut...@kt.com>
Subject RE: Sink Processor
Date Thu, 16 Aug 2012 09:43:36 GMT
Thank you very much Mark!! It works perfectly.

Thanks & Regards,
Ashutosh Sharma

-----Original Message-----
From: Stern, Mark [mailto:mstern@iskoot.com]
Sent: Thursday, August 16, 2012 3:15 PM
To: ashutosh(오픈플랫폼개발팀)
Cc: user@flume.apache.org
Subject: RE: Sink Processor

Hi Sharma,

If you want to use a failover mechanism, you should connect both sinks in your failover group
to a single channel. So I suggest you remove mem-channel1.

agent.channels = mem-channel mem-channel1

becomes

agent.channels = mem-channel

agent.sources.tail-source.channels = mem-channel mem-channel1

becomes

agent.sources.tail-source.channels = mem-channel

Delete
agent.sources.tail-source.selector.type = replicating

agent.sinks.avro-forward-sink2.channel = mem-channel1

becomes

agent.sinks.avro-forward-sink2.channel = mem-channel

and you can also delete the last 3 lines that define mem-channel1.

Hope this helps,

Mark

On Thu, 2012-08-16 at 01:23 +0000, ashutosh(오픈플랫폼개발팀) wrote:
> Hi Mark,
>
> Below is my configuration file. Please suggest if there is any mistake.
>
> ## Name the components on this agent
> agent.sources = tail-source
> agent.sinks = avro-forward-sink1 avro-forward-sink2 agent.channels =
> mem-channel mem-channel1 agent.sinkgroups = failover-sink-group
>
> ## Describe the tail source
> agent.sources.tail-source.type = exec
> agent.sources.tail-source.command = tail -F /home/ashutosh/test.txt
> agent.sources.tail-source.channels = mem-channel mem-channel1
> agent.sources.tail-source.selector.type = replicating
> agent.sources.tail-source.interceptors = host t-stamp
>
> ## Describe Host interceptor
> agent.sources.tail-source.interceptors.host.type =
> org.apache.flume.interceptor.HostInterceptor$Builder
> agent.sources.tail-source.interceptors.host.preserveExisting = false
> agent.sources.tail-source.interceptors.host.hostHeader = hostname
> agent.sources.tail-source.interceptors.host.useIP = false
>
> ## Describe Timestamp interceptor
> agent.sources.tail-source.interceptors.t-stamp.type =
> org.apache.flume.interceptor.TimestampInterceptor$Builder
>
> ## Describe the avro sink to forward the events to down the line
> agent1(consumer2) agent.sinks.avro-forward-sink1.type = avro
> agent.sinks.avro-forward-sink1.channel = mem-channel
> agent.sinks.avro-forward-sink1.hostname = 192.168.32.27
> agent.sinks.avro-forward-sink1.port = 61001
> agent.sinks.avro-forward-sink1.batch-size = 20
>
> ## Describe the avro sink to forward the events to down the line
> agent2(consumer1) agent.sinks.avro-forward-sink2.type = avro
> agent.sinks.avro-forward-sink2.channel = mem-channel1
> agent.sinks.avro-forward-sink2.hostname = 192.168.32.26
> agent.sinks.avro-forward-sink2.port = 61001
> agent.sinks.avro-forward-sink2.batch-size = 20
>
> ## Describe the sink processor for failover. Consumer1 takes charges if consumer2 fails.
> agent.sinkgroups.failover-sink-group.processor.type = failover
> agent.sinkgroups.failover-sink-group.sinks = avro-forward-sink1
> avro-forward-sink2
> agent.sinkgroups.failover-sink-group.processor.priority.avro-forward-s
> ink1 = 10
> agent.sinkgroups.failover-sink-group.processor.priority.avro-forward-s
> ink2 = 5
>
> ## Define the memory channel
> agent.channels.mem-channel.type = memory
> agent.channels.mem-channel.capacity = 1000
> agent.channels.mem-channel.transactionCapacity = 20
>
> ## Define the file channel
> agent.channels.mem-channel1.type = memory
> agent.channels.mem-channel1.capacity = 1000
> agent.channels.mem-channel1.transactionCapacity = 20
>
> Thanks & Regards,
> Ashutosh Sharma
>
>
>
> -----Original Message-----
> From: Stern, Mark [mailto:mstern@iskoot.com]
> Sent: Tuesday, August 14, 2012 8:45 PM
> To: user@flume.apache.org
> Subject: RE: Sink Processor
>
> This sounds like a configuration error. Can you provide your configuration file?
> ________________________________________
> From: ashutosh(오픈플랫폼개발팀) [sharma.ashutosh@kt.com]
> Sent: Tuesday, August 14, 2012 12:07 PM
> To: user@flume.apache.org
> Subject: Sink Processor
>
> Hi All,
>
> My query is related to the flume’s sink processor for the failover. I implemented the
failover data flow to transfer the event to downstream in case of agent failure.
>
> I have configured the flume’s sinkgroups for failover of the sinks. In case of the
failure of sink1, sink2 take charge of storing data into hdfs.
> I tested the functionality of the sinkgroups’s failover successfully. When sink1 fails,
sink2 successfully takes the charge and transfer the data to the destination.
>
> Here I am transferring the events by tailing a file contains 100 records. I shutdown
the sink1 when it has processed 40 records.. After that all data start transferred by sink2.
But, the file saved in hdfs contains 140 records. It means both downstream agents hold the
events, if one fails another start transfer the events from the start as it hold all data
in its defined channel.
>
> Is this the intended behavior of the flume’s sinkgroup in case of failover? How can
I avoid the duplicate events? Is there any solution as part of flume’s sink processor to
deal with the duplicates?
>
> Thanks & Regards,
> Ashutosh
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나
저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에
포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사
또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된
경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기
바랍니다.
> This E-mail may contain confidential information and/or copyright material. This email
is intended for the use of the addressee only. If you receive this email by mistake, please
either delete it without reproducing, distributing or retaining copies thereof or notify the
sender immediately.
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나
저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에
포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사
또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된
경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기
바랍니다.
> This E-mail may contain confidential information and/or copyright material. This email
is intended for the use of the addressee only. If you receive this email by mistake, please
either delete it without reproducing, distributing or retaining copies thereof or notify the
sender immediately.



이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을
포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의
전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는
것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인
또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is
intended for the use of the addressee only. If you receive this email by mistake, please either
delete it without reproducing, distributing or retaining copies thereof or notify the sender
immediately.
Mime
View raw message