Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 181D310534 for ; Thu, 6 Feb 2014 21:12:27 +0000 (UTC) Received: (qmail 68048 invoked by uid 500); 6 Feb 2014 21:12:25 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 67979 invoked by uid 500); 6 Feb 2014 21:12:24 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 67971 invoked by uid 99); 6 Feb 2014 21:12:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Feb 2014 21:12:24 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Matt.Kenison@disney.com designates 204.128.192.17 as permitted sender) Received: from [204.128.192.17] (HELO msg1.disney.com) (204.128.192.17) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Feb 2014 21:12:16 +0000 Received: from int1.disney.pvt (int1.disney.pvt [153.7.110.9]) by msg1.disney.com (Sentrion-MTA-4.2.2/Sentrion-MTA-4.2.2) with ESMTP id s16LBsXx023234 for ; Thu, 6 Feb 2014 21:11:54 GMT Received: from sm-cala-xht03.swna.wdpr.disney.com (SM-CALA-XHT03.swna.wdpr.disney.com [153.7.248.18]) by int1.disney.pvt (Sentrion-MTA-4.2.2/Sentrion-MTA-4.2.2) with ESMTP id s16LBHTC001767 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Thu, 6 Feb 2014 21:11:53 GMT Received: from sm-cala-vxmb06b.swna.wdpr.disney.com ([::1]) by sm-cala-xht03.swna.wdpr.disney.com ([::1]) with mapi; Thu, 6 Feb 2014 13:11:23 -0800 From: "Kenison, Matt" To: "user@flume.apache.org" Date: Thu, 6 Feb 2014 13:11:19 -0800 Subject: Re: Channel management: messages that will never be delivered Thread-Topic: Channel management: messages that will never be delivered Thread-Index: Ac8jf/0QH3bxBkopQYiUzZHu+CpkEw== Message-ID: In-Reply-To: <4A7D5110AA4DCF4F98905065947956E315C5B39B@BGB01XUD1008.national.core.bbc.co.uk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.2.4.120824 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CF19380F233Amattkenisondisneycom_" MIME-Version: 1.0 X-Flow-Control: Sendmail Flow Controller v2.2.5 int1.disney.pvt s16LBHTC001767 X-Flow-Control-Info: class=Exchange rcpts=1 size=17336 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CF19380F233Amattkenisondisneycom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable It's possible, but not easy to do. In our application, the individual custo= m sinks know which exceptions can rollback or are unrecoverable, but this d= oesn't work for the built-in sinks, and it doesn't take into account unexpe= cted failures. So, we control it manually with a JMX flag and by subclassin= g BasicTransactionSemantics. When the flag is set, and the transaction trie= s to rollback, it performs a commit instead (and directs the messages to a = failure channel). When a transaction is successful, the flag is reset. It's not the prettiest solution, but isn't a hack. It requires subclassing = the channel to provide a custom transaction, and overriding the default tra= nsaction behavior. Flume really doesn't make it easy to extend behavior of= any of the standard components. From: Paul Merry > Reply-To: "user@flume.apache.org" > Date: Thursday, February 6, 2014 12:28 AM To: "user@flume.apache.org" > Subject: RE: Channel management: messages that will never be delivered Thanks for the suggestion Ed, it's definitely something we could look at. I did find this ticket https://issues.apache.org/jira/browse/FLUME-2140 and= the linked discussion thread http://flume.markmail.org/thread/y3cks6hdgof3= kxu6#query:+page:1+mid:rx3zm53t4dhmqskk+state:results There are some suggestions for work arounds there, probably the use of fail= iover sink is most relevant but I'd be concerned for what might happen to l= egitimate messages in a situation where there is downtime or connection iss= ues with the endpoint. It seems we'd loose the correct channel retry logic = (in that scenario) and end up with messages that would need replaying. If there isn't much to add on the handling of 'bad messages' can anyone inf= orm on the handling of other messages in a batch with one or more of these = messages that will never deliver. Will they also not make it to their desti= nation or will they get rebatched? Also keen for anyone with an idea for how to clear these messages from a ch= annel once they are stuck, as the directory deletion can take good messages= down too. - Paul ________________________________ From: ed [edorsey@gmail.com] Sent: 05 February 2014 23:12 To: user@flume.apache.org Subject: Re: Channel management: messages that will never be delivered Hi Paul, Not sure if this would work for you but if you can error check prior to the= events reaching Elasticsearch you can handle this by writing a custom Inte= rceptor that validates your events. You can do more robust error checking = here than you can by just relying on the already existing event header fiel= ds as you'll have full access to the event header and body. Within the int= erceptor, if the event is not compatible with Elasticsearch you can add a b= oolean flag to the header of the event like "hasError". Then you can route= any events that have an error to a different "error" channel using a multi= plexing selector by checking for the hasError flag. The error channel can = either be connected to a NullSink or a FileRoll sink if you want to preserv= e the improperly formatted events. We've only used the memory channel so far so I'm afraid I can't comment on = the file channel specific questions you have. Hopefully someone on the lis= t with some more experience there can chime in. Best, Ed On Wed, Feb 5, 2014 at 5:34 PM, Paul Merry > wrote: Hi, We are using an Elasticsearch sink and have seen a file channel filling wit= h messages that will never be delivered as the format of the message is inc= ompatible with Elasticsearch itself. Example message from Flume logs: 24 Jan 2014 08:14:55,173 ERROR [SinkRunner-PollingRunner-DefaultSinkProcess= or] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver ev= ent. Exception follows. org.elasticsearch.indices.InvalidIndexNameException: [UpperCase-2014-01-23] Invalid index name [UpperCase-2014-01-23], must be lowercase In this case the index name comes from a header so we have a workaround usi= ng a multiplexing channel selector to detect and re-route messages based on= headers of this format. To clean up the channel this time we removed the data and checkpoint direct= ories, which is not ideal as we probably lost other messages in doing this. We are wary of similar situations occurring in future for messages that we = can't detect and divert in advance and so have a few questions: - What would be the recommended handling of this situation? - Is it possible to clear just these messages from the channel or does the = whole channel have to be deleted ? - Is there a way that we can divert these messages to another channel (dead= letter / invalid message style) ? Noting that they are not known to be pro= blematic until after an attempt is made to deliver them from the sink - What happens to other messages in a batch with a bad message ? Will they = also be stuck forever or will they be taken in another batch ? Thanks, Paul. ---------------------------- http://www.bbc.co.uk This e-mail (and any attachments) is confidential and may contain personal = views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance= on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this. --------------------- ---------------------------- http://www.bbc.co.uk This e-mail (and any attachments) is confidential and may contain personal = views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance= on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this. --------------------- --_000_CF19380F233Amattkenisondisneycom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
It's possible, but not e= asy to do. In our application, the individual custom sinks know which excep= tions can rollback or are unrecoverable, but this doesn't work for the buil= t-in sinks, and it doesn't take into account unexpected failures. So, we co= ntrol it manually with a JMX flag and by subclassing BasicTransactionS= emantics. When the flag is set, and the transaction tries to rollback, it p= erforms a commit instead (and directs the messages to a failure channel). W= hen a transaction is successful, the flag is reset. 

It's not the prettiest solution, but isn't a hack. It requires subc= lassing the channel to provide a custom transaction, and overriding the def= ault transaction behavior.  Flume really doesn't make it easy to exten= d behavior of any of the standard components.


=
From: Paul Merry <paul.merry@bbc.co.uk>
Reply-To: "user@flume.apache.org" <user@flume.apache.org>
Date: Thursday, February 6, 2014 12:28 AM
To: "user@flum= e.apache.org" <user@flume.a= pache.org>
Subject: RE: = Channel management: messages that will never be delivered

Thanks for the suggestion Ed, it's defin= itely something we could look at.

I did find this ticket https://issues.apache.org/jira/browse/FLUME-2140 and the linked discuss= ion thread http://flume.markmail.org/thread/y3cks6hdgof3kxu6#query:+page:1+mid:rx3zm53= t4dhmqskk+state:results

There are some suggestions for work arounds there, probably the use of fail= iover sink is most relevant but I'd be concerned for what might happen to l= egitimate messages in a situation where there is downtime or connection iss= ues with the endpoint. It seems we'd loose the correct channel retry logic (in that scenario) and end up w= ith messages that would need replaying.

If there isn't much to add on the handling of 'bad messages' can anyone inf= orm on the handling of other messages in a batch with one or more of these = messages that will never deliver. Will they also not make it to their desti= nation or will they get rebatched?

Also keen for anyone with an idea for how to clear these messages from a ch= annel once they are stuck, as the directory deletion can take good messages= down too.


- Paul


Fro= m: ed [edorsey@gmail.com]
<= b>Sent: 05 February 2014 23:12
To: user@flume.apache.org
Subject: Re: Channel man= agement: messages that will never be delivered

Hi Paul,

Not sure if this would work for you but if you can erro= r check prior to the events reaching Elasticsearch you can handle this by w= riting a custom Interceptor that validates your events.  You can do mo= re robust error checking here than you can by just relying on the already existing event header fields as you'll have full access to = the event header and body.  Within the interceptor, if the event is no= t compatible with Elasticsearch you can add a boolean flag to the header of= the event like "hasError".  Then you can route any events that have an error to a different "error" channel usi= ng a multiplexing selector by checking for the hasError flag.  The err= or channel can either be connected to a NullSink or a FileRoll sink if you = want to preserve the improperly formatted events.

We've only used the memory channel so far= so I'm afraid I can't comment on the file channel specific questions you h= ave.  Hopefully someone on the list with some more experience there ca= n chime in.

Best,

Ed


On Wed= , Feb 5, 2014 at 5:34 PM, Paul Merry <paul.merry@bb= c.co.uk> wrote:

<= /p>

Hi,

We are using an Elasticsearch sink and have seen a file channel filling wit= h messages that will never be delivered as the format of the message is inc= ompatible with Elasticsearch itself.

Example message from Flume logs:

24 Jan 2014 08:14:55,173 ERROR=
 [SinkRunner-PollingRunner-DefaultSinkProcessor] 
(org.apache.flume.Sink=
Runner$PollingRunner.run:160)  - Unable to deliver event. 
Exception fol=
lows.
org.elasticsearch.indices.InvalidIndexNameException: [UpperCase-2014-01-23]=
 
Invalid index name [UpperCase-2014-01-23], must be lowercase

In this case the index name comes from a header so we have a workaround usi= ng a multiplexing channel selector to detect and re-route messages based on= headers of this format.

To clean up the channel this time we removed the data and checkpoint direct= ories, which is not ideal as we probably lost other messages in doing this.=

We are wary of similar situations occurring in future for messages that we = can't detect and divert in advance and so have a few questions:

- What would be the recommended handling of this situation?

- Is it possible to clear just these messages from the channel or does the = whole channel have to be deleted ?

- Is there a way that we can divert these messages to another channel (dead= letter / invalid message style) ? Noting that they are not known to be pro= blematic until after an attempt is made to deliver them from the sink
- What happens to other messages in a batch with a bad message ? Will the= y also be stuck forever or will they be taken in another batch ?

Thanks,

Paul.

 

----------------------------
h= ttp://www.bbc.co.uk
This e-mail (and any attachments) is confidential and may contain personal = views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance= on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this.

---------------------


 

--------------= --------------

http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and may contain personal = views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance= on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or= received.
Further communication will signify your consent to this.

------------= ---------

--_000_CF19380F233Amattkenisondisneycom_--