Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
References: <20668C94-4670-4407-B852-CB47A58FC60F@nextdoor.com>
 <9C2AF22B-620C-4D76-9042-89DE7A973E87@nextdoor.com>
Mime-Version: 1.0 (1.0)
In-Reply-To: <9C2AF22B-620C-4D76-9042-89DE7A973E87@nextdoor.com>
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Message-Id: <7CCE47FE-30D5-4015-85D3-5668022D6BBD@apache.org>
Cc: "user@flume.apache.org" <user@flume.apache.org>
From: Mike Percy <mpercy@apache.org>
Subject: Re: How to get a bad message out of the channel?
Date: Fri, 10 May 2013 18:30:48 -0700
To: "user@flume.apache.org" <user@flume.apache.org>

Hook up a HDFS sink to them that doesn't use %Y, %m, etc in the configured p=
ath.

HTH,
Mike

On May 10, 2013, at 11:00 AM, Matt Wise <matt@nextdoor.com> wrote:

> Eek, this was worse than I thought. Turns out message continued to be adde=
d to the channels, but no transactions could complete to take messages out o=
f the channel. I've moved the file channels out of the way and restarted the=
 service for now ... but how can I recover the rest of the data in these fil=
echannels?=20
>=20
> On May 10, 2013, at 10:29 AM, Matt Wise <matt@nextdoor.com> wrote:
>=20
>> We were messing around with a few settings today and ended up getting a f=
ew messages into our channel that are bad (corrupt time field). How can I cl=
ear them out?
>>=20
>>> 10 May 2013 17:28:26,920 ERROR [SinkRunner-PollingRunner-DefaultSinkProc=
essor] (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deli=
ver event. Exception follows.
>>> org.apache.flume.EventDeliveryException: java.lang.RuntimeException: Flu=
me wasn't able to parse timestamp header in the event to resolve time based b=
ucketing. Please check that you're correctly populating timestamp header (fo=
r example using TimestampInterceptor source interceptor).
>>>    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.jav=
a:461)
>>>    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProc=
essor.java:68)
>>>    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)=

>>>    at java.lang.Thread.run(Thread.java:679)
>>> Caused by: java.lang.RuntimeException: Flume wasn't able to parse timest=
amp header in the event to resolve time based bucketing. Please check that y=
ou're correctly populating timestamp header (for example using TimestampInte=
rceptor source interceptor).
>>>    at org.apache.flume.formatter.output.BucketPath.replaceShorthand(Buck=
etPath.java:160)
>>>    at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPa=
th.java:343)
>>>    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.jav=
a:392)
>>>    ... 3 more
>>> Caused by: java.lang.NumberFormatException: null
>>>    at java.lang.Long.parseLong(Long.java:401)
>>>    at java.lang.Long.valueOf(Long.java:535)
>>>    at org.apache.flume.formatter.output.BucketPath.replaceShorthand(Buck=
etPath.java:158)
>>>    ... 5 more
>>=20
>> This message just keeps repeating over and over again.. new events are co=
ming through just fine.
>=20