flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: ERROR hdfs.HDFSEventSink: process failed. ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
Date Fri, 27 Jul 2012 14:17:45 GMT
It does http://flume.apache.org/releases/1.1.0.html but the root of the
issue is that the source doesn't add the timestamp header to the event.
Also 1.2 has just been released which has Interceptors which I think can
add the timestamp for you.

Brock

On Thu, Jul 26, 2012 at 12:59 PM, khadar basha <khadarskb@gmail.com> wrote:

> Hi Brock.
>
> flume-1.1.0 doesn't support this escape sequence options?
>
> -Khadar
>
> On Thu, Jul 26, 2012 at 10:34 PM, Brock Noland <brock@cloudera.com> wrote:
>
>> The flume user doc here http://flume.apache.org/releases/1.2.0.html  has
>> many examples.
>>
>> Brock
>>
>>
>> On Thu, Jul 26, 2012 at 11:37 AM, khadar basha <khadarskb@gmail.com>wrote:
>>
>>> Hi Brock,
>>>
>>> could you please provide me the sample configuration?
>>>
>>> Thanks,
>>> Khadar
>>>
>>>
>>> On Thu, Jul 26, 2012 at 7:23 PM, Brock Noland <brock@cloudera.com>wrote:
>>>
>>>> If you want to use a bucketed paths your events need to have a
>>>> "timestamp" header which tail source does not appear to add.
>>>>
>>>> Brock
>>>>
>>>> On Thu, Jul 26, 2012 at 6:32 AM, khadar basha <khadarskb@gmail.com>wrote:
>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>> we are using flume-1.1.0-cdh4.0.1 . we have the following
>>>>> configurations in flume-conf.properties.
>>>>>
>>>>> agent1Test1.sources = seqGenSrc
>>>>> agent1Test1.channels = memoryChannel
>>>>> agent1Test1.sinks = loggerSink
>>>>>
>>>>> # For each one of the sources, the type is defined
>>>>> agent1Test1.sources.seqGenSrc.type = exec
>>>>> agent1Test1.sources.seqGenSrc.command= tail -f
>>>>> /tomcat/logs/catalina.out
>>>>>
>>>>>
>>>>> # The channel can be defined as follows.
>>>>> agent1Test1.sources.seqGenSrc.channels = memoryChannel
>>>>>
>>>>> # Each sink's type must be defined
>>>>> agent1Test1.sinks.loggerSink.type = hdfs
>>>>> agent1Test1.sinks.loggerSink.hdfs.path =
>>>>> hdfs://hdfshost:port/data/%Y/%m/%{host}/mylogs
>>>>>
>>>>> agent1Test1.sinks.loggerSink.hdfs.fileType = DataStream
>>>>>
>>>>> #Specify the channel the sink should use
>>>>> agent1Test1.sinks.loggerSink.channel = memoryChannel
>>>>>
>>>>> # Each channel's type is defined.
>>>>> agent1Test1.channels.memoryChannel.type = memory
>>>>>
>>>>> # Other config values specific to each type of channel(sink or source)
>>>>> # can be defined as well
>>>>> # In this case, it specifies the capacity of the memory channel
>>>>> agent1Test1.channels.memoryChannel.capacity = 1000
>>>>>
>>>>>
>>>>> I am getting the following exception. Anybody facing this issue.
>>>>>
>>>>>
>>>>> *2012-07-26 13:07:52,002 ERROR hdfs.HDFSEventSink: process failed*
>>>>> *java.lang.NumberFormatException: null*
>>>>> *        at java.lang.Long.parseLong(Long.java:375)*
>>>>> *        at java.lang.Long.valueOf(Long.java:525)*
>>>>> *        at
>>>>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>>>> *
>>>>> *        at
>>>>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>>>> *
>>>>> *        at
>>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>>>> *
>>>>> *        at
>>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>>> *
>>>>> *        at
>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
>>>>> *        at java.lang.Thread.run(Thread.java:662)*
>>>>> *2012-07-26 13:07:52,003 ERROR flume.SinkRunner: Unable to deliver
>>>>> event. Exception follows.*
>>>>> *org.apache.flume.EventDeliveryException:
>>>>> java.lang.NumberFormatException: null*
>>>>> *        at
>>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>>>>> *
>>>>> *        at
>>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>>> *
>>>>> *        at
>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
>>>>> *        at java.lang.Thread.run(Thread.java:662)*
>>>>> *Caused by: java.lang.NumberFormatException: null*
>>>>> *        at java.lang.Long.parseLong(Long.java:375)*
>>>>> *        at java.lang.Long.valueOf(Long.java:525)*
>>>>> *        at
>>>>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>>>> *
>>>>> *        at
>>>>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>>>> *
>>>>> *        at
>>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>>>> *
>>>>> *        ... 3 more*
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Khadar
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Apache MRUnit - Unit testing MapReduce -
>>>> http://incubator.apache.org/mrunit/
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Khadar
>>>
>>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>
>
> --
> Thanks,
> Khadar
>
>


-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message