flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will McQueen <w...@cloudera.com>
Subject Re: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG
Date Wed, 04 Jul 2012 04:23:45 GMT
Right, you can specify the FQCN, or the alias 'TIMESTAMP'

Cheers,
Will

On Jul 3, 2012, at 7:19 PM, Juhani Connolly <juhani_connolly@cyberagent.co.jp> wrote:

> To use the date related escape sequences you need to provide a timestamp header.
> 
> The easiest way to do this would be configuring a timestamp interceptor onto your source,
ours looks something like:
> 
> sender1.sources.tail.interceptors = ts
> sender1.sources.tail.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> 
> 
> On 07/04/2012 11:00 AM, Shara Shi wrote:
>> HI
>>  
>> I failed to use /%Y/%m/%d/ ESCAPE SEQUENCE to roll file in hdfs.
>> I got following error messages
>>  
>> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>> java.lang.NumberFormatException: null
>>         at java.lang.Long.parseLong(Long.java:375)
>>         at java.lang.Long.valueOf(Long.java:525)
>>         at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>         at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
>> org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.NumberFormatException: null
>>         at java.lang.Long.parseLong(Long.java:375)
>>         at java.lang.Long.valueOf(Long.java:525)
>>         at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>         at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>         ... 3 more
>> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>> java.lang.NumberFormatException: null
>>         at java.lang.Long.parseLong(Long.java:375)
>>         at java.lang.Long.valueOf(Long.java:525)
>>         at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>         at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
>> org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.NumberFormatException: null
>>         at java.lang.Long.parseLong(Long.java:375)
>>         at java.lang.Long.valueOf(Long.java:525)
>>         at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>         at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>         ... 3 more
>> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>> java.lang.NumberFormatException: null
>>         at java.lang.Long.parseLong(Long.java:375)
>>         at java.lang.Long.valueOf(Long.java:525)
>>  
>> =============
>>  
>> collector2.channels.ch2.type = memory
>>  
>> collector2.sources.avro-source1.channels             = ch2
>> collector2.sources.avro-source1.type = avro
>> collector2.sources.avro-source1.bind = 0.0.0.0
>> collector2.sources.avro-source1.port = 41414
>>  
>> collector2.sinks.hdfs.channel = ch2
>> collector2.sinks.hdfs.type= hdfs
>> collector2.sinks.hdfs.hdfs.path=hdfs://namenode:8020/data/log/%Y/%m/%d/
>> collector2.sinks.hdfs.batchsize=1
>> collector2.sinks.hdfs.runner.type=polling
>> collector2.sinks.hdfs.runner.polling.interval = 60
>>  
>> collector2.sinks.hdfs.hdfs.rollInterval = 300
>> collector2.sinks.hdfs.hdfs.rollSize = 0
>> collector2.sinks.hdfs.hdfs.rollCount = 0
>>  
>> collector2.channels = ch2
>> collector2.sources = avro-source1
>> collector2.sinks = hdfs
>>  
>>  
>> Ruihong Shi
>>  
> 
> 

Mime
View raw message