Return-Path: X-Original-To: apmail-incubator-flume-user-archive@minotaur.apache.org Delivered-To: apmail-incubator-flume-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E0ECF9121 for ; Wed, 4 Jul 2012 02:20:17 +0000 (UTC) Received: (qmail 33606 invoked by uid 500); 4 Jul 2012 02:20:17 -0000 Delivered-To: apmail-incubator-flume-user-archive@incubator.apache.org Received: (qmail 33568 invoked by uid 500); 4 Jul 2012 02:20:17 -0000 Mailing-List: contact flume-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: flume-user@incubator.apache.org Delivered-To: mailing list flume-user@incubator.apache.org Received: (qmail 33560 invoked by uid 99); 4 Jul 2012 02:20:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jul 2012 02:20:17 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of juhani_connolly@cyberagent.co.jp designates 210.134.177.71 as permitted sender) Received: from [210.134.177.71] (HELO ipl2.aams0.jp) (210.134.177.71) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jul 2012 02:20:07 +0000 Received: from [192.168.177.168] (61-121-214-170.bitcat.net [61.121.214.170] (may be forged)) (authenticated bits=0) by ipl2.aams0.jp (Sentrion-MTA-4.0.2/Switch-3.2.5) with ESMTP id q642JfH6008098 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 4 Jul 2012 11:19:42 +0900 Message-ID: <4FF3A839.6070306@cyberagent.co.jp> Date: Wed, 04 Jul 2012 11:19:37 +0900 From: Juhani Connolly User-Agent: Mozilla/5.0 (X11; Linux i686; rv:13.0) Gecko/20120605 Thunderbird/13.0 MIME-Version: 1.0 To: flume-user@incubator.apache.org Subject: Re: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG References: <000c01cd5988$cd642890$682c79b0$@com> In-Reply-To: <000c01cd5988$cd642890$682c79b0$@com> Content-Type: multipart/alternative; boundary="------------010703030002070600040508" X-AAMS0-Virus-Status: clean X-AAMS0-Virus-Status: clean X-Aams0-Archive-Original-S: YES This is a multi-part message in MIME format. --------------010703030002070600040508 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit To use the date related escape sequences you need to provide a timestamp header. The easiest way to do this would be configuring a timestamp interceptor onto your source, ours looks something like: sender1.sources.tail.interceptors = ts sender1.sources.tail.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder On 07/04/2012 11:00 AM, Shara Shi wrote: > > HI > > I failed to use /%Y/%m/%d/ ESCAPE SEQUENCE to roll file in hdfs. > > I got following error messages > > 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed > > java.lang.NumberFormatException: null > > at java.lang.Long.parseLong(Long.java:375) > > at java.lang.Long.valueOf(Long.java:525) > > at > org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220) > > at > org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310) > > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402) > > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > > at > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > at java.lang.Thread.run(Thread.java:662) > > 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. > Exception follows. > > org.apache.flume.EventDeliveryException: > java.lang.NumberFormatException: null > > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469) > > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > > at > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > at java.lang.Thread.run(Thread.java:662) > > Caused by: java.lang.NumberFormatException: null > > at java.lang.Long.parseLong(Long.java:375) > > at java.lang.Long.valueOf(Long.java:525) > > at > org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220) > > at > org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310) > > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402) > > ... 3 more > > 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed > > java.lang.NumberFormatException: null > > at java.lang.Long.parseLong(Long.java:375) > > at java.lang.Long.valueOf(Long.java:525) > > at > org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220) > > at > org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310) > > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402) > > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > > at > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > at java.lang.Thread.run(Thread.java:662) > > 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. > Exception follows. > > org.apache.flume.EventDeliveryException: > java.lang.NumberFormatException: null > > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469) > > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > > at > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > at java.lang.Thread.run(Thread.java:662) > > Caused by: java.lang.NumberFormatException: null > > at java.lang.Long.parseLong(Long.java:375) > > at java.lang.Long.valueOf(Long.java:525) > > at > org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220) > > at > org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310) > > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402) > > ... 3 more > > 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed > > java.lang.NumberFormatException: null > > at java.lang.Long.parseLong(Long.java:375) > > at java.lang.Long.valueOf(Long.java:525) > > ============= > > collector2.channels.ch2.type = memory > > collector2.sources.avro-source1.channels = ch2 > > collector2.sources.avro-source1.type = avro > > collector2.sources.avro-source1.bind = 0.0.0.0 > > collector2.sources.avro-source1.port = 41414 > > collector2.sinks.hdfs.channel = ch2 > > collector2.sinks.hdfs.type= hdfs > > *collector2.sinks.hdfs.hdfs.path=hdfs://namenode:8020/data/log/%Y/%m/%d/* > > collector2.sinks.hdfs.batchsize=1 > > collector2.sinks.hdfs.runner.type=polling > > collector2.sinks.hdfs.runner.polling.interval = 60 > > collector2.sinks.hdfs.hdfs.rollInterval = 300 > > collector2.sinks.hdfs.hdfs.rollSize = 0 > > collector2.sinks.hdfs.hdfs.rollCount = 0 > > collector2.channels = ch2 > > collector2.sources = avro-source1 > > collector2.sinks = hdfs > > Ruihong Shi > --------------010703030002070600040508 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
To use the date related escape sequences you need to provide a timestamp header.

The easiest way to do this would be configuring a timestamp interceptor onto your source, ours looks something like:

sender1.sources.tail.interceptors = ts
sender1.sources.tail.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder


On 07/04/2012 11:00 AM, Shara Shi wrote:

HI

 

I failed to use /%Y/%m/%d/ ESCAPE SEQUENCE to roll file in hdfs.

I got following error messages

 

12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed

java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)

        at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)

        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:662)

12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.

org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)

        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)

        at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)

        ... 3 more

12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed

java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)

        at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)

        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:662)

12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.

org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)

        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)

        at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)

        ... 3 more

12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed

java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

 

=============

 

collector2.channels.ch2.type = memory

 

collector2.sources.avro-source1.channels = ch2

collector2.sources.avro-source1.type = avro

collector2.sources.avro-source1.bind = 0.0.0.0

collector2.sources.avro-source1.port = 41414

 

collector2.sinks.hdfs.channel = ch2

collector2.sinks.hdfs.type= hdfs

collector2.sinks.hdfs.hdfs.path=hdfs://namenode:8020/data/log/%Y/%m/%d/

collector2.sinks.hdfs.batchsize=1

collector2.sinks.hdfs.runner.type=polling

collector2.sinks.hdfs.runner.polling.interval = 60

 

collector2.sinks.hdfs.hdfs.rollInterval = 300

collector2.sinks.hdfs.hdfs.rollSize = 0

collector2.sinks.hdfs.hdfs.rollCount = 0

 

collector2.channels = ch2

collector2.sources = avro-source1

collector2.sinks = hdfs

 

 

Ruihong Shi

 



--------------010703030002070600040508--