Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C8F4DDD26 for ; Thu, 26 Jul 2012 17:05:07 +0000 (UTC) Received: (qmail 71247 invoked by uid 500); 26 Jul 2012 17:05:07 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 71205 invoked by uid 500); 26 Jul 2012 17:05:07 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 71178 invoked by uid 99); 26 Jul 2012 17:05:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jul 2012 17:05:06 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of brock@cloudera.com designates 209.85.212.51 as permitted sender) Received: from [209.85.212.51] (HELO mail-vb0-f51.google.com) (209.85.212.51) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jul 2012 17:05:01 +0000 Received: by vbip1 with SMTP id p1so1822848vbi.38 for ; Thu, 26 Jul 2012 10:04:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:x-gm-message-state; bh=d6tsnaCebWOTZBp/C5/kD4FJ8lhm5XI8dCi3NCstkuw=; b=Lte3hz0KTxWc/SEY1+lRXD+ggKox8DSZbsS6LzcF0hZx3iyHrmifmMd4Wi1+Gyh1O2 ijmqfqAxgCDMimiQcn5NHVnYC4gfQBe3RevpPse79y1gnQPgCHQaBL89BAVOa6SNJqgP WqiuabaE2OGFFcPUqJzcz/WEMOtF7wxQis1enbINKZFzoPfMMV5YE4gYv8sZsEueebQn Faq9EhNYSY8E+RHmUF30qzRgaqxvh9DtM79/YTqHJMKhHsniajZW95IkA4qRod7ttpf/ ZsPLjzCkEyYekOy69RmsW36EskHG/xDbPte/ap7SmEeLd9R+KD8h6EQaCFwpvQUj7e8t BTLA== Received: by 10.220.149.130 with SMTP id t2mr13168059vcv.8.1343322280768; Thu, 26 Jul 2012 10:04:40 -0700 (PDT) MIME-Version: 1.0 Received: by 10.58.95.42 with HTTP; Thu, 26 Jul 2012 10:04:19 -0700 (PDT) In-Reply-To: References: From: Brock Noland Date: Thu, 26 Jul 2012 12:04:19 -0500 Message-ID: Subject: Re: ERROR hdfs.HDFSEventSink: process failed. ERROR flume.SinkRunner: Unable to deliver event. Exception follows. To: user@flume.apache.org Cc: dontariq@gmail.com Content-Type: multipart/alternative; boundary=f46d043c812c5d342704c5be97ba X-Gm-Message-State: ALoCoQlPZuBER7V6SaJGV11a+e0FqbRAnpBoII9VYD62/hh48D5x5RwyJijzpTiDjPuT9ggPBzWC X-Virus-Checked: Checked by ClamAV on apache.org --f46d043c812c5d342704c5be97ba Content-Type: text/plain; charset=ISO-8859-1 The flume user doc here http://flume.apache.org/releases/1.2.0.html has many examples. Brock On Thu, Jul 26, 2012 at 11:37 AM, khadar basha wrote: > Hi Brock, > > could you please provide me the sample configuration? > > Thanks, > Khadar > > > On Thu, Jul 26, 2012 at 7:23 PM, Brock Noland wrote: > >> If you want to use a bucketed paths your events need to have a >> "timestamp" header which tail source does not appear to add. >> >> Brock >> >> On Thu, Jul 26, 2012 at 6:32 AM, khadar basha wrote: >> >>> >>> Hi >>> >>> we are using flume-1.1.0-cdh4.0.1 . we have the following configurations >>> in flume-conf.properties. >>> >>> agent1Test1.sources = seqGenSrc >>> agent1Test1.channels = memoryChannel >>> agent1Test1.sinks = loggerSink >>> >>> # For each one of the sources, the type is defined >>> agent1Test1.sources.seqGenSrc.type = exec >>> agent1Test1.sources.seqGenSrc.command= tail -f /tomcat/logs/catalina.out >>> >>> >>> # The channel can be defined as follows. >>> agent1Test1.sources.seqGenSrc.channels = memoryChannel >>> >>> # Each sink's type must be defined >>> agent1Test1.sinks.loggerSink.type = hdfs >>> agent1Test1.sinks.loggerSink.hdfs.path = >>> hdfs://hdfshost:port/data/%Y/%m/%{host}/mylogs >>> >>> agent1Test1.sinks.loggerSink.hdfs.fileType = DataStream >>> >>> #Specify the channel the sink should use >>> agent1Test1.sinks.loggerSink.channel = memoryChannel >>> >>> # Each channel's type is defined. >>> agent1Test1.channels.memoryChannel.type = memory >>> >>> # Other config values specific to each type of channel(sink or source) >>> # can be defined as well >>> # In this case, it specifies the capacity of the memory channel >>> agent1Test1.channels.memoryChannel.capacity = 1000 >>> >>> >>> I am getting the following exception. Anybody facing this issue. >>> >>> >>> *2012-07-26 13:07:52,002 ERROR hdfs.HDFSEventSink: process failed* >>> *java.lang.NumberFormatException: null* >>> * at java.lang.Long.parseLong(Long.java:375)* >>> * at java.lang.Long.valueOf(Long.java:525)* >>> * at >>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220) >>> * >>> * at >>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310) >>> * >>> * at >>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402) >>> * >>> * at >>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) >>> * >>> * at >>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)* >>> * at java.lang.Thread.run(Thread.java:662)* >>> *2012-07-26 13:07:52,003 ERROR flume.SinkRunner: Unable to deliver >>> event. Exception follows.* >>> *org.apache.flume.EventDeliveryException: >>> java.lang.NumberFormatException: null* >>> * at >>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469) >>> * >>> * at >>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) >>> * >>> * at >>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)* >>> * at java.lang.Thread.run(Thread.java:662)* >>> *Caused by: java.lang.NumberFormatException: null* >>> * at java.lang.Long.parseLong(Long.java:375)* >>> * at java.lang.Long.valueOf(Long.java:525)* >>> * at >>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220) >>> * >>> * at >>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310) >>> * >>> * at >>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402) >>> * >>> * ... 3 more* >>> >>> -- >>> Thanks, >>> Khadar >>> >>> >> >> >> -- >> Apache MRUnit - Unit testing MapReduce - >> http://incubator.apache.org/mrunit/ >> > > > > -- > Thanks, > Khadar > > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/ --f46d043c812c5d342704c5be97ba Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable The flume user doc here=A0http://flume.apache.org/releases/1.2.0.html =A0has many examples= .

Brock

On Thu, Jul 26= , 2012 at 11:37 AM, khadar basha <khadarskb@gmail.com> wro= te:
Hi Brock,

could you pleas= e provide me the sample configuration?

Thanks,
Khadar


On Thu= , Jul 26, 2012 at 7:23 PM, Brock Noland <brock@cloudera.com> wrote:
If you want to use a bucketed paths you= r events need to have a "timestamp" header which tail source does= not appear to add.

Brock

On = Thu, Jul 26, 2012 at 6:32 AM, khadar basha <khadarskb@gmail.com><= /span> wrote:

Hi=A0

<= /div>
we are using flume-1.1.0-cdh4.0.1 . we have the following configu= rations in flume-conf.properties.

agent1Test1.sources =3D seqGenSrc
agent1= Test1.channels =3D memoryChannel
agent1Test1.sinks =3D loggerSink

# For each o= ne of the sources, the type is defined
agent1Test1.sources.seqGen= Src.type =3D exec
agent1Test1.sources.seqGenSrc.command=3D tail -= f /tomcat/logs/catalina.out


# The channel can be defined as follows.=
agent1Test1.sources.seqGenSrc.channels =3D memoryChannel

# Each sink's type must be defined
agent1Te= st1.sinks.loggerSink.type =3D hdfs
agent1Test1.sinks.loggerSink.hdfs.path =3D hdfs://hdfshost:port/data/%= Y/%m/%{host}/mylogs

agent1Test1.sinks.loggerSink.h= dfs.fileType =3D DataStream

#Specify the channel t= he sink should use
agent1Test1.sinks.loggerSink.channel =3D memoryChannel

<= /div>
# Each channel's type is defined.
agent1Test1.chann= els.memoryChannel.type =3D memory

# Other config v= alues specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the ca= pacity of the memory channel
agent1Test1.channels.memoryChannel.c= apacity =3D 1000


I am getting= the following exception. Anybody facing this issue.


20= 12-07-26 13:07:52,002 ERROR hdfs.HDFSEventSink: process failed
java.lang.NumberFormatExce= ption: null
=A0 =A0 =A0 =A0 at java.lang.Long.par= seLong(Long.java:375)
=A0 =A0 =A0 =A0 at java.lang.Long.valueOf(Long.java:525)=
=A0 =A0 =A0 =A0 at org.apache.flume.f= ormatter.output.BucketPath.replaceShorthand(BucketPath.java:220)=
=A0 =A0 =A0 =A0 at org.a= pache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
=A0 =A0 =A0 =A0 at org.apache.flume.s= ink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
=A0 =A0 =A0 =A0 at org.apache.flum= e.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
=A0 =A0 =A0 =A0 at org.apache.flume.S= inkRunner$PollingRunner.run(SinkRunner.java:147)
=A0 =A0 =A0 =A0 at java.lang.Thread.run(= Thread.java:662)
2012-07-26 13:07:52,003 ERROR flume.S= inkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryExce= ption: java.lang.NumberFormatException: null
=A0 =A0 =A0 =A0 at org.apache.flume.s= ink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
=A0 =A0 =A0 =A0 at org.apache.flum= e.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
=A0 =A0 =A0 =A0 at org.apache.flume.S= inkRunner$PollingRunner.run(SinkRunner.java:147)
=A0 =A0 =A0 =A0 at java.lang.Thread.run(= Thread.java:662)
Caused by: java.lang.NumberFormatExce= ption: null
= =A0 =A0 =A0 =A0 at java.lang.Long.parseLong(Long.java:375)
=A0 =A0 =A0 =A0 at java.lang.Long.val= ueOf(Long.java:525)
=A0 =A0 =A0 =A0 at org.apache.flume.formatter.output.BucketPath.repla= ceShorthand(BucketPath.java:220)
=A0 =A0 =A0 =A0 at org.apache.flume.f= ormatter.output.BucketPath.escapeString(BucketPath.java:310)
=A0 =A0 =A0 =A0 at org.apach= e.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)<= /span>
=A0 =A0 =A0 =A0 ... 3 more=

--
Than= ks,
Khadar




--
Apache MRUnit - Unit testing Ma= pReduce - http://incubator.apache.org/mrunit/



--
Thanks,
Kha= dar




--
Apache MRUni= t - Unit testing MapReduce - http://incubator.apache.org/mrunit/
--f46d043c812c5d342704c5be97ba--