Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2D1DB1097A for ; Thu, 24 Oct 2013 09:17:17 +0000 (UTC) Received: (qmail 5218 invoked by uid 500); 24 Oct 2013 09:17:15 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 4756 invoked by uid 500); 24 Oct 2013 09:17:08 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 4152 invoked by uid 99); 24 Oct 2013 09:17:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Oct 2013 09:17:05 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of deepak.subhramanian@gmail.com designates 209.85.212.182 as permitted sender) Received: from [209.85.212.182] (HELO mail-wi0-f182.google.com) (209.85.212.182) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Oct 2013 09:16:59 +0000 Received: by mail-wi0-f182.google.com with SMTP id ez12so2194385wid.3 for ; Thu, 24 Oct 2013 02:16:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=wplz/25Kqz+Cs6CASHVFi3UwovyVUtU7okec0JooMRY=; b=WUxR0P9LxJHdSnx2nXVzHCazNwmI+CgygqGNYRbAa89cs+YjUGfnwQEaQTFoCVpmL+ yZfkQXRhW/+cx9aIOUEz0qG7bWV6cI9+OfLQynapJoT9TF/Sxlk6WsWWfZFKaoQ9Ol6w gMt/4XjxTkQyfaQb/elt2mbb8XCnNT/UptXpMC1CJuNvZKrDbcQ0JoDfSvsJOus8rCng TSUM3P9twiIUIj8hxknwKquwDdfgvA+TGPKCS3QxD3QWoUyNRvy4Q9QmpVX07HXofLXt 6pNXH6q3uJpozvZkGwZxPI3oYKOkVkzyagvJ9HXtyX8ecjyalLx8Qrm9vMqPnmHSrGx4 rzsg== MIME-Version: 1.0 X-Received: by 10.194.216.103 with SMTP id op7mr45380wjc.93.1382606198626; Thu, 24 Oct 2013 02:16:38 -0700 (PDT) Received: by 10.216.49.131 with HTTP; Thu, 24 Oct 2013 02:16:38 -0700 (PDT) In-Reply-To: References: Date: Thu, 24 Oct 2013 10:16:38 +0100 Message-ID: Subject: Re: Not able to set date %d%m%Y in Sink Directory name From: Deepak Subhramanian To: user@flume.apache.org Content-Type: multipart/alternative; boundary=089e014935e855622f04e979179e X-Virus-Checked: Checked by ClamAV on apache.org --089e014935e855622f04e979179e Content-Type: text/plain; charset=ISO-8859-1 I am using a SimpleHTTPEventHandler which implements HTTPSourceHandler which reads a log message from HTTP Request and create a Flume event. May be I have to add the header to the event with the timestamp. String message = (String)request.getParameter("log"); //LOG.info(message ); LOG.info("Msg Read" ); Event event = EventBuilder.withBody(message.getBytes()); eventList.add(event); On Wed, Oct 23, 2013 at 6:40 PM, David Sinclair < dsinclair@chariotsolutions.com> wrote: > The sink is dependent on a header with the key "timestamp" being present > in the event for this to work. What source are you using? > > > On Wed, Oct 23, 2013 at 11:25 AM, Deepak Subhramanian < > deepak.subhramanian@gmail.com> wrote: > >> Hi , >> >> I am trying to store my logs in folders named with date for my file_roll >> and hdfssink. For some reason when I pass %d%m%Y in the sink directory it >> is not working . Any thoughts . >> >> My Flume source is a simple HTTP Handler extended from HTTPSourceHandler >> >> tier1.sinks.filesink1.type = file_roll >> tier1.sinks.filesink1.channel = c2 >> tier1.sinks.filesink1.batchSize = 100000 >> tier1.sinks.filesink1.rollInterval = 0 >> #tier1.sinks.filesink1.rollInterval = 86400 >> tier1.sinks.filesink1.sink.directory = /projects/test/logs/%d%m%Y >> >> >> 13/10/23 16:20:36 ERROR flume.SinkRunner: Unable to deliver event. >> Exception follows. >> org.apache.flume.EventDeliveryException: Failed to open file >> /projects/test/logs/%d%m%Y/1382541251307-1 while delivering event >> at >> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177) >> at >> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) >> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) >> at java.lang.Thread.run(Unknown Source) >> Caused by: java.io.FileNotFoundException: >> /projects/ETHAN_FLUME/logs/%d%m%Y/1382541251307-1 (No such file or >> directory) >> at java.io.FileOutputStream.open(Native Method) >> at java.io.FileOutputStream.(Unknown Source) >> at java.io.FileOutputStream.(Unknown Source) >> at >> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169) >> >> >> Thanks, >> Deepak Subhramanian >> > > -- Deepak Subhramanian --089e014935e855622f04e979179e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I am using a SimpleHTTPEventHandler which implements HTTPS= ourceHandler which reads a log message from HTTP Request and create a Flume= event. May be I have to add the header to the event with the timestamp.

String message =3D=A0 (String)request.getParameter("log");=A0

<= /span> //LOG.info(message );

LOG.info("Msg R= ead" );


Event event =3D EventBuilder.withBody(message.getBytes());


eventList.add(event);=A0

On Wed, Oct 23, 2013 at 6:40 PM, David Sinclair= <dsinclair@chariotsolutions.com> wrote:
The sink is dependent on a = header with the key "timestamp" being present in the event for th= is to work. What source are you using?


=
On Wed, Oct 23, 2013 at 11:25 AM, Deepak Subhram= anian <deepak.subhramanian@gmail.com> wrote:
Hi ,=A0

= I am trying to store my logs in folders named with date for my file_roll an= d hdfssink. For some reason when I pass %d%m%Y in the sink directory it is = not working . Any thoughts .=A0

My Flume source is a simple HTTP Handler extended from = HTTPSourceHandler

tier1.sinks.filesink1.type =3D file_roll
tier1.sinks.filesink1.channel =3D c2
tier1.sinks.filesink1.batch= Size =3D 100000
tier1.sinks.filesink1.rollInterval =3D 0
#tier1.sinks.filesink1.rollInterval =3D 86400
tier1.sinks.filesink1.sink.directory =3D /projects/test/logs/%d%m%Y


13/10/23 16:20:36 ERROR flume.Si= nkRunner: Unable to deliver event. Exception follows.
org.apache.= flume.EventDeliveryException: Failed to open file /projects/test/logs/%d%m%= Y/1382541251307-1 while delivering event
at org.apache.flume.sink.= RollingFileSink.process(RollingFileSink.java:177)
at org.apache.flume.sink.DefaultSinkProcessor= .process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkR= unner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.FileNotFoundException: /projects/ETHAN_FLUME/logs/%= d%m%Y/1382541251307-1 (No such file or directory)
at java.io.FileOutputStream.open(Native Metho= d)
at java.io.FileOutputStre= am.<init>(Unknown Source)
at java.io.FileOutputStream.<init>(Unknown Source)
at org.apache.flume.sink.= RollingFileSink.process(RollingFileSink.java:169)


Thanks,=A0
Deepak Subhrama= nian




--
= Deepak Subhramanian
--089e014935e855622f04e979179e--