flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manohar CS <Manohar...@itcinfotech.com>
Subject RE: Notification support from flume?
Date Sat, 06 Dec 2014 06:31:35 GMT
Thanks Hari for your response.


My requirement goes like this -


1) There are bunch of files coming in at regular intervals (hourly or daily) in my spoolDir

2) I wan tthem to be moved into HDFS via HDFS sink using reg-ex like /target/%Y-%M%D so each
day file gets into different destination HDFS

3) Now once this flume completes copying files , I want to kick off my MR job.


Thanks,

Manohar

________________________________
From: Hari Shreedharan <hshreedharan@cloudera.com>
Sent: Saturday, December 6, 2014 7:16 AM
To: user@flume.apache.org
Cc: user@flume.apache.org
Subject: Re: Notification support from flume?

Looking at .COMPLETED is not an indication that the data has been written out to HDFS. As
of now, unfortunately there is no way to tag an event as coming from a specific file. I can't
think of a way to do this in a fool-proof way off the top of my mind. What is your use-case,
there might be another way to do the same thing?

Thanks,
Hari



On Fri, Dec 5, 2014 at 4:19 AM, Manohar CS <Manohar.CS@itcinfotech.com<mailto:Manohar.CS@itcinfotech.com>>
wrote:
Hi All,


I wanted to know if there is a way of notification mechanism or some way of finding out if
flume has finished transfer of certain file from spoolDir to HDFS sink? We know by looking
at .COMPLETED files in spoolDir we can assume its completed but wanted to know if there is
more reliable way of call back mechanism ?




Thanks,
Manohar.



Please consider the environment before printing this e-mail

Disclaimer: This  communication  is  for the exclusive use of the intended recipient(s) and
 shall  not attach any liability on the originator or ITC Infotech India Ltd./its  Holding
company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of
this e-mail are intended for your use only and it shall  not be forwarded to any third party,
without first obtaining written authorization from the originator or ITC Infotech India Ltd./
its Holding company/its  Subsidiaries/ its Group Companies. It may contain information which
is confidential and legally privileged and the same shall not be used or dealt with  by any
 third  party  in  any manner whatsoever without the specific consent  of  ITC  Infotech India
Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.

Please consider the environment before printing this e-mail

Disclaimer: This  communication  is  for the exclusive use of the intended recipient(s) and
 shall  not attach any liability on the originator or ITC Infotech India Ltd./its  Holding
company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of
this e-mail are intended for your use only and it shall  not be forwarded to any third party,
without first obtaining written authorization from the originator or ITC Infotech India Ltd./
its Holding company/its  Subsidiaries/ its Group Companies. It may contain information which
is confidential and legally privileged and the same shall not be used or dealt with  by any
 third  party  in  any manner whatsoever without the specific consent  of  ITC  Infotech India
Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.

Mime
View raw message