flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakati, Pavan" <pavan.jak...@cgi.com>
Subject RE: How to read content from files which are in nested directory
Date Mon, 20 Jul 2015 12:37:39 GMT
Hi Ashish 

Thanks for the help. Can you pls help with procedure to update or pull the patch into my system
. Thanks. 

Regards,
PavaN


-----Original Message-----
From: Ashish [mailto:paliwalashish@gmail.com] 
Sent: Saturday, July 18, 2015 12:44 AM
To: user@flume.apache.org
Subject: Re: How to read content from files which are in nested directory

One more way to achieve the same is using this patch
https://issues.apache.org/jira/browse/FLUME-1899

On Fri, Jul 17, 2015 at 7:14 AM, Johny Rufus <jrufus@cloudera.com> wrote:
> The spooling directory source as of now supports only reading from a 
> flat directory and wont read files from subdirectories.
> You could write an external script that transfers all the files in all 
> the date directories to a common directory which spooling source 
> points to. (If this fits your use case)
>
> Thanks,
> Rufus
>
> On Fri, Jul 17, 2015 at 6:40 AM, Jakati, Pavan <pavan.jakati@cgi.com> wrote:
>>
>> Hi Folks,
>>
>>
>>
>> I am consuming Apache flume to read logs which are stored in 
>> following format , logs under date directory . How do I read the logs 
>> using spooldir source.
>>
>>
>>
>>
>>
>> Directory Structure :
>>
>>
>>
>>
>>
>> flume]# ll /ADAPTORS/MAIL_CONNECT/logs/
>>
>> total 0
>>
>> drwxr-xr-x 2 root root 29 Jun  3 18:04 06-03-15
>>
>> drwxr-xr-x 2 root root 56 Jun  4 18:16 06-04-15
>>
>> drwxr-xr-x 2 root root 29 Jun  5 14:03 06-05-15
>>
>> drwxr-xr-x 2 root root 29 Jun  8 12:43 06-08-15
>>
>> drwxr-xr-x 2 root root 29 Jun  9 14:47 06-09-15
>>
>> drwxr-xr-x 2 root root 29 Jun 10 18:49 06-10-15
>>
>> drwxr-xr-x 2 root root 29 Jun 11 17:22 06-11-15
>>
>> drwxr-xr-x 2 root root 29 Jun 12 11:37 06-12-15
>>
>> drwxr-xr-x 2 root root 29 Jun 15 11:39 06-15-15
>>
>> drwxr-xr-x 2 root root 29 Jun 24 10:45 06-23-15
>>
>> drwxr-xr-x 2 root root 29 Jun 24 10:35 06-24-15
>>
>> drwxr-xr-x 2 root root 29 Jun 25 17:28 06-25-15
>>
>>
>>
>>
>>
>> My current Configuration :
>>
>> agent1.sources.source1.type = spooldir
>>
>> agent1.sources.source1.channels = channel1
>>
>> agent1.sources.source1.spoolDir = 
>> /ADAPTORS/MAIL_CONNECT/logs/07-17-15
>>
>> agent1.sources.source1.fileHeader = true
>>
>>
>>
>>
>>
>> Pls suggest if i need to use some other source type .
>>
>>
>>
>> Can I use any script inside conf file to change the directory 
>> dynamically . Thanks
>>
>>
>>
>>
>>
>> Regards,
>>
>> PavaN
>>
>>
>
>



--
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Mime
View raw message