apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Priyanka Gugale <priya...@datatorrent.com>
Subject Re: Multiple directories
Date Fri, 08 Jul 2016 07:24:41 GMT
In fact you can take a look at FSInputModule in Malhar. This module will
filter as well as read all file blocks (files matching regex from all input
directories) for you. Check this json based application for sample usage:
https://github.com/apache/apex-malhar/tree/master/apps/filecopy

Only catch is as I understand from previous mail communication from you,
you need to read rectories serially i.e. one after the other, this is not
directly supported but can you check if you can modify sanner thread in
FileSplitterIunput to serve your needs.

-Priyanka



On Fri, Jul 8, 2016 at 11:43 AM, Priyanka Gugale <priyanka@datatorrent.com>
wrote:

> Hi,
>
> Take a look at TimeBasedDirectoryScanner in FileSplitterInput, this
> scanner accepts list of files/directories to scan. Also it accepts regex to
> filter on file names. I think you can pick ides on how to scan multiple
> directories from there.
>
> -Priyanka
>
> On Thu, Jul 7, 2016 at 6:59 PM, Mukkamula, Suryavamshivardhan (CWM-NR) <
> suryavamshivardhan.mukkamula@rbc.com> wrote:
>
>> Hi Yunhan,
>>
>>
>>
>> This example I am already using for reading the data from multiple
>> directories in parallel. Hear each directory is given to an operator in
>> parallel.
>>
>>
>>
>> My requirement is I would like add multiple directories to a single
>> operator.
>>
>>
>>
>> Regards,
>>
>> Surya Vamshi
>>
>>
>>
>> *From:* Yunhan Wang [mailto:yunhan@datatorrent.com]
>> *Sent:* 2016, July, 06 4:37 PM
>> *To:* users@apex.apache.org
>> *Subject:* Re: Multiple directories
>>
>>
>>
>> Hi Surya,
>>
>>
>>
>> Please check our fileIO-multiDir example.
>> https://github.com/DataTorrent/examples/tree/master/tutorials/fileIO-multiDir
>> .
>>
>> Hope this can help.
>>
>>
>>
>> Thanks,
>>
>> Yunhan
>>
>>
>>
>> On Wed, Jul 6, 2016 at 1:26 PM, Mukkamula, Suryavamshivardhan (CWM-NR) <
>> suryavamshivardhan.mukkamula@rbc.com> wrote:
>>
>> Hi,
>>
>>
>>
>> Can you please let me know, How would I add multiple directories to an
>> Operator which extends ‘AbstractFileInputOperator’?
>>
>>
>>
>> I would like to read from multiple directories by a single operator by
>> selecting multiple files using ‘filePatternRegExp’.
>>
>>
>>
>> Regards,
>>
>> Surya Vamshi
>>
>>
>>
>> _______________________________________________________________________
>>
>> If you received this email in error, please advise the sender (by return
>> email or otherwise) immediately. You have consented to receive the attached
>> electronically at the above-noted email address; please retain a copy of
>> this confirmation for future reference.
>>
>> Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
>> immédiatement, par retour de courriel ou par un autre moyen. Vous avez
>> accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à
>> l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de
>> cette confirmation pour les fins de reference future.
>>
>>
>>
>> _______________________________________________________________________
>>
>> If you received this email in error, please advise the sender (by return
>> email or otherwise) immediately. You have consented to receive the attached
>> electronically at the above-noted email address; please retain a copy of
>> this confirmation for future reference.
>>
>> Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
>> immédiatement, par retour de courriel ou par un autre moyen. Vous avez
>> accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à
>> l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de
>> cette confirmation pour les fins de reference future.
>>
>>
>

Mime
View raw message