apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munagala Ramanath <...@datatorrent.com>
Subject Re: Reading Multiple directories in parallel
Date Tue, 28 Jun 2016 18:03:24 GMT
Not sure I fully understand the question but you can add whatever fields
you need
to your class that extends *AbstractFileInputOperator*. For example,

https://github.com/DataTorrent/examples/blob/master/tutorials/fileIO-multiDir/src/main/java/com/example/fileIO/FileReaderMultiDir.java
defines fields *directories* and *partitionCounts*.

You can then set these fields as needed in *definePartitions*.

Ram

On Tue, Jun 28, 2016 at 10:31 AM, Mukkamula, Suryavamshivardhan (CWM-NR) <
suryavamshivardhan.mukkamula@rbc.com> wrote:

> Hi Ram,
>
> Can you please suggest , how would I add another variable (like
> ‘directory’) while creating multiple partitions of
> AbstractFileInputOperator in the define partition method.
>
> I have currently added variables in the AbstractFileInputOperator , which
> I guess not a better way.
>
> These variables are basically used to scan directories in parallel
> differently.
>
> Regards,
> Surya Vamshi
>
>
> _______________________________________________________________________
>
> If you received this email in error, please advise the sender (by return
> email or otherwise) immediately. You have consented to receive the attached
> electronically at the above-noted email address; please retain a copy of
> this confirmation for future reference.
>
> Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
> immédiatement, par retour de courriel ou par un autre moyen. Vous avez
> accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à
> l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de
> cette confirmation pour les fins de reference future.
>
>

Mime
View raw message