apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukkamula, Suryavamshivardhan (CWM-NR)" <suryavamshivardhan.mukkam...@rbc.com>
Subject RE: Reading Multiple directories in parallel
Date Tue, 28 Jun 2016 18:17:43 GMT
Hi Ram,

I would like to add the parameters for each partition like below. Each operator to be given
with its own configuration file and source identifier. If there is any other way please let
me know ?
In my current definepartition() method , I am doing similarly like below, but I have to add
setter and getter methods in AbstractFileInputOperator class.

for (int j = 0; j < numDirs; ++j) {
      int first = sliceFirstIndex[j];
      int last = first + partitionCounts[j];
      String dir = directories[j];
      LOG.info("definePartitions: first = {}, last = {}, dir = {}", first, last, dir);
      for (int i = first; i < last; ++i) {
        AbstractFileInputOperator<String> oper = cloneObject(kryo, this);
        oper.setDirectory(dir);
        oper.setSourceId(<sourceId>);
        oper.setConfigFile(<fileName>);
        //oper.setpIndex(i);
        SlicedDirectoryScanner scn = (SlicedDirectoryScanner) scanners.get(i);
        scn.setStartIndex(first);
        scn.setEndIndex(last);
        scn.setDirectory(dir);

        oper.setScanner(scn);
        newPartitions.add(new DefaultPartition<>(oper));
        newManagers.add(oper.getIdempotentStorageManager());
      }
    }


Regards,
Surya Vamshi

From: Munagala Ramanath [mailto:ram@datatorrent.com]
Sent: 2016, June, 28 2:03 PM
To: users@apex.apache.org
Subject: Re: Reading Multiple directories in parallel

Not sure I fully understand the question but you can add whatever fields you need
to your class that extends AbstractFileInputOperator. For example,
  https://github.com/DataTorrent/examples/blob/master/tutorials/fileIO-multiDir/src/main/java/com/example/fileIO/FileReaderMultiDir.java
defines fields directories and partitionCounts.

You can then set these fields as needed in definePartitions.

Ram

On Tue, Jun 28, 2016 at 10:31 AM, Mukkamula, Suryavamshivardhan (CWM-NR) <suryavamshivardhan.mukkamula@rbc.com<mailto:suryavamshivardhan.mukkamula@rbc.com>>
wrote:
Hi Ram,

Can you please suggest , how would I add another variable (like ‘directory’) while creating
multiple partitions of AbstractFileInputOperator in the define partition method.

I have currently added variables in the AbstractFileInputOperator , which I guess not a better
way.

These variables are basically used to scan directories in parallel differently.

Regards,
Surya Vamshi


_______________________________________________________________________

If you received this email in error, please advise the sender (by return email or otherwise)
immediately. You have consented to receive the attached electronically at the above-noted
email address; please retain a copy of this confirmation for future reference.

Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par
retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s)
ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver
une copie de cette confirmation pour les fins de reference future.

_______________________________________________________________________
If you received this email in error, please advise the sender (by return email or otherwise)
immediately. You have consented to receive the attached electronically at the above-noted
email address; please retain a copy of this confirmation for future reference.  

Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par
retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s)
ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver
une copie de cette confirmation pour les fins de reference future.
Mime
View raw message