flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paschek, Robert" <robert.pasc...@tu-berlin.de>
Subject Writing Intermediates to disk
Date Thu, 05 May 2016 13:29:46 GMT
Hi Mailing List,

I want to write and read intermediates to/from disk.
The following foo- codesnippet may illustrate my intention:

public void mapPartition(Iterable<T> tuples, Collector<T> out) {

                for (T tuple : tuples) {

                               if (Condition)
                                               out.collect(tuple);
                               else
                                               writeTupleToDisk
                }

                While ('TupleOnDisk')
                               out.collect('ReadNextTupleFromDisk');
}

I'am wondering if flink provides an integrated class for this purpose. I also have to precise
identify the files with the intermediates due parallelism of mapPartition.


Thank you in advance!
Robert

Mime
View raw message