flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mihail Vieru <vi...@informatik.hu-berlin.de>
Subject Re: write data set to a single file
Date Wed, 13 May 2015 18:14:54 GMT
Awesome, it works. Thanks! :)

On 13.05.2015 20:05, Stephan Ewen wrote:
> If you want to write a single file, you need to write it with one 
> task. So, you can run a program with parallelism 100 and just set the 
> sink operator to parallelism 1.
>
> You can set the parallelism of each individual operator by calling 
> "setParallelism()" after the operation, for example 
> "result.writeAsText(path).setParallelism(1)".
>
>
> On Wed, May 13, 2015 at 8:02 PM, Mihail Vieru 
> <vieru@informatik.hu-berlin.de <mailto:vieru@informatik.hu-berlin.de>> 
> wrote:
>
>     Hi,
>
>     I need to write a data set to a single file without setting the
>     parallelism to 1.
>     How can I achieve this?
>
>     Cheers,
>     Mihail
>
>     P.S.: it's for persisting intermediate results in loops and
>     reading those in the next iteration.
>     Which btw work for higher iteration counts with explicit persistence.
>
>


Mime
View raw message