flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Prevent job/operator from spilling to disk
Date Wed, 04 May 2016 16:19:01 GMT
Hi Max,

it is not possible to deactivate spilling to disk at the moment.
It might be possible to implement, but this would require a few more
changes to make it feasible.
For instance, we would need to add more fine-grained control about how
memory is distributed among operators.
This is currently statically decided by the optimizer and there is no
parameter to influence it.

Cheers, Fabian

2016-05-04 13:46 GMT+02:00 Maximilian Bode <maximilian.bode@tngtech.com>:

> Hi everyone,
>
> is there a way to prevent operators from spilling to disk? If not, would
> it be conceivable to make this configurable either per job or operator?
>
> The use case is a batch application with the formal requirement not to
> persist in-flight data to disk (even temporarily) so it would be preferable
> to see the job fail and then be able to grant sufficient memory and run it
> again.
>
> Cheers,
> Max
>
> —
> Maximilian Bode * Software Consultant * maximilian.bode@tngtech.com
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>

Mime
View raw message