flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Any way to increase sort buffer size?
Date Fri, 18 Nov 2016 10:02:48 GMT
Hi Gabor,

I don't think there is a way to tune the memory settings for specific
operators.

For that you would need to change the memory allocation in the optimizers,
which is possible but not a lightweight change either.
If you want to get something working, you could add a method to the API to
manually specify a memory fraction. The information could be passed through
the API to the optimizer which takes the explicitly specified fraction into
account when assigning memory budgets (see Optimizer.java [1] and
PlanFinalizer.java [2]) for how that works.

Cheers,
Fabian

[1]
https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/Optimizer.java
[2]
https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/traversals/PlanFinalizer.java

2016-11-16 17:29 GMT+01:00 Gábor Hermann <mail@gaborhermann.com>:

> Hi all,
>
> Is there any way to increase the sort buffer size other than increasing
> the overall TaskManager memory?
> The following error comes up running a job with huge matrix block objects
> on a cluster:
>
> Error obtaining the sorted input: Thread 'SortMerger Reading Thread'
> terminated due to an exception: The record exceeds the maximum size of a
> sort buffer (current maximum: 100499456 bytes).
>
> Every TM has at least 40 GB of memory while the maximum sort buffer size
> is at 100 MB. What is the reason for this limit? Sorry if I'm missing
> something, but I have not found any related discussion or documentation yet.
>
> Cheers,
> Gabor
>
>

Mime
View raw message