Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
Subject: Re: Any way to increase sort buffer size?
To: user@flink.apache.org
References: <fd04b4af-6f7d-942e-b9b9-8aa3ab323ece@gaborhermann.com>
 <CAAdrtT14W0mvmu-0HqAn+4VAD46wxXuzd6P70_dT0TujrZw3BQ@mail.gmail.com>
From: =?UTF-8?Q?G=c3=a1bor_Hermann?= <mail@gaborhermann.com>
Message-ID: <56a43e16-0c4b-2c39-6283-6cdc8f5382ec@gaborhermann.com>
Date: Fri, 18 Nov 2016 17:56:00 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Thunderbird/45.4.0
MIME-Version: 1.0
In-Reply-To: <CAAdrtT14W0mvmu-0HqAn+4VAD46wxXuzd6P70_dT0TujrZw3BQ@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------367B2D90DD00A9485360A79F"
archived-at: Fri, 18 Nov 2016 16:56:18 -0000

This is a multi-part message in MIME format.
--------------367B2D90DD00A9485360A79F
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Fabian,

Thanks for your answer!

I see that it's not a lightweight change. I guess it's easier if I find 
a workaround for using smaller objects.

Cheers,
Gabor


On 2016-11-18 11:02, Fabian Hueske wrote:
> Hi Gabor,
>
> I don't think there is a way to tune the memory settings for specific 
> operators.
>
> For that you would need to change the memory allocation in the 
> optimizers, which is possible but not a lightweight change either.
> If you want to get something working, you could add a method to the 
> API to manually specify a memory fraction. The information could be 
> passed through the API to the optimizer which takes the explicitly 
> specified fraction into account when assigning memory budgets (see 
> Optimizer.java [1] and PlanFinalizer.java [2]) for how that works.
>
> Cheers,
> Fabian
>
> [1] 
> https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/Optimizer.java 
>
> [2] 
> https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/traversals/PlanFinalizer.java
>
> 2016-11-16 17:29 GMT+01:00 Gábor Hermann <mail@gaborhermann.com 
> <mailto:mail@gaborhermann.com>>:
>
>     Hi all,
>
>     Is there any way to increase the sort buffer size other than
>     increasing the overall TaskManager memory?
>     The following error comes up running a job with huge matrix block
>     objects on a cluster:
>
>     Error obtaining the sorted input: Thread 'SortMerger Reading
>     Thread' terminated due to an exception: The record exceeds the
>     maximum size of a sort buffer (current maximum: 100499456 bytes).
>
>     Every TM has at least 40 GB of memory while the maximum sort
>     buffer size is at 100 MB. What is the reason for this limit? Sorry
>     if I'm missing something, but I have not found any related
>     discussion or documentation yet.
>
>     Cheers,
>     Gabor
>
>


--------------367B2D90DD00A9485360A79F
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Hi Fabian,<br>
      <br>
      Thanks for your answer!<br>
      <br>
      I see that it's not a lightweight change. I guess it's easier if I
      find a workaround for using smaller objects.<br>
      <br>
      Cheers,<br>
      Gabor<br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 2016-11-18 11:02, Fabian Hueske
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAAdrtT14W0mvmu-0HqAn+4VAD46wxXuzd6P70_dT0TujrZw3BQ@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>
          <div>
            <div>
              <div>
                <div>Hi Gabor,<br>
                  <br>
                </div>
                I don't think there is a way to tune the memory settings
                for specific operators. <br>
                <br>
              </div>
              For that you would need to change the memory allocation in
              the optimizers, which is possible but not a lightweight
              change either.<br>
            </div>
            If you want to get something working, you could add a method
            to the API to manually specify a memory fraction. The
            information could be passed through the API to the optimizer
            which takes the explicitly specified fraction into account
            when assigning memory budgets (see Optimizer.java [1] and
            PlanFinalizer.java [2]) for how that works.<br>
            <br>
          </div>
          Cheers,<br>
        </div>
        Fabian <br>
        <br>
        [1] <a moz-do-not-send="true"
href="https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/Optimizer.java">https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/Optimizer.java</a>
        <br>
        [2] <a moz-do-not-send="true"
href="https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/traversals/PlanFinalizer.java">https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/traversals/PlanFinalizer.java</a><br>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">2016-11-16 17:29 GMT+01:00 Gábor
          Hermann <span dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:mail@gaborhermann.com" target="_blank">mail@gaborhermann.com</a>&gt;</span>:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
            <br>
            Is there any way to increase the sort buffer size other than
            increasing the overall TaskManager memory?<br>
            The following error comes up running a job with huge matrix
            block objects on a cluster:<br>
            <br>
            Error obtaining the sorted input: Thread 'SortMerger Reading
            Thread' terminated due to an exception: The record exceeds
            the maximum size of a sort buffer (current maximum:
            100499456 bytes).<br>
            <br>
            Every TM has at least 40 GB of memory while the maximum sort
            buffer size is at 100 MB. What is the reason for this limit?
            Sorry if I'm missing something, but I have not found any
            related discussion or documentation yet.<br>
            <br>
            Cheers,<br>
            Gabor<br>
            <br>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </body>
</html>

--------------367B2D90DD00A9485360A79F--