flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Taskmanager memory
Date Wed, 09 Dec 2015 09:44:36 GMT
@Sebastian: Getting memory away from the JVM is tricky always, completely
independent of pre-allocation of managed memory or lazy allocation.

But here is something that may work:
  - Start Flink in streaming mode - that will make it allocate managed
memory lazily
  - Set the memory to offheap memory. That way the JVM heap is small. The
off-heap memory is returned when no longer used deallocated - this releases
memory much better than JVM shrinking the heap.



On Wed, Dec 9, 2015 at 10:06 AM, Fabian Hueske <fhueske@gmail.com> wrote:

> Hi Sebastian,
>
> There is no way to return memory from a Flink process except shutting the
> process down.
> I think YARN could help in your setup. In a YARN setup, you can flexibly
> start and stop Flink sessions with different configurations (memory, TMs,
> slots) or run a single job. When running a single job, Flink will allocate
> resources and free them after the job is done.
>
> Best, Fabian
>
> 2015-12-09 9:46 GMT+01:00 Kruse, Sebastian <Sebastian.Kruse@hpi.de>:
>
>> Hi everyone,
>>
>>
>> I am currently looking into how Flink can coexist and interoperate with
>> other frameworks in a cluster, such as plain single-machine processes
>> or Spark​. ​Tachyon seems to be nice solution to exchange data between
>> them.
>>
>>
>> However, I think it is a problem that Flink's taskmanagers allocate their managed
>> memory upfront - in contrast to Spark, as far as I know. If I want ​a taskmanager
>> to yield its main memory, so that another process can use that memory, is
>> there any other option besides shutting that taskmanager down? Would it be
>> beneficial to use YARN?
>>
>> Thanks for your help!
>>
>>
>> Cheers,
>>
>> Sebastian
>>
>
>

Mime
View raw message