flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chiwan Park <chiwanp...@apache.org>
Subject Re: Code related to spilling data to disk
Date Wed, 22 Jun 2016 12:05:17 GMT

I’m not sure about the reason to use JVM heap instead of managed memory, but It seems that
the reason is using JVM heap makes development easier. Maybe Stephan can give exact answer
to you. I think managed memory still has benefit in terms of GC  time and memory utilization.

The Flink community has a plan [1] to move data structures for streaming operators to managed

[1]: https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-778DAD7eqw5GANwE/edit#

Chiwan Park

> On Jun 22, 2016, at 8:39 PM, Tae-Geon Um <taegeonum@gmail.com> wrote:
> Thank you for your answer to my question, Chiwan :)  
> Can I ask another question?  
>> On Jun 22, 2016, at 7:22 PM, Chiwan Park <chiwanpark@apache.org> wrote:
>> Hi Tae-Geon,
>> AFAIK, spilling *data* to disk happens only when managed memory is used. Currently,
streaming API (DataStream) doesn’t use managed memory yet. `MutableHashTable` is one of
representative usage of managed memory with disk spilling. Note that some special structures
such as `CompactingHashTable` doesn’t spill data to disk even though they use the manage
memory to achieve high performance.
> As far as I understand, spilling data is only performed on batch mode. 
> Do you know why streaming mode does not use managed memory? 
> Is this because the performance gain is negligible?
>> About spilling *states*, I think that it depends on how state backends is implemented.
For example, `FsStateBackend` saves states to file system but `MemoryStateBackend` doesn’t.
`RocksDBStateBackend` uses memory first and also can spill states to disk.
> I’ve found a nice document on the state backend [1]. I will take a look at this doc
to know the detail. 
> Thanks! 
> Taegeon
> [1]: https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/state_backends.html#state-backends
>> Regards,
>> Chiwan Park
>>> On Jun 22, 2016, at 3:27 PM, Tae-Geon Um <taegeonum@gmail.com> wrote:
>>> I have another question. 
>>> Is the spilling only executed on batch mode? 
>>> What happen on streaming mode?  
>>>> On Jun 22, 2016, at 1:48 PM, Tae-Geon Um <taegeonum@gmail.com> wrote:
>>>> Hi, all
>>>> As far as I know, Flink spills data (states?) to disk if the data exceeds
memory threshold or there exists memory pressure.
>>>> i’d like to know the detail of how Flink spills data to disk. 
>>>> Could you please let me know which codes do I have to investigate? 
>>>> Thanks,
>>>> Taegeon

View raw message