flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mxm <...@git.apache.org>
Subject [GitHub] flink pull request: [FLINK-1320] Add an off-heap variant of the ma...
Date Thu, 08 Jan 2015 11:14:15 GMT
GitHub user mxm opened a pull request:


    [FLINK-1320] Add an off-heap variant of the managed memory

    The MemorySegment class has been converted into an abstract class. Its old JVM
    heap implementation can now be found in HeapMemorySegment. In addition, an
    implementation which uses direct (outside the JVM heap) memory allocation can be
    found in DirectMemorySegment. Both of the classes use the sun.misc.Unsafe class
    which modifies the memory directly. This method is unsafe in the sense that any
    incorrectly written bytes may crash the JVM. By default, both classes perform
    boundary checks when writing to the memory.
    The DefaultMemoryManager has been renamed to HeapMemoryManager. In addition, a
    DirectMemoryManager has been added. The classes' main difference is the queue
    freeSegments which holds the memory segments. In the HeapMemoryManager, the
    queue holds byte arrays while in the DirectMemoryManager, the queue holds
    The direct (off-heap) memory management is enabled for the task manager when the
    config entry "taskmanager.memory.directAllocation" is set to "true". Like for
    the heap memory management, if "taskmanager.memory.size" is set to a value
    greater 0, the amount of memory in mega bytes will be allocated for the memory
    allocation. Otherwise, a fraction (0.7) of the task manager JVM heap will be
    used to determine the amount of memory to allocate. As of now, the user has to
    take care to properly adjust the task manager's heap memory size (as configured
    in "taskmanager.heap.mb") when using direct (off-heap) memory allocation.
    The tests for all classes have been changed to test both classes.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mxm/flink off_heap_rebased

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #290
commit 9d711ccd1bbd581845e3b690cf1cc5d8dfcfa603
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-12-16T11:27:24Z

    [FLINK-1320] Mockup draft for off heap memory segments

commit a44309373da2449c483463cf1f093667e3ccfb79
Author: Max <max@posteo.de>
Date:   2014-12-18T12:27:08Z

    [FLINK-1320] rename DefaultMemoryManager to HeapMemoryManager, create DirectMemoryManager,
add default config entry

commit 9e64df34ff6b01f3bea17e3ffc0bbafc2dc769e6
Author: Max <max@posteo.de>
Date:   2014-12-19T10:49:22Z

    [FLINK-1320] add compare and swapBytes methods into abstract class MemorySegment, implement
swapBytes method differently without supplying a swapping buffer, make HeapMemorySegement
the default in all tests

commit 3a447e2afb5644bd294874f144b708380ec1b7b0
Author: Max <max@posteo.de>
Date:   2014-12-19T15:00:59Z

    [FLINK-1320] set default byte encoding to big endian

commit cbf8e0057975cba6a8fa3b6c71aca612a5f8865f
Author: Max <max@posteo.de>
Date:   2014-12-21T22:46:07Z

    [FLINK-1320] rename DefaultMemoryManager to HeapMemoryManager

commit 4e7a5407a53ac9bf9552005b48998161f8285e25
Author: Max <max@posteo.de>
Date:   2014-12-22T19:33:41Z

    [FLINK-1320] configurable switch for heap and direct (off-heap)  memory allocation

commit ff3432c61295210f1081ffb02738dfdc2b1bc2bc
Author: Max <max@posteo.de>
Date:   2015-01-05T13:03:22Z

    [FLINK-1320] add default parameter for local execution, rename test suite

commit 021f094124cf8a057e055dba9e728f446aa6ea54
Author: Max <max@posteo.de>
Date:   2015-01-06T18:31:03Z

    [FLINK-1320] add documentation and correct code formatting

commit df36f7abfcc53829ea1d97f18f6491ab200356b4
Author: Max <max@posteo.de>
Date:   2015-01-07T10:46:28Z

    [FLINK-1320] fix isFreed() method to return correct status

commit ca4614b25fa734fdf468b0ec26aabf29d3d890fe
Author: Max <max@posteo.de>
Date:   2015-01-07T10:47:25Z

    [FLINK-1320] fix boundary checks in for get and put

commit 2783a807af3f14c40bb733038688e64ab350aa0e
Author: Max <max@posteo.de>
Date:   2015-01-07T11:32:00Z

    [FLINK-1320] rename HeapMemorySegmentTest to MemorySegmentTest
    this class should test both implementations of MemorySegment

commit bdad3318a502c7d00dd56afae8273635cb96c813
Author: Max <max@posteo.de>
Date:   2015-01-07T14:21:08Z

    [FLINK-1320] adapt MemorySegmentTest to run tests with both heap and direct memory segments

commit 02c27b0e59f9b64203ae94a0c1a0b750dcc7303d
Author: Max <max@posteo.de>
Date:   2015-01-07T14:52:29Z

    [FLINK-1320] rename HeapMemoryManagerTest to MemoryManagerTest

commit 4d319565542973ece1a4796d14f9437ef35bb785
Author: Max <max@posteo.de>
Date:   2015-01-07T16:40:57Z

    [FLINK-1320] adapt MemoryManagerTest to run tests with both heap and direct memory managers


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message