arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary <ga...@apache.org>
Subject A Proposal Apache Incubator Mnemonic as an alternative infra. for Apache Arrow
Date Thu, 24 Mar 2016 02:16:11 GMT
Hello,

   We have created a patch for Apache Arrow to leverage Apache incubator
Mnemonic as an alternative infra. for underlying memory resources
allocation, you can find it as below forked repo.

https://github.com/NonVolatileComputing/arrow

    By this way, Apache Arrow could take some structural benefits from
Mnemonic project they are

    - Arrow is able to leverage larger capacity of high performance
hybrid storage devices. e.g. high-end SSD, NVMe

    - Mnemonic provide a potential opportunity for Arrow to
optimize/tuning its allocation algorithms as a native Arrow-oriented
allocation services

    - The non-volatile features of  Mnemonic make it possible that Arrow
could make its columnar in-memory data shared between different
applications or across life-cycle of single application

    - Arrow could take advantages of coming Mnemonic features of memory
clustering/DOG (distributed object graph) and massive native computing

    - Mnemonic helps to reduce the pressure of main memory utilization
and its related system wide overheads.

   Our this patch is designed to minimize the changes for user to use
Arrow, please check out the test cases provided by this patch for your
reference.

   Note that, we need to put allocator services to a specified position
(indicated by pom.xml) for Mnemonic backed Arrow related test cases to
run because those services are required for external memory-like device
management.

   Please give your comments and review feedback for better
collaboration of Apache Arrow and Mnemonic, Thanks.

Best Regards.
Gary.



Mime
View raw message