Mailing-List: contact dev-help@arrow.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@arrow.apache.org
MIME-Version: 1.0
In-Reply-To: <927E21E7-69DD-4636-A299-9C45C0B23058@intel.com>
References: <56FB11EF.9050701@apache.org>
	<927E21E7-69DD-4636-A299-9C45C0B23058@intel.com>
Date: Wed, 30 Mar 2016 17:17:25 -0700
Message-ID: 
 <CALuGr6Yqi1uL_1Fg_+4dJdBYvvm5mTGcW62GaPrd+uW=xU6JVw@mail.gmail.com>
Subject: Re: A Proposal Apache Incubator Mnemonic as an alternative infra. for
 Apache Arrow
From: Henry Saputra <henry.saputra@gmail.com>
To: dev@mnemonic.incubator.apache.org
Cc: "dev@arrow.apache.org" <dev@arrow.apache.org>,
	"dev@mnemonic.apache.org" <dev@mnemonic.apache.org>
Content-Type: multipart/alternative; boundary=089e011843e0d6850b052f4d2f8f

--089e011843e0d6850b052f4d2f8f
Content-Type: text/plain; charset=UTF-8

The community for both podlings are bigger than the ones show up at Strata
=)

Would love to have the summary of the discussions in the dev@ list if
indeed some discussions happening at Strata.

- Henry

On Wed, Mar 30, 2016 at 5:03 PM, Wang, Yanping <yanping.wang@intel.com>
wrote:

> Hi, All
>
> I met with Jacques today at Strata, we think it would be great that Arrow
> and Mnemonic communities can have a F2F meeting together to talk about our
> integration.
> I have following two days, 4/11 Monday afternoon, or 4/15 Friday.
> We can meet at  intel SC campus.
>
> Would you let me know if you are able to join us and which day you'd
> prefer?
>
> Thanks
> Yanping
>
>
> On Mar 29, 2016, at 4:38 PM, Gary <garyw@apache.org<mailto:
> garyw@apache.org>> wrote:
>
> Yes, I agree with you and that's great if we could brainstorm here to
> collect more ideas about enabling non-volatile memory usage for Apache
> Arrow through Mnemonic.
>
> for the questions, my ideas are:
>
>
> - Right now you are using unpooled persistent memory. Does that make sense
> or does chunking make more sense?
>
> Gary: I think it could make some sense if developer knows that their
> datasets are very big and they want Apache Arrow to keep most of them in
> memory for intensive computing e.g. sort.
>           the developer certainly can spill their Mnemonic managed
> datasets into disk but this way seems a bit inefficient in some scenarios
> that might depend on concrete application logic .
>
>
> - What do you think is the right way to transition back and forth between
> persistent and ephemeral memory? What do you think will be the first
> pattern to be adopted. For example, do you think we should try to use it as
> a tiered storage for sort spilling (before hitting the disk), or should we
> use it for caching?
> Gary: my 2 cents, the netty library looks not yet provide a elegant switch
> mechanism for Arrow to use, probably we can change the logic around
> "initialCapacity > directArena.chunkSize" to control which buffer put on
> off-heap or managed by Mnemonic, another approach is to let memory
> clustering mechanism of Mnemonic managing hybrid memory-like spaces instead
> of part logics of class PooledByteBufAllocatorL.
> Regarding the sorting, I think it is a typical case of random access to
> the data, we should avoid spilling as much as possible.
> my 2 cents, the performance could be
> all in off-heap if possible > mnemonic used as cache > all in mnemonic
> using NVMe/disk >  off-heap + spilling
> the code simplicity would be
> all in off-heap if possible >  all in mnemonic using NVMe/disk > mnemonic
> used as cache >  off-heap + spilling
>
> the reason why the mode "mnemonic used as cache + spilling" probably
> unnecessary is mnemonic could provide nearly equivalent capacity of disk.
>
> Thanks.
> Gary.
>
>
> -----Original Message-----
>
> From: Jacques Nadeau [mailto:jacques@apache.org]
>
> Sent: Tuesday, March 29, 2016 8:05 AM
>
> To: <mailto:dev@arrow.apache.org> dev@arrow.apache.org<mailto:
> dev@arrow.apache.org>
>
> Subject: Re: A Proposal Apache Incubator Mnemonic as an alternative infra.
> for Apache Arrow
>
>
>
> This is super cool. A couple of questions:
>
>
>
> - Right now you are using unpooled persistent memory. Does that make sense
> or does chunking make more sense?
>
> - What do you think is the right way to transition back and forth between
> persistent and ephemeral memory? What do you think will be the first
> pattern to be adopted. For example, do you think we should try to use it as
> a tiered storage for sort spilling (before hitting the disk), or should we
> use it for caching?
>
>
>
> I think it will be much easier to think about this in the context of a
> primary or first use case. Do you have something in mind or should we
> brainstorm here?
>
>
>
> On Wed, Mar 23, 2016 at 7:16 PM, Gary <garyw@apache.org<mailto:
> garyw@apache.org>> wrote:
>
>
>
> > Hello,
>
> >
>
> >    We have created a patch for Apache Arrow to leverage Apache
>
> > incubator Mnemonic as an alternative infra. for underlying memory
>
> > resources allocation, you can find it as below forked repo.
>
> >
>
> > <https://github.com/NonVolatileComputing/arrow>
> https://github.com/NonVolatileComputing/arrow
>
> >
>
> >     By this way, Apache Arrow could take some structural benefits from
>
> > Mnemonic project they are
>
> >
>
> >     - Arrow is able to leverage larger capacity of high performance
>
> > hybrid storage devices. e.g. high-end SSD, NVMe
>
> >
>
> >     - Mnemonic provide a potential opportunity for Arrow to
>
> > optimize/tuning its allocation algorithms as a native Arrow-oriented
>
> > allocation services
>
> >
>
> >     - The non-volatile features of  Mnemonic make it possible that
>
> > Arrow could make its columnar in-memory data shared between different
>
> > applications or across life-cycle of single application
>
> >
>
> >     - Arrow could take advantages of coming Mnemonic features of
>
> > memory clustering/DOG (distributed object graph) and massive native
>
> > computing
>
> >
>
> >     - Mnemonic helps to reduce the pressure of main memory utilization
>
> > and its related system wide overheads.
>
> >
>
> >    Our this patch is designed to minimize the changes for user to use
>
> > Arrow, please check out the test cases provided by this patch for your
>
> > reference.
>
> >
>
> >    Note that, we need to put allocator services to a specified
>
> > position (indicated by pom.xml) for Mnemonic backed Arrow related test
>
> > cases to run because those services are required for external
>
> > memory-like device management.
>
> >
>
> >    Please give your comments and review feedback for better
>
> > collaboration of Apache Arrow and Mnemonic, Thanks.
>
> >
>
> > Best Regards.
>
> > Gary.
>
> >
>
> >
>
> >
>
> <smime.p7m>
> <gpgol000.txt>
>

--089e011843e0d6850b052f4d2f8f--