From user-return-554-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Sat Jul 25 23:25:55 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 1974B18064D for ; Sun, 26 Jul 2020 01:25:55 +0200 (CEST) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id BF3FE126DCA for ; Sat, 25 Jul 2020 23:25:53 +0000 (UTC) Received: (qmail 38592 invoked by uid 500); 25 Jul 2020 23:25:53 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 38583 invoked by uid 99); 25 Jul 2020 23:25:53 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2020 23:25:53 +0000 Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id EC78B40286 for ; Sat, 25 Jul 2020 23:25:52 +0000 (UTC) Received: by mail-oo1-f43.google.com with SMTP id r13so1002465ool.3 for ; Sat, 25 Jul 2020 16:25:52 -0700 (PDT) X-Gm-Message-State: AOAM533ZbFfkz1ocBGbvyrIUgIC/NKT07hGXqfd6C/LVncuLPLgy5A0q v5YK2GhHTFvhuqDqd4FFaNB0vKO80aog+3rcP+8= X-Google-Smtp-Source: ABdhPJyLHWFcpZtup0hEJBLuVwNF32u2qqOh8wjzPq6Uuh4ZRR7PagrSpA4x3yFHxJ+vSn/I8pzqTkYYXWyf+RlqKgM= X-Received: by 2002:a4a:a782:: with SMTP id l2mr12799592oom.62.1595719552556; Sat, 25 Jul 2020 16:25:52 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jacques Nadeau Date: Sat, 25 Jul 2020 16:25:41 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: memory mapped record batches in Java To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="00000000000017659005ab4c693d" --00000000000017659005ab4c693d Content-Type: text/plain; charset="UTF-8" The current code doesn't preclude this path, it just doesn't have it implemented. In many cases, a more intelligent algorithm can page data into or out of main memory more efficiently (albeit with more work). This should be fairly straightforward to do. The easiest way to get started would probably be to implement a new allocation manager that uses MMap memory as backing instead of the current ones (Netty [1] and Unsafe [2]). From there, you could then enhance the reading to use that allocator to map the right offsets into the existing vectors. 1: https://github.com/apache/arrow/blob/master/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/NettyAllocationManager.java 2: https://github.com/apache/arrow/blob/master/java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/UnsafeAllocationManager.java On Sat, Jul 25, 2020 at 5:46 AM Chris Nuernberger wrote: > Hey, I am the author to a Clojure dataframe library, tech.ml.dataset > and we are looking to > upgrade our ability to handle out-of-memory datasets. > > I was hoping to use Arrow for this purpose specifically to have a > conversion mechanism where I could stream data into a single Arrow file > with multiple record batches and then load that file and mmap each record > batch. > > The current loading mechanism appears quite poor for this use case; it > assumes batch-at-a-time loading by mutating member variables of the root > schema and file loading mechanism and it copies each batch into process > memory. > > It seems to me that, assuming each batch is less than 2 GB, > FileChannel.map could be used for each record batch and this would allow > one to access data in those batches in a random-access order as opposed to > a single in-order traverse and it may allow larger-than-memory files to be > operated on. > > Is there any interest in this pathway? It seems like Arrow is quite close > to realizing this possibility or that it is already possible from nearly > all the other languages but the current Java design, unless I am misreading > the code, precludes this pathway. > > Thanks for any thoughts, feedback, > > Chris > --00000000000017659005ab4c693d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
The current code doesn't preclude this path, it just d= oesn't have it implemented. In many cases, a more intelligent algorithm= can page data into or out of main memory more efficiently (albeit with mor= e work). This should be fairly straightforward to do. The easiest way to ge= t started would probably be to implement a new allocation manager that uses= MMap memory as backing instead of the current ones (Netty [1] and Unsafe [= 2]). From there, you could then enhance the reading to use that allocator t= o map the right offsets into the existing vectors.


On Sat, Jul 25, 2020 at 5:46 AM Chris Nuernberger <chris@techascent.com> wrote:

Hey, I am the author=C2=A0to a Clojure dataframe libr= ary, tech.ml.dataset and we are looking to upgrade our ability to hand= le out-of-memory datasets.

I was hoping to use Arrow for this purpos= e specifically to have a conversion mechanism where I could stream data int= o a single Arrow file with multiple record batches and then load that file = and mmap each record batch.

The current loading mechanism appears qui= te poor for this use case; it assumes batch-at-a-time loading by mutating m= ember variables of the root schema and file loading mechanism and it copies= each batch into process memory.

It seems to me that, assuming each batch = is less than 2 GB, FileChannel.map could be used for each record batch and = this would allow one to access data in those batches in a random-access ord= er as opposed to a single in-order traverse and it may allow larger-than-me= mory files to be operated on.

Is there any interest in this pathway? I= t seems like Arrow is quite close to realizing this possibility or that it = is already possible from nearly all the other=C2=A0languages but the curren= t Java design, unless I am misreading the code, precludes this pathway.

Thanks for any thoughts, feedback,

Chris

--00000000000017659005ab4c693d--