arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nuernberger <>
Subject Bulk copy methods to/from Java vectors
Date Sat, 25 Jul 2020 13:01:58 GMT

I would like to have bulk methods for copying data into a vector.
Specifically, I have an existing data table so I know the desired lengths
of the columns.  I can also precalculate the necessary buffer sizes for any
variable sized column.

What I don't see is how to pre-allocate columns of a given size.  When I
use setValueCount on a column and then use the set method I get a netty
error.  What I was hoping for is some allocation method, especially for
primitive data, that allocates the desired uninitialized memory for the
valide and buffer data and then hands those two buffers back to me so I can
use memcpy and friends as opposed to repeated calls to setSafe.

Repeated calls to setSafe are time consuming, not parallelizable, and
unnecessary when I know the data rectangle I would like to transfer into a
record batch.

In my case I have the data pre-cut.  How would you recommend copying bulk
portions of data (that may be in java arrays or in some more abstract
interface) into a record batch?

Thanks for any help,


View raw message